Data storage method, device, storage device and readable storage medium

Abstract

The present invention relates to the field of storage technology, and provides a data storage method, device, storage device and readable storage medium, wherein the method comprises: receiving a write request sent by a client for writing data to be written, wherein the write request comprises a file to be written, a length to be written of the data to be written, and a primary cache area and a secondary cache area are pre-bound to the file to be written; if the length to be written is not greater than a preset value, the data to be written is written into the primary cache area and the secondary cache area, so that the data to be written is stored from the primary cache area or the secondary cache area to the storage area; if the length to be written is greater than a preset value, the data to be written is written into the secondary cache area, so that the data to be written is stored from the secondary cache area to the storage area; and a response message for the write request is returned to the client. The present invention can quickly return a response message to the client, thereby improving the overall write performance of the storage device in the storage system.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data storage method, apparatus, storage device and readable storage medium. Background Art

[0002] In recent years, the field of storage technology has developed rapidly. In large-capacity storage scenarios, massive amounts of data are usually generated and require a large amount of storage resources to store. Based on cost considerations, mechanical hard disks are usually used to complete the storage of large-scale data resources. However, due to the performance bottleneck of mechanical hard disks, the processing capacity is poor when faced with a large number of continuous small access requests. Faced with such defects, a large number of fragmented small data write requests have become the performance bottleneck of the storage system. How to effectively improve the overall write performance of the storage system has become a problem that technicians in this field need to solve urgently. Summary of the invention

[0003] The object of the present invention is to provide a data storage method, apparatus, storage device and readable storage medium, which can improve the overall writing performance of the storage device in the storage system, thereby improving the finishing writing performance of the storage system.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a data storage method, which is applied to a storage device, wherein the storage device is communicatively connected to a client, the storage device includes a first-level cache area, a second-level cache area and a storage area, the write performance of the first-level cache area and the write performance of the second-level cache area are both higher than the write performance of the storage area, and the method includes: receiving a write request sent by the client for writing data to be written, wherein the write request includes a file to be written and a length to be written of the data to be written, and the first-level cache area and the second-level cache area are pre-bound to the file to be written; if the length to be written is not greater than a preset value, writing the data to be written into the first-level cache area and the second-level cache area, so that the data to be written is stored from the first-level cache area or the second-level cache area to the storage area; if the length to be written is greater than the preset value, writing the data to be written into the second-level cache area, so that the data to be written is stored from the second-level cache area to the storage area; and returning a response message to the write request to the client.

[0006] Optionally, the first-level cache area is divided into a plurality of cache blocks according to a first preset length, the second-level cache area is divided into a plurality of cache slices according to a second preset length, the storage area is divided into a plurality of storage blocks according to the second preset length, the cache blocks correspond to the cache slices one-to-one, and the cache slices correspond to the storage blocks one-to-one, and the step of writing the to-be-written data into the first-level cache area and the second-level cache area includes:

[0007] Obtain a target storage block to be written to the file to be written and a first offset position in the target storage block;

[0008] Determine a target cache block in the first-level cache area corresponding to the target storage block;

[0009] Determine a target cache slice in the secondary cache area corresponding to the target storage block;

[0010] The data to be written is written into the target cache block, and the data to be written is written into the target cache slice according to the first offset position.

[0011] Optionally, the secondary cache area is divided into a plurality of cache slices according to a second preset length, and the method further includes:

[0012] Determine whether each of the cache slices in the secondary cache area meets a preset disk flushing condition;

[0013] A cache slice that meets the preset disk flushing condition is determined as a cache slice to be flushed, and the cache slice to be flushed contains data that needs to be stored in the storage area.

[0014] Optionally, the step of determining whether each cache slice in the secondary cache area meets a preset disk flushing condition includes:

[0015] For any target cache slice in the secondary cache area, if the length of data in the target cache slice that has not been flushed to disk is greater than a preset length, it is determined that the target cache slice meets the preset flushing condition;

[0016] If the file to be written is in a to-be-closed state, determining that the cache slices with unflushed data in all the cache slices meet the preset flushing condition;

[0017] If the current time is within the preset disk flushing cycle, it is determined that the cache slices with unflushed data in all the cache slices meet the preset disk flushing condition.

[0018] Optionally, the method further comprises:

[0019] Determine whether the actual data to be flushed in the cache slice to be flushed exists in the first-level cache area;

[0020] If the actual data to be flushed exists in the first-level cache area, storing the actual data to be flushed from the first-level cache area to the storage area;

[0021] If the actual data to be flushed does not exist in the primary cache area, the actual data to be flushed is stored from the secondary cache area to the storage area.

[0022] Optionally, the first-level cache area is divided into a plurality of cache blocks according to a first preset length, each of the cache blocks includes a first length of cached block data and an end position of the block data in a storage block corresponding to each of the cache blocks, and the step of determining whether the actual data to be flushed in the cache slice to be flushed exists in the first-level cache area includes:

[0023] Obtaining a second offset position of the current disk-to-be-flushed data in a storage block corresponding to the disk-to-be-flushed cache slice;

[0024] Determine a to-be-flushed disk cache block corresponding to the to-be-flushed disk cache slice from the multiple cache blocks;

[0025] Obtaining a second length of actual data to be flushed according to the end position of the cache block to be flushed and the second offset position;

[0026] If the second length is smaller than the first length of the to-be-flushed cache block, it is determined that the to-be-flushed data in the to-be-flushed cache slice exists in the first-level cache area;

[0027] If the second length is not less than the first length of the cache block to be flushed, it is determined that the data to be flushed in the cache block to be flushed does not exist in the first-level cache area.

[0028] Optionally, the secondary cache area is divided into a plurality of cache slices according to a second preset length, the storage area is divided into a plurality of storage blocks according to the second preset length, the cache slices correspond to the storage blocks one by one, and the method further includes:

[0029] When opening the abnormally closed to-be-written file, taking the cache slice with valid data in the secondary cache area as the cache slice to be restored;

[0030] The valid data in the cache slice to be restored is stored in a storage block in the storage area corresponding to the cache slice to be restored.

[0031] In a second aspect, an embodiment of the present invention provides a data storage device, which is applied to a storage device, wherein the storage device is communicatively connected to a client, wherein the storage device includes a primary cache area, a secondary cache area, and a storage area, wherein the write performance of the primary cache area and the write performance of the secondary cache area are both higher than the write performance of the storage area, and the device includes:

[0032] A receiving module, configured to receive a write request sent by the client for writing data to be written, wherein the write request includes a file to be written and a length of the data to be written, and the first-level cache area and the second-level cache area are pre-bound to the file to be written;

[0033] A writing module, used for writing the data to be written into the first-level cache area and the second-level cache area if the length to be written is not greater than a preset value, so that the data to be written is stored from the first-level cache area or the second-level cache area to the storage area;

[0034] The writing module is further used for writing the data to be written into the secondary cache area if the length to be written is greater than the preset value, so that the data to be written is stored from the secondary cache area to the storage area;

[0035] A response module is used to return a response message to the client for the write request.

[0036] In a third aspect, an embodiment of the present invention provides a storage device, including a processor and a memory; the memory is used to store a program; and the processor is used to implement the data storage method described in the first aspect when executing the program.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data storage method as described in the first aspect above.

[0038] Compared with the prior art, the data storage method, apparatus, storage device and readable storage medium provided by the embodiments of the present invention, when the client needs to write data to be written to the storage device, sends a write request to the storage device. If the length of the data to be written is greater than a preset value, the data to be written is written to the first-level cache. If the length of the data to be written is not greater than the preset value, the data to be written is written to the first-level cache and the second-level cache, and then returned to the client. The data in the first-level cache and the second-level cache are finally stored in the storage area. Since the write performance of the first-level cache and the second-level cache are both greater than the write performance of the storage area, the storage device can quickly return a response message to the client, thereby improving the overall write performance of the storage device in the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 An example diagram of an application scenario provided by an embodiment of the present invention.

[0041] Figure 2 An example diagram of another specific application scenario provided by an embodiment of the present invention.

[0042] Figure 3 A schematic diagram of the hardware structure of a storage device provided in an embodiment of the present invention.

[0043] Figure 4 An example of a data storage method provided by the present invention Figure 1 .

[0044] Figure 5 An example diagram of the mapping relationship between the first-level cache area and the second-level cache area provided in an embodiment of the present invention.

[0045] Figure 6 This is an example diagram of writing the to-be-written data into the target cache block provided by an embodiment of the present invention.

[0046] Figure 7 This is an example diagram of actual disk data to be flushed provided by an embodiment of the present invention.

[0047] Figure 8 An example diagram of the overall implementation of the patrol thread and the write-back thread provided in an embodiment of the present invention.

[0048] Fig. 9 An example diagram of the layout of the secondary cache area provided in an embodiment of the present invention.

[0049] Fig.10 A block diagram of a data storage device provided by an embodiment of the present invention.

[0050] Icons: 10 - storage device; 11 - processor; 12 - memory; 13 - bus; 14 - communication interface; 20 - client; 30 - data acquisition input terminal; 40 - central dispatch server; 50 - monitoring device; 100 - data storage device; 110 - receiving module; 120 - writing module; 130 - response module; 140 - recovery module. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0055] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0056] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0057] In the field of storage technology, different storage media in storage devices have different write performance and prices. Generally, storage media with good write performance have higher prices, while storage media with poor write performance have lower prices. In addition, different storage media are based on different storage principles, and their write characteristics are also different. For example, solid-state drives can still maintain relatively good performance when processing small write requests, while traditional mechanical hard drives have poor write performance when facing a large number of continuous small write requests. In order to fully utilize the characteristics of different storage media in storage devices to improve the overall write performance of storage devices, the embodiments of the present invention provide a data storage method, apparatus, storage device, and readable storage medium, which will be described in detail below.

[0058] Please refer to Figure 1 , Figure 1 An example diagram of an application scenario is provided for an embodiment of the present invention. Figure 1 In the embodiment, the storage device 10 and the client 20 are in communication connection, and the client 20 sends a write request to the storage device 10 to write the data to be written to the file to be written to the storage device 10. The storage device 10 includes a first-level cache area, a second-level cache area and a storage area. The file to be written and the first-level cache area and the second-level cache area are pre-bound. The storage area is a non-volatile storage medium with the lowest write performance, the second-level cache area is a non-volatile storage medium with low write performance, the first-level cache area has the highest write performance, and the first-level cache area can be a volatile storage medium or a non-volatile storage medium.

[0059] After the storage device 10 receives the data to be written of the file to be written, if the length of the data to be written is not greater than the preset value, it will be written to the primary cache area and the secondary cache area at the same time, and then a response message will be returned to the client 20. If the length of the data to be written is greater than the preset value, it will only be written to the secondary cache area, and then a response message will be returned to the client 20. After that, the data in the primary cache area or the secondary cache area will be stored in the storage area through the background thread. On the one hand, since the write performance of the primary cache area and the secondary cache area is greater than the write performance of the storage area, the write request sent by the client can be responded to in a timely manner, thereby improving the overall write performance of the storage device. On the other hand, the spatially continuous data to be written by multiple requests can be aggregated in the primary cache area and / or the secondary cache area, and the aggregated data will be written back to the mechanical hard disk at one time after reaching a certain amount, so as to further improve the write performance of the storage system.

[0060] Since the higher the performance of the storage medium, the higher its cost, in order to reduce the cost, the size of the first-level cache area can be smaller than the second-level cache area. Of course, the size of the second-level cache area can also be smaller than the storage area.

[0061] As a specific implementation method, the first-level cache area can be a volatile storage medium with a higher write performance, the second-level cache area can be a non-volatile storage medium with a lower write performance, and the storage area can be a non-volatile storage medium with the lowest write performance among the three. For example, the first-level cache area can be a memory with the highest write performance and the smallest storage capacity. After power failure, the data therein will be lost. The second-level cache area can be a solid-state drive with a lower write performance than the memory but higher than the storage area. The storage area can be a mechanical hard disk with the lowest write performance.

[0062] The storage device 10 may be a storage server, a storage array, a storage system, or a storage node in a storage cluster, and is a provider of data storage services.

[0063] The client 20 may be a host, a mobile terminal, a laptop computer, etc., and is a demander of data storage services. The client 20 stores the data to be stored in the storage device 10 by sending a write request to the storage device 10 .

[0064] based on Figure 1 The embodiment of the present invention also provides another specific application scenario example diagram, please refer to Figure 2 , Figure 2 An example diagram of another specific application scenario provided by an embodiment of the present invention, Figure 2 yes Figure 1The storage device in the monitoring storage system is used as an example diagram of a storage service node in the monitoring storage system. The monitoring storage system includes a data acquisition input terminal 30, a central scheduling server 40, a monitoring device 50 and a storage device 10. In the monitoring storage system, the storage device 10 is also called a storage service node. The monitoring device 50 is responsible for monitoring. The data acquisition input terminal 30 collects monitoring data from the monitoring device 50 and flows the collected data into the central scheduling server 40. The central scheduling server 40 makes a decision to select the storage device 10 and flows the data into the selected storage device 10 for storage. The central scheduling server 40 mainly makes the data storage of each storage device 10 balanced. Figure 1 The client 20 in the example may be a central scheduling server 40 .

[0065] exist Figure 1 and Figure 2 Based on this, the embodiment of the present invention also provides Figure 1 and Figure 2 Please refer to the hardware structure diagram of the storage device 10 in FIG. Figure 3 , Figure 3 The hardware structure diagram of the storage device provided in the embodiment of the present invention is as follows: the storage device 10 includes a processor 11, a memory 12, a bus 13, and a communication interface 14; the processor 11, the memory 12, and the communication interface 14 exchange data via the bus 13.

[0066] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the storage device 10. In other embodiments, the storage device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0067] The processor 11 may be an integrated circuit chip having the ability to process signals. In the implementation process, each step of the data storage method may be completed by an integrated logic circuit of hardware in the processor 11 or by instructions in the form of software. The processor 11 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0068] The memory 12 is used to store programs, such as the data storage device in the embodiment of the present invention. The data storage device includes at least one software function module that can be stored in the memory 12 in the form of software or firmware or fixed in the operating system (OS) of the storage device 10. After receiving the execution instruction, the processor 11 executes the program to implement the data storage method disclosed in the above embodiment.

[0069] The storage device 10 communicates with the client 20 via the communication interface 14 .

[0070] In addition, as a specific implementation, the memory 12 also includes a memory for providing a primary cache area, a solid state hard disk for providing a secondary cache area, and a mechanical hard disk for providing a storage area. Figure 3 The fact that something is not specifically identified does not mean that it does not exist.

[0071] The embodiment of the present invention also provides a method for applying Figure 1 to Figure 3 For a data storage method of the storage device 10, please refer to Figure 4 , Figure 4 An example of a data storage method provided by the present invention Figure 1 , the method comprises the following steps:

[0072] Step S100, receiving a write request sent by a client for writing data to be written, wherein the write request includes a file to be written and a length of the data to be written, and the first-level cache area and the second-level cache area are pre-bound to the file to be written.

[0073] In this embodiment, the storage device 10 includes a primary cache area, a secondary cache area, and a storage area. The write performance of the primary cache area and the write performance of the secondary cache area are both greater than the write performance of the storage area. After the data to be written is written into the primary cache area or the secondary cache area, a response message is returned to the client 20, and then a dedicated background thread stores the data to be written in the primary cache area or the secondary cache area to the storage area when the preset disk flushing condition is met. In the scenario of multiple small write requests, the data to be written of multiple small write requests can be merged in the primary cache area or the secondary cache area, and the data to be written of multiple small write requests can be written to the storage area at one time, which can greatly alleviate the problem of slow write performance caused by multiple small write request processing in the storage area.

[0074] In this embodiment, if the file to be written is in a closed state, the file to be written is first opened, and the file to be written is bound to the first-level cache area and the second-level cache area, that is, the relationship between the file to be written and the first-level cache area and the second-level cache area is established. As a specific implementation method, the storage device can support concurrent writing of data to be written to multiple files to be written. In this case, it is necessary to provide corresponding first-level cache areas and second-level cache areas for each file to be written, and bind each file to be written with its respective first-level cache areas and second-level cache areas when opening it. For example, if the storage device supports concurrent writing of 8 files, the storage device reserves 8 first-level cache areas and 8 second-level cache areas. When opening the file to be written, the idle first-level cache areas are determined from the reserved first-level cache areas, and the idle second-level cache areas are determined from the reserved second-level cache areas, and then the file to be written is bound to the determined first-level cache areas and second-level cache areas, and the status of the bound first-level cache areas and second-level cache areas is set to occupied status. When the file to be written is closed, the file to be written and its bound first-level cache areas and second-level cache areas are unbound, and then the status of the unbound first-level cache areas and second-level cache areas is set to idle status.

[0075] Step S101 , if the length to be written is not greater than a preset value, the data to be written is written into the primary cache area and the secondary cache area, so that the data to be written is stored from the primary cache area or the secondary cache area to the storage area.

[0076] In this embodiment, the preset value can be set as needed, for example, the preset value is set to 64KB. As a specific implementation method, the reason why the data to be written is written into the primary cache area and the secondary cache area at the same time is, on the one hand, because the primary cache area is a volatile storage medium and the secondary cache area is a non-volatile storage medium, for application scenarios with high data reliability requirements, when the storage device loses power abnormally, the data that has not been written to the storage area and is still temporarily stored in the secondary cache area can be stored from the secondary cache area to the storage area when the storage device is powered on. On the other hand, the cost of the primary cache area is usually higher than that of the secondary cache area. In order to reduce costs, the size of the primary cache area is usually smaller than that of the secondary cache area. The primary cache area is used to store the data to be written whose length is not greater than the preset value and is the latest written data to be written.

[0077] Step S102: if the length of the data to be written is greater than a preset value, the data to be written is written into the secondary cache area, so that the data to be written is stored from the secondary cache area to the storage area.

[0078] In this embodiment, for a to-be-written length greater than a preset value, since the size of the first-level cache is insufficient to temporarily store all the to-be-written data of this write request at one time, it is directly written into the second-level cache.

[0079] In this embodiment, the data to be written actually includes two asynchronous processes when it is finally stored in the storage area: a process of writing to the secondary cache area and / or the primary cache area; and a process of storing from the secondary cache area or the primary cache area to the storage area; at the end of the former process, the storage device 10 returns a response message of the write request to the client 20, and at the end of the latter process, the data to be written is stored in its final storage area, i.e., the storage area.

[0080] Step S103: Return a response message to the write request to the client.

[0081] In this embodiment, the response message is used to represent the processing result of the write request, and the processing result includes but is not limited to success, failure, timeout, etc.

[0082] In the above method provided by the embodiment of the present invention, the write performance of the first-level cache and the second-level cache are both greater than the write performance of the storage area. Therefore, after the data to be written is written into the first-level cache and / or the second-level cache, the storage device can quickly return a response message to the client, thereby improving the overall write performance of the storage device in the storage system.

[0083] In this embodiment, in order to more efficiently write the data to be written into the primary cache area and / or the secondary cache area, the embodiment of the present invention also provides a specific implementation method for writing into the primary cache area and the secondary cache area. For the convenience of explanation, the mapping relationship between the primary cache area, the secondary cache area and the storage area is first introduced. For the convenience of processing, the primary cache area is divided into multiple cache blocks according to the first preset length, the secondary cache area and the storage area are of the same size and have a one-to-one correspondence, and the secondary cache area and the storage area are respectively divided into multiple cache slices and storage blocks according to the second preset length. The relative offset position of the data in the secondary cache area is the same as the relative offset position in the storage area. For example, if the data to be written is written to an address with an offset position of 512KB in the secondary cache area, when it is stored in the storage area, it is also stored to an address with an offset position of 512KB in the storage area. Therefore, the mapping relationship between the secondary cache area and the storage area is not described in detail, and the first preset length and the second preset length can be set as needed.

[0084] Please refer to Figure 5 , Figure 5 An example diagram of the mapping relationship between the first-level cache area and the second-level cache area provided in an embodiment of the present invention. Figure 5 The first preset length is 64KB, the second preset length is 1MB, the first-level cache area is divided into 1024 cache blocks, each cache block size is 64KB, the second-level cache area is divided into 1024 cache slices, each cache slice size is 1MB, and one cache block corresponds to one cache slice. Figure 5In the example, cache block 1 corresponds to cache slice 1, cache block 2 corresponds to cache slice 2, and so on. A cache block is used to store the latest 64KB of data written in the corresponding cache slice. At this time, the preset value is the size of the first-level cache block, that is, 64KB. That is to say, for any write request, if the length of the data to be written of the write request is not greater than 64KB, it will be written to the first-level cache area and the second-level cache area and then the response request will be returned to the client 20. If the length of the data to be written of the write request is greater than 64KB, it will be written to the second-level cache area and then the response request will be returned to the client 20. It can be understood that based on Figure 5 The storage area is also divided into multiple storage blocks of 1MB.

[0085] based on Figure 5 It is explained that the specific implementation process of writing the data to be written into the first-level cache area and the second-level cache area can be:

[0086] First, the target storage block to be written into the file to be written and the first offset position in the target storage block are obtained.

[0087] In this embodiment, the data to be written is sequentially written into the file to be written. When the file to be written is opened, the end position of the last written data before the file to be written is closed can be obtained by calling the function of opening the file, that is, the starting position of the current to be written. Since the size of the file to be written is preset and the same as the size of the storage area, the target storage block to be written by the data to be written in the multiple storage blocks in the storage area and the first offset position in the target storage block can be determined according to the starting position. For example, the storage area has 1024 1MB storage blocks, numbered 1 to 1024, that is, the size of the storage area is 1GB, that is, the size of the file to be written is 1GB, and the end position of the last written data before the file to be written is 1050MB, then 1050 and 1024 are divided, the quotient is 1, the remainder is 26, the quotient + 1 is the number of the target storage block, and the remainder is the first offset position in the target storage block. It should be noted that the numbering can also start from 0, and the method for determining the numbering of the target storage block is similar to that, which will not be repeated here.

[0088] Secondly, a target cache block corresponding to the target storage block in the first-level cache area is determined.

[0089] In this embodiment, since the cache blocks correspond to the storage blocks one by one, the target cache block corresponding to the target storage block can be determined according to the target storage block.

[0090] Third, determine the target cache slice corresponding to the target storage block in the secondary cache area.

[0091] In this embodiment, since the cache slices correspond to the storage blocks one by one, the target cache slice corresponding to the target storage block can be determined according to the target storage block.

[0092] Finally, the data to be written is written into the target cache block, and the data to be written is written into the target cache slice according to the first offset position.

[0093] In this embodiment, when writing the data to be written into the target cache block, if the size of the remaining space in the target cache block is greater than or equal to the length of the data to be written, the data to be written is directly written into the target cache block in an append write manner, and the data length of the cached data of the target cache block is updated; otherwise, the data in the target cache block is cleared, and the data to be written is then written into the target cache block, and the first offset position of the data cached in the target cache block in the corresponding target storage block is updated at the same time. Please refer to Figure 6 , Figure 6 The example diagram of writing the data to be written into the target cache block provided by the embodiment of the present invention, the data length of the cached data in the target cache block is 25KB, and is cached at the position of 0 to 25KB in the target cache block, such as Figure 6 As shown in the slashed box, the first offset position of the cached data in the target cache block in the corresponding target storage block is 10KB. If the data to be written is 15KB, the data to be written is written from 25KB to 15KB. At this time, the data length of the cached data in the target cache block is 25KB+15KB=40KB; if the data to be written is 50KB, the cache block size is 64KB, and the first offset position of the data to be written in the target storage block is 35KB. At this time, the data in the target cache block needs to be cleared, and then 50KB of the data to be written is written to the position 0-50KB of the target cache block. At this time, the first offset position of the cached data in the target cache block in the corresponding target storage block is 35KB.

[0094] In this embodiment, since the target cache slice and the target storage block are in one-to-one correspondence and the offset position of the data to be written in both is the same, the data to be written can be directly written to the target cache slice according to the first offset position.

[0095] In this embodiment, if the length to be written is greater than the preset value, the method of writing the data to be written into the secondary cache area is the same as the above method, that is:

[0096] First, the target storage block to be written into the file to be written and the first offset position in the target storage block are obtained.

[0097] Secondly, determine the target cache slice corresponding to the target storage block in the secondary cache area.

[0098] Finally, the data to be written is written to the target cache slice according to the first offset position.

[0099] The detailed process has been described above and will not be repeated here.

[0100] The above method provided by the embodiment of the present invention can quickly determine the target cache block and target cache slice corresponding to the target storage block where the data to be written is to be written, so as to quickly write the data to be written into the target cache block and target cache slice and respond to the client 20 in time.

[0101] In this embodiment, in order to timely store the data temporarily stored in the first-level cache area or the second-level cache area into the storage area, the embodiment of the present invention also provides a specific implementation method:

[0102] First, it is determined whether each cache slice in the secondary cache area meets the preset disk flushing condition.

[0103] In this embodiment, since the secondary cache area is only used to temporarily store data, the storage area is the area where the temporarily stored data needs to be stored eventually. In order to release the secondary cache area in time after the writing of the file to be written is completed, so as to be used for other files to be written, the data temporarily stored in the secondary cache area needs to be stored in the storage area in time. The preset flushing condition is used to characterize the timing when the data temporarily stored in the secondary cache area needs to be flushed, that is, the timing when the data temporarily stored in the secondary cache area is stored in the storage area. Satisfying the preset flushing condition includes at least the following three situations:

[0104] (1) For any target cache slice in the secondary cache area, if the length of the data in the target cache slice that has not been flushed to disk is greater than a preset length, it is determined that the target cache slice meets the preset flushing condition.

[0105] In this embodiment, the target cache slice is any cache slice in the secondary cache area, and it can be determined whether it meets the preset flushing condition by traversing all cache slices in the secondary cache area. The unflushed data in the target cache slice is the data in the target cache slice that needs to be stored in the storage area. The unflushed data in the target cache slice can be all or part of the data in the target cache slice. The preset length can be set according to the actual scenario. For example, the preset length is set to 16KB, that is, if the length of the unflushed data in the target cache slice is greater than 16KB, it is determined that the target cache slice meets the preset flushing condition.

[0106] (2) If the file to be written is in the to-be-closed state, it is determined that the cache slices with unflushed data in all cache slices meet the preset flushing conditions.

[0107] In this embodiment, the fact that the file to be written is in the to-be-closed state means that the file to be written is about to be closed. At this time, in order to ensure the security of the data in the file to be written, it is necessary to promptly store the unflushed data in all cache slices in the secondary cache area bound to the file to be written to the storage area. At this time, regardless of whether the length of the unflushed data in the cache slice is greater than the preset length, it will be judged as meeting the preset flushing condition.

[0108] (3) If the current time is within the preset disk flushing cycle, it is determined that the cache slices with unflushed data in all cache slices meet the preset disk flushing conditions.

[0109] In this embodiment, the preset disk flushing cycle is used to represent the period for forcing the unflushed data in the cache slice to be stored in the storage area. The preset disk flushing cycle can be set as needed. For example, the preset disk flushing cycle is set to 1 second, that is, every 1 second, the cache slice with unflushed data is judged as meeting the preset disk flushing conditions, so that the unflushed data therein is stored in the storage area.

[0110] Secondly, the cache slices satisfying the preset disk flushing condition are determined as the cache slices to be flushed, and the cache slices to be flushed contain data that needs to be stored in the storage area.

[0111] It should be noted that if there is no unflushed data in the cache slice, then no matter whether the cache slice is in any of the above three situations, the cache slice will not be determined as a cache slice to be flushed.

[0112] In this embodiment, in order to avoid unnecessary brushing and to improve brushing efficiency as much as possible, the embodiment of the present invention further provides a specific implementation method of brushing, and the steps are as follows:

[0113] First, determine whether the actual data to be flushed in the cache slice to be flushed exists in the primary cache area.

[0114] In this embodiment, the actual data to be flushed in the disk cache may exist in both the first-level cache area and the second-level cache area, or may only exist in the second-level cache area. If it exists in both the first-level cache area and the second-level cache area, in order to store the actual data to be flushed in the storage area more quickly, the data to be flushed is directly read from the first-level cache area and written to the storage area.

[0115] In this embodiment, in order to more quickly determine whether the actual data to be flushed exists in the primary cache area, this embodiment also provides a specific determination method, which is:

[0116] The first step is to obtain the second offset position of the current disk data to be flushed in the storage block corresponding to the disk cache slice to be flushed.

[0117] In this embodiment, the current data to be flushed is the data that currently needs to be stored in the storage area, and the second offset position is the offset position of the current data to be flushed in the storage block corresponding to the cache piece to be flushed. For example, the cache piece to be flushed is piece B, and the storage block corresponding to piece B is block B'. ​​The offset position of the data to be flushed in block B' is 25KB, and the second offset position is 25KB.

[0118] In this embodiment, as a specific implementation method, since the size of the file to be written is predetermined, the size of the storage area corresponding to the file to be written is also predetermined. The second offset position is obtained by recording the position of the current data to be flushed in the file to be written and the number of storage blocks in the storage area corresponding to the file to be written.

[0119] Step 2: Determine the disk cache block to be flushed corresponding to the disk cache slice to be flushed from multiple cache blocks.

[0120] In this embodiment, block data is data cached in a cache block. For each cache block, each cache block includes a first length of the cached block data and an end position of the block data in a storage block corresponding to each cache block. Slice data is data cached in a disk cache slice to be flushed. For each cache slice, each cache slice includes slice data in the disk cache slice to be flushed and a second offset position of the slice data in a storage block corresponding to the disk cache slice to be flushed.

[0121] Step 3: According to the end position of the disk cache block to be flushed and the second offset position, the second length of the actual data to be flushed is obtained.

[0122] In this embodiment, the end position of the disk cache block to be flushed is the end position of the cache data in the disk cache block to be flushed in the storage block corresponding to the disk cache block to be flushed. For example, the disk cache block to be flushed is block A, and the storage block corresponding to block A is block A', wherein the cache data is cached at a position of 0 to 15KB in block A, and the offset position of the cache data in block A' is 50KB, then the end position of the cache data in block A' is 50KB+15KB=65KB, that is, the end position of block A.

[0123] In this embodiment, the second length is the actual length of the data to be flushed, which can be calculated by the following formula: a=ulOffset-ulLastWrite, where a represents the second length, ulOffset represents the end position, and ulLastWrite represents the second offset position.

[0124] Step 4: If the second length is smaller than the first length of the cache block to be flushed, it is determined that the data to be flushed in the cache block to be flushed exists in the first-level cache area.

[0125] In this embodiment, the first length is the length of the valid data temporarily stored in the disk cache block to be flushed, and the range of the first length is: [0, first length]. For example, the first preset length of the disk cache block to be flushed is 64KB, of which the first 30KB temporarily stores data, then the length of its valid data is 30KB, and its first length is 30KB.

[0126] Step 5: If the second length is not less than the first length of the disk cache block to be flushed, it is determined that the disk data to be flushed in the disk cache block to be flushed does not exist in the first-level cache area.

[0127] In this embodiment, the actual data to be flushed can be determined at the same time. If the data to be flushed in the cache block to be flushed exists in the first-level cache area, a represents the second length, ulOffset represents the end position, ulLastWrite represents the second offset position, and ulLenth is the first length, then the actual data to be flushed is the data between ulLenth-a and ulLenth in the cache block to be flushed, and the data between 0 and ulLenth-a in the cache block to be flushed has been flushed before, that is, it has been stored in the storage area. At this time, there is no need to flush the disk again. Please refer to Figure 7 , Figure 7 This is an example diagram of actual disk flushing data provided by an embodiment of the present invention. Figure 7 In the example, since the length between the end position and the second offset position is less than the first preset length, the actual data to be flushed exists in the first-level cache area, and the actual data to be flushed is the data from the second offset position to the end position in the cache block to be flushed. According to the offset address of the cache data in the cache block to be flushed relative to the corresponding storage block, the offset of the actual data to be flushed in the cache block to be flushed can be calculated. If the data to be flushed in the cache piece to be flushed does not exist in the first-level cache area, the data from the second offset position in the cache piece to be flushed to the valid data in the cache piece to be flushed is determined as the actual data to be flushed.

[0128] Secondly, if the actual data to be flushed exists in the first-level cache area, the actual data to be flushed is stored from the first-level cache area to the storage area.

[0129] Finally, if the actual data to be flushed does not exist in the primary cache area, the actual data to be flushed is stored from the secondary cache area to the storage area.

[0130] As a specific implementation method, the embodiment of the present invention also provides a Figure 2 The specific implementation method in the specific application scenario, Figure 2 Each storage device (i.e. storage node) uses the Linux operating system. Each storage node is equipped with 13 8TB mechanical hard disks, which together form a storage pool for storage, i.e. storage area. Each storage node is equipped with an NVMe solid-state drive as a secondary cache area to provide write acceleration. The secondary cache areas between each storage node are independent of each other. Each storage node has 8 processes for storage services, which can be used to write 8 files at the same time. The size of each file is 1GB. It should be noted that the specific implementation process for writing each file is the same.

[0131] The entire implementation process includes the following parts:

[0132] (1) Resource initialization operation

[0133] (1.1) Connect the NVMe SSD to the PCIE slot of the storage node. After the storage node recognizes the SSD, there is no need to format the file system. The NVMe SSD exists as a block device in the system. Go to (1.2).

[0134] (1.2) The first 64G+64*8KB area of ​​the NVMe SSD is allocated to the L2 cache area. The first 8KB area of ​​each L2 cache area (1G+8KB) is flushed to 0, and the formatting is completed.

[0135] (1.3) The eight storage service processes are allocated the secondary cache area in the order of their process IDs. Each storage service process obtains an 8*(1G+8KB) secondary cache area. Each process can support opening eight large files in write cache mode at the same time.

[0136] (1.4) Resource initialization completed.

[0137] (2) Business process initialization

[0138] An exception check will be performed during initialization. Since the write cache is configured for the first time, there is no abnormal exit and the recovery process will not be entered here.

[0139] (3) Open the large file file1

[0140] Get a free cache block to bind the large file file1. The specific process is as follows:

[0141] (3.1) Call the IO Open interface to open the large file file1 in the storage layer (mechanical hard disk), and record its fd as fd1, where fd is the file descriptor (file descriptor, fd). The file can be read and written through fd.

[0142] In this embodiment, the user can decide whether to enable the data storage method of the embodiment of the present invention according to his own needs to achieve good compatibility with the existing data storage method. If it is not enabled, when opening the file file1, the fd1 corresponding to file1 is directly returned. If it is enabled, it is checked whether there is a duplicate fd1. If so, the opening of file1 fails. If not, continue (3.2).

[0143] (3.2) Query the cache block resources recorded in the memory (a total of 8 blocks, i.e., 8 first-level cache areas), and obtain that the first-level cache area No. 0 is free.

[0144] (3.3) Initialize the exception recovery related information of the first-level cache area 0.

[0145] In this embodiment, the abnormal recovery related information includes, but is not limited to, the path of file1, a check area mark used to indicate that there is a write operation in the first-level cache area 0, and the like.

[0146] (3.4) Mark the first-level cache block 0 as non-free, and record the binding relationship between fd1 and cache block 0 in the memory, and then open the large file file1 to end.

[0147] (4) Writing data

[0148] (4.1) Receive a write request and write data to file1 through fd1.

[0149] (4.2) Get the first-level cache area bound to fd1, which is the first-level cache area 0

[0150] (4.3) Obtain the cache slice of the secondary cache area that needs to be written based on the write offset / 1MB calculation.

[0151] (4.4) If the cache slice does not overflow in the cache block of the corresponding first-level cache area, the data is directly appended to the corresponding cache block and simultaneously written to the corresponding position of the corresponding cache slice in the second-level cache area.

[0152] (4.5) Update the information of the cache block in the first-level cache area, that is, the current write position (that is, the offset position corresponding to the storage area) and the used length of the cache block (that is, the first length).

[0153] (4.6) A write success response is returned for the write request.

[0154] (5) Background writeback

[0155] (5.1) The inspection thread traverses the cache slices of each L2 cache area and obtains that the length of the data to be flushed in cache slice 0 of L2 cache area 0 exceeds 16KB.

[0156] (5.2) The information of cache slice 0 is transferred to the disk flushing queue and marked as being flushed.

[0157] (5.3) The write-back thread obtains cache slice 0 from the disk flushing queue, and based on the information of cache slice 0, determines that the data to be written back (i.e., the data to be flushed to disk) of cache slice 0 exists in the first-level cache area, flushes the data to be written back into the corresponding mechanical hard disk area (i.e., the storage area) of file1, and clears the flag in the disk flushing of cache slice 0.

[0158] (5.4) This round of write-back is completed and waiting for the next round of inspection.

[0159] (6) Close the file

[0160] Closing a file is divided into two stages. In the first stage, the corresponding cache slice is marked as to be closed, and a response message indicating that the closing is completed is returned to the client 20. In the second stage, the file in the to-be-closed state is processed in the inspection thread.

[0161] The first-stage processing flow is as follows:

[0162] (6.1.1) Call the close interface to close fd1.

[0163] (6.1.2) Get the L2 cache area bound to fd1, which is L2 cache area 0.

[0164] (6.1.3) Mark L2 cache area 0 as pending closure.

[0165] (6.1.4) Return the closure success to the client 20.

[0166] The second-stage processing flow is as follows:

[0167] (6.2.1) The inspection thread traverses each L2 cache area and detects that L2 cache area 0 is in a waiting-to-be-closed state.

[0168] (6.2.2) Traverse each cache slice in the secondary cache area 0 and find that there is still data in cache slices 1 and 3 that has not been written back (ie, not flushed to disk).

[0169] (6.2.3) The information of cache slices 1 and 3 is put into the disk flushing queue, and the disk flushing thread writes it back and waits for the next round of inspection.

[0170] (6.2.4) Entering the next round of inspection, it is found that all cache slices in the secondary cache area 0 no longer have unwritten back data.

[0171] (6.2.5) Call the io close function to close fd1 at the storage layer (mechanical hard disk).

[0172] (6.2.6) Clear the exception recovery related information used for exception recovery in the secondary cache area 0.

[0173] (6.2.7) Mark L2 cache area 0 as free.

[0174] (6.2.8) Closing the file is complete.

[0175] In this embodiment, in order to facilitate the description of the interaction between the inspection thread and the write-back thread, the embodiment of the present invention also provides an example of the overall implementation of the inspection thread and the write-back thread, please refer to Figure 8 , Figure 8 An example diagram of the overall implementation of the inspection thread and the write-back thread provided in the embodiment of the present invention, Figure 8In the process, the inspection thread periodically inspects the secondary cache areas 0 to 7, including the following steps: periodically traverse the secondary cache areas 0 to 7, and for any secondary cache area i, determine whether the secondary cache area i is in a waiting-to-be-closed state; if it is not in a waiting-to-be-closed state, determine whether the secondary cache area i has cache slices to be flushed to disk, if not, close the file, and make a judgment on the secondary cache area i+1; if yes, traverse each cache slice in the secondary cache area, if cache slice j is being flushed to disk, make a judgment on the next cache slice, otherwise, continue to judge whether the cache slice meets the flushing condition, if yes, put it into the flushing queue, set the flushing flag, if not, make a judgment on the next cache slice, until all cache slices are traversed. If it is in a waiting-to-be-closed state, determine whether there are cache slices to be flushed to disk, if not, close the corresponding file, set the corresponding secondary cache area to idle, if yes, execute the process of no cache slices to be flushed to disk.

[0176] The write-back thread obtains the cache slices to be flushed from the disk-to-be-flushed queue in sequence. For the current cache slice to be flushed, if the data to be flushed is in the first-level cache area, the data is stored from the first-level cache area to the storage area. If the data to be flushed is not in the first-level cache area, the data is read from the second-level cache area and stored in the storage area. After the data to be flushed in the current cache slice to be flushed is flushed, the current disk position to be flushed is updated, the flush flag of the current cache slice to be flushed is cleared, and the next cache slice to be flushed is obtained from the disk-to-be-flushed queue.

[0177] It should be noted that there can be 8 inspection threads and 8 write-back threads respectively, so as to concurrently and independently process the disk flushing process of 8 files to be written, and minimize the impact of disk flushing on the overall performance of the storage device.

[0178] In this embodiment, when a file to be written is abnormally closed, in order to ensure that the data in the abnormally closed file to be written is accurate and reliable when it is opened again, this embodiment also provides a data recovery method, specifically:

[0179] First, when opening a file to be written that was abnormally closed, a cache slice with valid data in the secondary cache area is used as a cache slice to be restored.

[0180] In this embodiment, the file to be written may be abnormally closed due to a write error. The write error may be caused by a failure in the storage device network or an abnormality in the area to be written. At this time, there may be data in the secondary cache area bound to the file to be written that has not yet been stored in the storage area. In order to ensure the reliability of the data in the file to be written, when the abnormally closed file to be written is opened again, the data in the secondary cache area that has not yet been stored in the storage area needs to be stored in the storage area.

[0181] In this embodiment, the existence of valid data in the cache slice means that the cache slice contains data that needs to be stored in the storage area, and the cache slice containing valid data is the cache slice to be restored.

[0182] For ease of explanation, the present invention first introduces an example of the layout of the secondary cache area. Fig. 9 , Fig. 9 An example diagram of the layout of the secondary cache area provided in an embodiment of the present invention, Fig. 9 In the , the secondary cache area includes a check block (Checkblock) (4KB), a bitmap (Point map) (4KB) and a data cache area (Idx data) (1GB). The check block also includes a magic field (Recover), a path length field (Path len), a file path field (File Path) and a reserved field (Notused). Among them, the magic field is used for abnormal recovery. When applying for a secondary cache area for a large file, a string of magic numbers will be first brushed on its magic field to mark that the secondary cache area is in use. The magic field is cleared when the large file is closed. Therefore, when the inspection process is initialized, the magic field can be checked to determine whether the file is abnormally closed. If the preset magic number data is stored in the , it indicates that an exception occurs when the file to be written is closed. The file length field is used to record the length of the file to be written, and the file path field is used to record the path of the file to be written. The bitmap includes 1024 4B, each 4B corresponds to a cache slice, and each cache slice is 1MB. When the file to be written is formatted, the bitmap is all set to 1, that is, the initial value of the bitmap of each cache slice is 0xFF. When a cache slice is written for the first time, the offset position of its first write is recorded in the bitmap corresponding to the cache slice. When marked as the initial value, it indicates that there is no data written to the corresponding cache slice, and no recovery or disk flushing is required, otherwise the corresponding cache slice needs to be written back from the starting position recorded by the bitmap. For example, if the starting offset recorded in the bitmap is 512KB, the data from 512KB to 1M in the cache slice needs to be written back when the corresponding segment is restored.

[0183] As a specific implementation manner, when an abnormally closed to-be-written file is opened, it is possible to traverse the bitmap to determine whether a cache slice corresponding to each bitmap is a cache slice to be restored.

[0184] Secondly, the valid data in the cache slice to be restored is stored in the storage block corresponding to the cache slice to be restored in the storage area.

[0185] In order to execute the corresponding steps in the above embodiment and various possible implementations, an implementation method of the data storage device 100 is provided below. Fig.10 , Fig.10It is a block diagram of a data storage device 100 provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the data storage device 100 provided in this embodiment are the same as those of the above embodiments, and are not mentioned in this embodiment for the sake of brief description.

[0186] The data storage device 100 includes a receiving module 110 , a writing module 120 , a responding module 130 and a restoring module 140 .

[0187] The receiving module 110 is used to receive a write request sent by the client for writing the data to be written, wherein the write request includes the file to be written and the length of the data to be written, and the first-level cache area and the second-level cache area are pre-bound with the file to be written.

[0188] The writing module 120 is used to write the data to be written into the first-level cache area and the second-level cache area if the length to be written is not greater than a preset value, so that the data to be written is stored from the first-level cache area or the second-level cache area to the storage area.

[0189] Specifically, the first-level cache area is divided into multiple cache blocks according to a first preset length, the second-level cache area is divided into multiple cache slices according to a second preset length, and the storage area is divided into multiple storage blocks according to the second preset length. The cache blocks correspond to the cache slices one-to-one, and the cache slices correspond to the storage blocks one-to-one. The write module 120 is specifically used to: obtain the target storage block to be written by the file to be written and the first offset position in the target storage block; determine the target cache block corresponding to the target storage block in the first-level cache area; determine the target cache slice corresponding to the target storage block in the second-level cache area; write the data to be written to the target cache block, and write the data to be written to the target cache slice according to the first offset position.

[0190] The writing module 120 is further configured to write the data to be written into the secondary cache area if the length of the data to be written is greater than the preset value, so that the data to be written is stored from the secondary cache area to the storage area.

[0191] Specifically, the secondary cache area is divided into multiple cache slices according to a second preset length, and the write module 120 is specifically used to: determine whether each of the cache slices in the secondary cache area meets the preset disk flushing condition; determine the cache slices that meet the preset disk flushing condition as cache slices to be flushed, and the cache slices to be flushed contain data that needs to be stored in the storage area.

[0192] Specifically, when the write module 120 determines whether each cache slice in the secondary cache area meets the preset disk flushing condition, it is specifically used to: for any target cache slice in the secondary cache area, if the length of data that has not been flushed to disk in the target cache slice is greater than the preset length, then the target cache slice is determined to meet the preset disk flushing condition; if the file to be written is in a to-be-closed state, then all the cache slices with data that have not been flushed to disk are determined to meet the preset disk flushing condition; if the current time is within the preset disk flushing cycle, then all the cache slices with data that have not been flushed to disk are determined to meet the preset disk flushing condition.

[0193] Specifically, the write module 120 is also used to: determine whether the actual data to be flushed in the cache slice to be flushed exists in the first-level cache area; if the actual data to be flushed exists in the first-level cache area, the actual data to be flushed is stored from the first-level cache area to the storage area; if the actual data to be flushed does not exist in the first-level cache area, the actual data to be flushed is stored from the second-level cache area to the storage area.

[0194] Specifically, the first-level cache area is divided into multiple cache blocks according to a first preset length, each of the cache blocks includes a first length of the cached block data and an end position of the block data in a storage block corresponding to each of the cache blocks, and the write module 120 is specifically used to determine whether the actual data to be flushed in the cache piece to be flushed exists in the first-level cache area: obtaining a second offset position of the current data to be flushed in the storage block corresponding to the cache piece to be flushed; determining the cache block to be flushed corresponding to the cache piece to be flushed from the multiple cache blocks; obtaining a second length of the actual data to be flushed according to the end position of the cache block to be flushed and the second offset position; if the second length is less than the first length of the cache block to be flushed, determining that the data to be flushed in the cache piece to be flushed exists in the first-level cache area; if the second length is not less than the first length of the cache block to be flushed, determining that the data to be flushed in the cache piece to be flushed does not exist in the first-level cache area.

[0195] The response module 130 is used to return a response message to the client for the write request.

[0196] Specifically, the secondary cache area is divided into multiple cache slices according to a second preset length, and the storage area is divided into multiple storage blocks according to the second preset length. The cache slices correspond to the storage blocks one by one. The recovery module 140 is used to: when opening the abnormally closed file to be written, use the cache slice with valid data in the secondary cache area as the cache slice to be recovered; store the valid data in the cache slice to be recovered in the storage block in the storage area corresponding to the cache slice to be recovered.

[0197] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data storage method as described above is implemented.

[0198] In summary, an embodiment of the present invention provides a data storage method, apparatus, storage device and readable storage medium, which are applied to a storage device, the storage device is communicatively connected to a client, the storage device includes a first-level cache area, a second-level cache area and a storage area, the write performance of the first-level cache area and the write performance of the second-level cache area are both higher than the write performance of the storage area, the method includes: receiving a write request sent by a client for writing data to be written, wherein the write request includes a file to be written and a length of the data to be written, and the first-level cache area and the second-level cache area are pre-bound to the file to be written; if the length to be written is not greater than a preset value, the data to be written is written into the first-level cache area and the second-level cache area, so that the data to be written is stored from the first-level cache area or the second-level cache area to the storage area; if the length to be written is greater than the preset value, the data to be written is written into the second-level cache area, so that the data to be written is stored from the second-level cache area to the storage area; and a response message to the write request is returned to the client. Compared with the prior art, when the client needs to write data to be written to the storage device, it sends a write request to the storage device. If the length of the data to be written is greater than the preset value, the data to be written is written to the first-level cache. If the length of the data to be written is not greater than the preset value, the data to be written is written to the first-level cache and the second-level cache, and then returned to the client. The data in the first-level cache and the second-level cache are finally stored in the storage area. Since the write performance of the first-level cache and the second-level cache are greater than the write performance of the storage area, the storage device can quickly return a response message to the client, thereby improving the overall write performance of the storage device in the storage system.

[0199] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data storage method, characterized in that: Applied to a storage device, the storage device is connected to a client for communication, the storage device includes a primary cache area, a secondary cache area, and a storage area, the write performance of the primary cache area and the write performance of the secondary cache area are both higher than the write performance of the storage area, the method includes: receiving a write request sent by the client for writing data to be written, wherein the write request includes a file to be written and a length of the data to be written, the first-level cache area and the second-level cache area are pre-bound to the file to be written, and the second-level cache area is divided into a plurality of cache slices according to a second preset length; If the length to be written is not greater than a preset value, writing the data to be written into the first-level cache area and the second-level cache area, so that the data to be written is stored from the first-level cache area or the second-level cache area to the storage area; If the length to be written is greater than the preset value, writing the data to be written into the secondary cache area, so that the data to be written is stored from the secondary cache area to the storage area; Returning a response message to the write request to the client; The method further comprises: Determine whether each of the cache slices in the secondary cache area meets a preset disk flushing condition; Determine a cache slice that meets the preset disk flushing condition as a cache slice to be flushed, wherein the cache slice to be flushed contains data that needs to be stored in the storage area; Determine whether the actual data to be flushed in the to-be-flushed disk cache slice exists in the first-level cache area, the first-level cache area is divided into a plurality of cache blocks according to a first preset length, each of the cache blocks includes a first length of the cached block data and an end position of the block data in a storage block corresponding to each of the cache blocks; If the actual data to be flushed exists in the first-level cache area, storing the actual data to be flushed from the first-level cache area to the storage area; If the actual data to be flushed does not exist in the primary cache area, storing the actual data to be flushed from the secondary cache area to the storage area; The step of determining whether the actual data to be flushed in the cache slice to be flushed exists in the primary cache area comprises: Obtaining a second offset position of the current disk-to-be-flushed data in a storage block corresponding to the disk-to-be-flushed cache slice; Determine a to-be-flushed disk cache block corresponding to the to-be-flushed disk cache slice from the multiple cache blocks; Obtaining a second length of actual data to be flushed according to the end position of the cache block to be flushed and the second offset position; If the second length is smaller than the first length of the to-be-flushed cache block, it is determined that the to-be-flushed data in the to-be-flushed cache slice exists in the first-level cache area; If the second length is not less than the first length of the cache block to be flushed, it is determined that the data to be flushed in the cache block to be flushed does not exist in the first-level cache area.

2. The data storage method according to claim 1, characterized in that: The first-level cache area is divided into a plurality of cache blocks according to a first preset length, the second-level cache area is divided into a plurality of cache slices according to a second preset length, the storage area is divided into a plurality of storage blocks according to the second preset length, the cache blocks correspond to the cache slices one-to-one, and the cache slices correspond to the storage blocks one-to-one, and the step of writing the to-be-written data into the first-level cache area and the second-level cache area includes: Obtain a target storage block to be written to the file to be written and a first offset position in the target storage block; Determine a target cache block in the first-level cache area corresponding to the target storage block; Determine a target cache slice in the secondary cache area corresponding to the target storage block; The data to be written is written into the target cache block, and the data to be written is written into the target cache slice according to the first offset position.

3. The data storage method according to claim 1, characterized in that: The step of determining whether each cache slice in the secondary cache area meets the preset disk flushing condition comprises: For any target cache slice in the secondary cache area, if the length of data in the target cache slice that has not been flushed to disk is greater than a preset length, it is determined that the target cache slice meets the preset flushing condition; If the file to be written is in a to-be-closed state, determining that the cache slices with unflushed data in all the cache slices meet the preset flushing condition; If the current time is within the preset disk flushing cycle, it is determined that the cache slices with unflushed data in all the cache slices meet the preset disk flushing condition.

4. The data storage method according to claim 1, characterized in that: The secondary cache area is divided into a plurality of cache slices according to a second preset length, the storage area is divided into a plurality of storage blocks according to the second preset length, the cache slices correspond to the storage blocks one by one, and the method further includes: When opening the abnormally closed to-be-written file, taking the cache slice with valid data in the secondary cache area as the cache slice to be restored; The valid data in the cache slice to be restored is stored in a storage block in the storage area corresponding to the cache slice to be restored.

5. A data storage device, characterized in that: Applied to a storage device, the storage device is connected to a client for communication, the storage device includes a primary cache area, a secondary cache area and a storage area, the write performance of the primary cache area and the write performance of the secondary cache area are both higher than the write performance of the storage area, the device includes: a receiving module, configured to receive a write request sent by the client for writing data to be written, wherein the write request includes a file to be written and a length to be written of the data to be written, the first-level cache area and the second-level cache area are pre-bound to the file to be written, and the second-level cache area is divided into a plurality of cache slices according to a second preset length; A writing module, used for writing the data to be written into the first-level cache area and the second-level cache area if the length to be written is not greater than a preset value, so that the data to be written is stored from the first-level cache area or the second-level cache area to the storage area; The writing module is further used for writing the data to be written into the secondary cache area if the length to be written is greater than the preset value, so that the data to be written is stored from the secondary cache area to the storage area; A response module, used for returning a response message to the client for the write request; The writing module is also used for: Determine whether each of the cache slices in the secondary cache area meets a preset disk flushing condition; Determine a cache slice that meets the preset disk flushing condition as a cache slice to be flushed, wherein the cache slice to be flushed contains data that needs to be stored in the storage area; Determine whether the actual data to be flushed in the to-be-flushed disk cache slice exists in the first-level cache area, the first-level cache area is divided into a plurality of cache blocks according to a first preset length, each of the cache blocks includes a first length of the cached block data and an end position of the block data in a storage block corresponding to each of the cache blocks; If the actual data to be flushed exists in the first-level cache area, storing the actual data to be flushed from the first-level cache area to the storage area; If the actual data to be flushed does not exist in the primary cache area, storing the actual data to be flushed from the secondary cache area to the storage area; When the writing module is used to determine whether the actual data to be flushed in the to-be-flushed disk cache slice exists in the first-level cache area, it is specifically used to: Obtaining a second offset position of the current disk-to-be-flushed data in a storage block corresponding to the disk-to-be-flushed cache slice; Determine a to-be-flushed disk cache block corresponding to the to-be-flushed disk cache slice from the multiple cache blocks; Obtaining a second length of actual data to be flushed according to the end position of the cache block to be flushed and the second offset position; If the second length is smaller than the first length of the to-be-flushed cache block, it is determined that the to-be-flushed data in the to-be-flushed cache slice exists in the first-level cache area; If the second length is not less than the first length of the cache block to be flushed, it is determined that the data to be flushed in the cache block to be flushed does not exist in the first-level cache area.

6. A storage device, characterized in that: It comprises a processor and a memory; the memory is used to store a program; the processor is used to implement the data storage method as described in any one of claims 1 to 4 when executing the program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 4 is implemented.

Images