Data read-write request processing method, electronic equipment, storage medium and program product

Abstract

The invention discloses a data read-write request processing method, electronic equipment, a storage medium and a program product, and relates to the technical field of redundant arrays of independent disks, and the data read-write request processing method comprises the following steps: if the category of a data read-write request is a random read request, if target data corresponding to the random read request does not exist in a target cache, executing the random read request; the random read request is directly issued to a corresponding redundant array of independent disks for processing, and pre-read processing is not carried out; if the category of the data read-write request is the random write request, the random write request is stored by using the random write cache and the random write backup cache, and when the proportion of the random read-write request is less than the preset proportion threshold, the random write request is issued to the corresponding redundant array of independent disks for processing, and the random write request does not need to be merged; and the number of the random write requests issued in a centralized manner is reduced, the technical problem that the performance of the storage system is reduced by a mode of processing the random read-write requests in related technologies is solved, and the technical effect of improving the performance of the storage system is achieved.

Description

Technical Field

[0001] This application relates to the technical field of redundant arrays of independent disks, and in particular to a method for processing data read and write requests, an electronic device, a storage medium, and a program product. Background Art

[0002] In a cloud computing data center, data storage performance is crucial. The redundant array of independent disks (RAID) technology combines multiple physical disks into a logical disk, improving the input / output (IO) performance of the storage system. By providing parity redundancy technology, it enhances the data security of the storage system, but at the same time introduces the problem of write amplification, which affects the performance of the storage system.

[0003] To optimize the performance of the storage system, related technologies adopt the method of establishing a cache module on top of RAID, using the cache module to cache IO data, prefetching IO data, and merging IO requests to process sequential IO requests. However, for random IO requests, due to the lack of locality and sequentiality of random IO requests, using this method for processing actually reduces the performance of the storage system. Summary of the Invention

[0004] This application provides a method for processing data read and write requests, an electronic device, a storage medium, and a program product, to at least solve the problem that the method of processing random read and write requests in related technologies reduces the performance of the storage system.

[0005] This application provides a method for processing data read and write requests, including: Receiving a data read and write request and determining the category of the data read and write request; If the category of the data read and write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, sending the random read request to the corresponding redundant array of independent disks for processing; If the category of the data read and write request is a random write request, sending the random write request to the random write cache, and when the memory amount occupied by the random write cache exceeds the first memory amount threshold, sending a preset number of random write requests in the random write cache to the random write backup cache; When the proportion of random read and write requests in the received data read and write requests is less than the preset proportion threshold within the first preset time period, sending the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing.

[0006] The present application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above data read / write request processing methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any of the above data read / write request processing methods when executed by a processor.

[0008] The present application also provides a computer program product including a computer program, which implements the steps of any of the above data read / write request processing methods when executed by a processor.

[0009] Through the present application, since if the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is directly sent to the corresponding redundant array of independent disks for processing without prefetching; if the category of the data read / write request is a random write request, the random write cache and the random write backup cache are used to store the random write request. When the ratio of random read / write requests is less than the preset ratio threshold, the random write request is sent to the corresponding redundant array of independent disks for processing without merging the random write requests, and the number of centralized random write requests is reduced, solving the technical problem that the method of processing random read / write requests in the related art reduces the performance of the storage system, and achieving the technical effect of improving the performance of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 It is a schematic diagram of data striping of RAID5 provided by an embodiment of the present application; Figure 2 It is a schematic structural diagram of a data read / write request processing system provided by an embodiment of the present application; Figure 3 It is a schematic flowchart of a data read / write request processing method provided by an embodiment of the present application; Figure 4 It is a schematic structural diagram of a storage system provided by an embodiment of the present application; Figure 5 It is a schematic flowchart of another data read / write request processing method provided by an embodiment of the present application; Figure 6Schematic diagram of the first hash table corresponding to the RAID with RAID ID 0 provided by the embodiments of the present application; Figure 7 Schematic diagram of the first linked list corresponding to the RAID with RAID ID 0 provided by the embodiments of the present application; Figure 8 Schematic diagram of the random write backup cache storing random write logs provided by the embodiments of the present application; Figure 9 Schematic diagram of the structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0012] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0013] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0015] In a cloud computing data center, data storage performance is crucial. With the continuous improvement of hardware performance, optimizing the software layer performance of the storage system has become the key to breaking through the overall system performance bottleneck.

[0016] As the core solution of the storage system, the RAID technology combines multiple independent physical disks into a logical disk in different ways, providing higher reliability and storage performance than a single disk. Through striping technology, the blocks on multiple physical disks are divided into stripes, and the IO operations are converted into parallel operations on multiple physical disks, improving the IO performance of the storage system. The RAID technology also provides a parity redundancy technology. By calculating the parity block from the data in the data blocks in the stripe, if a data block is damaged, the damaged data block can be recovered through the parity block, improving the data security of the storage system.

[0017] However, while the parity redundancy technology improves the data security of the storage system, it also brings factors that affect the performance of RAID, such as write amplification, that is, factors that affect the performance of the storage system. Especially when updating random small-block data, the parity block must be updated synchronously, resulting in a very large write amplification for small-block data. Exemplarily, Figure 1 is a schematic diagram of data striping of RAID5 provided by an embodiment of the present application, as Figure 1 shown, RAID5 combines 4 physical disks into a logical disk, that is, a RAID. Among them, each physical disk is sliced into disk blocks of the same size according to a fixed size, such as data block D1, data block D2, data block D3, and parity block P. Disk blocks with the same number on each physical disk form a stripe. For example, the stripe with stripe number 0 includes data block D1 of physical disk 1, data block D2 of physical disk 2, data block D3 of physical disk 3, and parity block P of physical disk 4. For the composition of stripe 1, stripe 2, and stripe 3, refer to Figure 1 , and the detailed description is similar to that of stripe 0 and will not be elaborated here.

[0018] For example, to update the D3 data block of stripe 0, that is, D3 new , it is necessary to first read out the old data of D3, that is, D3 old , calculate the intermediate parity value with the new data and the old data, that is, Delta = D3 new XOR D3 old . Then read out the old parity data, that is, P old , and calculate the new parity data, that is, P new = Delta XOR P old , and then write the new D3 new and P new to the disk. There are four disk operations in the whole process, and the write amplification is 4 times. If the RAID type is RAID6, two disk operations for the additional parity data Q are added, then the write amplification is 6 times. Such a high write amplification seriously affects the performance of RAID. The disk operations of RAID become the bottleneck of the storage system and may cause an abnormally high IO latency at a certain moment, resulting in problems such as system service interruption or even collapse in a short time.

[0019] In order to optimize the performance of the storage system, the related technology adopts the method of establishing a cache module on the upper layer of RAID, using the cache module to cache IO data, pre-read the IO data, and merge the IO requests to process sequential IO requests, reduce the write amplification of RAID, and optimize the performance of RAID and the storage system.

[0020] For sequential read IO requests, a pre-read strategy is adopted, that is, the read data blocks are predicted in advance and read into the cache to optimize the latency of the read IO requests. For sequential write IO requests, they can be returned after being stored in the cache to reduce write latency, and dirty data (data that has not been written to the disk) is regularly flushed to the disk. At the same time, read IO requests or write IO requests for the same stripe are merged as much as possible to reduce disk operations. This greatly reduces the write amplification of RAID and improves system performance.

[0021] The cache design in the related art is based on the principle of locality of disk data access, that is, after the data on the disk is accessed, its adjacent data will also be accessed within a short period of time. This is called IO locality or IO sequentiality, and this type of IO request can also be called a sequential IO request. However, with the rapid development of the Internet, the number of concurrent IO operations performed by different users on the network is increasing, and different IO data of different users are unrelated to each other. There are more and more access scenarios for different small data by the same user. Many of these IOs do not have locality and sequentiality, and this type of IO request can be called a random IO request. In a certain scenario or a certain period of time, the proportion of random IO requests in the storage system is very large. For random IO requests, the related art adopts the same processing method as sequential IO requests, that is, merging, pre-reading and other strategies for processing. This method not only does not optimize the performance of the storage system, but because it pre-reads a large amount of invalid data, the IO request merging and waiting increases the retention time of the IO request, which greatly reduces the performance of the storage system.

[0022] There is also a situation where random write IO requests are issued in a concentrated manner, which causes the RAID write amplification to be very large and may cause abnormally high IO latency at a certain moment, resulting in system service interruption or even crash in a short period of time.

[0023] To solve the above problems, an embodiment of the present application provides a method for processing data read / write requests. The method includes: receiving a data read / write request and determining the category of the data read / write request; if the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, sending the random read request to the corresponding redundant array of independent disks (RAID) for processing; if the category of the data read / write request is a random write request, sending the random write request to the random write cache, and when the memory amount occupied by the random write cache exceeds the first memory amount threshold, sending a preset number of random write requests in the random write cache to the random write backup cache; within the first preset time period, when the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold, sending the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing. In the method provided by the above solution, if the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, directly sending the random read request to the corresponding redundant array of independent disks for processing without pre-reading; if the category of the data read / write request is a random write request, using the random write cache and the random write backup cache to store the random write requests, and when the proportion of random read / write requests is less than the preset proportion threshold, sending the random write requests to the corresponding redundant array of independent disks for processing, without the need to merge the random write requests, and reducing the data volume of centralized sending of random write requests, thus solving the technical problem that the method of processing random read / write requests in the related art reduces the performance of the storage system, and achieving the technical effect of improving the performance of the storage system.

[0024] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the data read / write request processing method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0025] The data read / write request processing method, electronic device, storage medium and program product provided by the embodiment of the present application are applicable to processing data read / write requests. For example Figure 2As shown in the figure, it is a schematic structural diagram of the data read / write request processing system based on the embodiments of the present application, mainly including a server and a storage system. Among them, the server is used to issue data read / write requests, and the storage system is used to receive data read / write requests and determine the category of data read / write requests; if the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is issued to the corresponding redundant array of independent disks (RAID) for processing; if the category of the data read / write request is a random write request, the random write request is issued to the random write cache, and when the memory amount occupied by the random write cache exceeds the first memory amount threshold, a preset number of random write requests in the random write cache are issued to the random write backup cache; within the first preset time period, when the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold, the random write requests in the random write cache and the random write backup cache are issued to the corresponding redundant array of independent disks (RAID) for processing.

[0026] An embodiment of the present application provides a method for processing data read / write requests, which is applied to a storage system. Figure 3 It is a schematic flowchart of the data read / write request processing method provided by the embodiments of the present application. As Figure 3 shown, the process includes the following steps: Step S301: Receive a data read / write request and determine the category of the data read / write request.

[0027] Figure 4 It is a schematic structural diagram of the storage system provided by the embodiments of the present application. As Figure 4 shown, the storage system includes a request classification module, a cache array, and a RAID module. Among them, the request classification module is used to classify the received data read / write requests, that is, I / O requests, and issue the data read / write requests to the corresponding caches in the cache array according to the category of the data read / write requests.

[0028] The cache array includes a sequential read / write cache, a random write cache, and a random write backup cache. The RAID module is used to process the data read / write requests issued by the cache array.

[0029] Step S302: If the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is issued to the corresponding redundant array of independent disks (RAID) for processing.

[0030] Among them, the target cache includes a sequential read / write cache and a random write cache.

[0031] If the target data corresponding to the random read request does not exist in the target cache, it means that the target cache misses, and the random read request needs to be sent to the RAID module for processing without any caching and processing operations. It can be understood that the RAID module includes multiple RAIDs, and the random read request is sent to the corresponding RAID based on the metadata of the random read request. Among them, the metadata of the random read request includes the RAID identifier (ID) of the RAID corresponding to the random read request.

[0032] It is understandable that, for a random read request, no cache is performed on the target data of the random read request, and no data is pre-read into the target cache.

[0033] Step S303: If the category of the data read and write request is a random write request, the random write request is sent to the random write cache. When the amount of memory occupied by the random write cache exceeds the first memory amount threshold, a preset number of random write requests in the random write cache are sent to the random write backup cache.

[0034] The random write cache is established in the memory. The first memory amount threshold and the preset amount are set by the technicians and are not specifically limited here. It is understandable that a preset amount of memory can be allocated to the random write cache in advance, and the first memory amount threshold can be a preset amount of memory pre-allocated to the random write cache.

[0035] Step S304: within the first preset time period, if the proportion of random read and write requests in the received data read and write requests is less than the preset proportion threshold, the random write requests in the random write cache and the random write backup cache are sent to the corresponding independent disk redundant array for processing.

[0036] The random read and write request includes a random read request and a random write request. The preset ratio threshold is set by a technician and is not specifically limited here. For example, the preset ratio threshold is 10%. The first preset time period is set by a technician and is not specifically limited here. The first preset time period is a preset time period before the current moment and is updated as the current moment is updated.

[0037] Since random read and write requests are more frequent only at a certain stage, and sequential read and write requests are the mainstream in general, when the proportion of random read and write requests received is less than 10% within the first preset time period, the random write requests in the random write cache and the random write backup cache are sent to the corresponding independent disk redundant array for processing.

[0038] The random write cache and the random write backup cache periodically query the ratio of random read / write requests of the request classification module within the first preset time period. When the ratio of random read / write requests is less than 10%, the random write cache batches and issues the random write requests therein. The random write backup cache reads the random write logs in the random write backup cache from front to back in units of stripes, parses the content of the random write logs, and batches and issues the random write requests corresponding to the random write logs to the corresponding RAID.

[0039] Within the first preset time period, when the ratio of sequential read / write requests in the received data read / write requests is less than the preset ratio threshold, the step of issuing the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing is suspended until within the first preset time period, the ratio of random read / write requests in the received data read / write requests is less than the preset ratio threshold again.

[0040] The data read / write request processing method provided by the embodiments of the present application, if the category of the data read / write request is a random read request, directly issues the random read request to the corresponding redundant array of independent disks for processing without pre-reading processing when the target data corresponding to the random read request does not exist in the target cache; if the category of the data read / write request is a random write request, uses the random write cache and the random write backup cache to store the random write requests, and when the ratio of random read / write requests is less than the preset ratio threshold, issues the random write requests to the corresponding redundant array of independent disks for processing, without merging the random write requests, and reduces the number of randomly issued write requests in a centralized manner, suppressing the RAID write amplification caused by the instantaneously burst random write requests of the system, improving the performance of the storage system, and reducing the risk of system freezing and crashing. It solves the technical problem that the method of processing random read / write requests in the related art reduces the performance of the storage system, and achieves the technical effect of improving the performance of the storage system.

[0041] An embodiment of the present application provides a data read / write request processing method, which is applied to a storage system. Figure 5 It is a schematic flowchart of the data read / write request processing method provided by the embodiments of the present application, as Figure 5 shown, the process includes the following steps: Step S501, receive a data read / write request and determine the category of the data read / write request.

[0042] Specifically, the above step S501 includes: Step S5011, for any data read / write request, determine the first redundant array of independent disks and the first stripe number corresponding to the data read / write request.

[0043] In the request classification module, a first hash table and a first linked list corresponding to each redundant array of independent disks (RAID) are established based on the RAID ID of each RAID. Among them, the first hash table includes a plurality of first hash bucket linked lists corresponding to a plurality of first hash buckets.

[0044] It should be noted that through the RAID ID and the logical block address (Logical Block Addressing, abbreviated as: LBA) in the metadata of the data read / write request, the stripe number corresponding to the data read / write request can be determined.

[0045] Step S5012: Based on the first stripe number, determine the first hash bucket value corresponding to the first stripe number.

[0046] Specifically, the first hash bucket value corresponding to the first stripe number is determined by the following formula: First hash bucket value = first stripe number % first hash table size Among them, % is the modulo operation. The first hash table size is the number of first hash buckets in the first hash table, generally a power of 2, to optimize the efficiency of the modulo operation.

[0047] It should be noted that the hash table size is usually preset according to the system memory and performance requirements.

[0048] By determining the hash bucket value corresponding to the stripe number in the above manner, it can be ensured that the stripe number is evenly mapped into the hash buckets. This uniform distribution helps to avoid hash conflicts, thereby improving the search efficiency.

[0049] Step S5013: Based on the first RAID and the first hash bucket value, store the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket.

[0050] Among them, the metadata includes the access logical block address, that is, the logical block address, the data length of the data read / write request, the RAID ID corresponding to the data read / write request, etc.

[0051] It can be understood that based on the first RAID, the corresponding target first hash table is determined. Based on the first hash bucket value, the target first hash bucket linked list corresponding to the target first hash bucket with the first hash bucket value is determined from the target first hash table, and the metadata of the data read / write request is stored at the end of the target first hash bucket linked list.

[0052] Exemplarily, Figure 6 is a schematic diagram of the first hash table corresponding to the RAID with RAID ID 0 provided by the embodiment of the present application. As Figure 6As shown, the first hash table includes 2048 first hash buckets, i.e., the first hash bucket 0 to the first hash bucket 2047. Among them, the linked list of the first hash bucket 0 corresponding to the first hash bucket 0 includes the metadata of the data read / write request 1 with the stripe number being stripe 0. The linked list of the first hash bucket 3 corresponding to the first hash bucket 3 includes the metadata of the data read / write request 1 with the stripe number being stripe 3. The linked list of the first hash bucket 5 corresponding to the first hash bucket 5 includes the metadata of the data read / write request 1 with the stripe number being stripe 5 and the metadata of the data read / write request 2 with the stripe number being stripe 5. Each node in the linked list corresponds to the metadata of a data read / write request. In the linked list, the pointer field of the last node is set to NULL to indicate the end of the linked list.

[0053] Step S5014: Detect whether there is a first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number corresponding to the metadata of this data read / write request.

[0054] Step S5015: If it exists, determine whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata of this data read / write request.

[0055] If it exists, it means that there is a data read / write request with the same RAID ID and stripe number as this data read / write request. Then, further determine whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata of this data read / write request.

[0056] Optionally, if there is a first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number corresponding to the metadata of this data read / write request, determine that this data read / write request is a sequential read / write request, i.e., a sequential read request or a sequential write request. If there is no first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number corresponding to the metadata of this data read / write request, determine that this data read / write request is a random read / write request, i.e., a random read request or a random write request.

[0057] Step S5016: If the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata, determine that this data read / write request is a sequential read / write request, where the sequential read / write request includes a sequential read request and a sequential write request.

[0058] If the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata, determine that this data read / write request is a sequential read request or a sequential write request.

[0059] Step S502: If the category of the data read / write request is a random read request, and in the case that the target data corresponding to the random read request does not exist in the target cache, the random read request is sent to the corresponding redundant array of independent disks (RAID) for processing. For details, please refer to Figure 3 Step S302 of the embodiment shown, which will not be elaborated here.

[0060] Step S503: If the category of the data read / write request is a random write request, the random write request is sent to the random write cache. In the case that the memory amount occupied by the random write cache exceeds the first memory amount threshold, a preset number of random write requests in the random write cache are sent to the random write backup cache. For details, please refer to Figure 3 Step S303 of the embodiment shown, which will not be elaborated here.

[0061] Step S504: In the case that within the first preset time period, the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold, the random write requests in the random write cache and the random write backup cache are sent to the corresponding redundant array of independent disks (RAID) for processing. For details, please refer to Figure 3 Step S304 of the embodiment shown, which will not be elaborated here.

[0062] The data read / write request processing method provided by the embodiment of the present application can quickly locate relevant metadata by determining the redundant array of independent disks (RAID) and stripe number corresponding to the data read / write request and classifying and storing them in combination with the hash bucket value. If it is detected that there is metadata adjacent to or coincident with the logical block address of the data read / write request in the target hash bucket linked list, the data read / write request is determined as a sequential read / write request. This method can effectively identify the sequential read / write mode, thereby optimizing the data read / write path and reducing the performance overhead caused by random access.

[0063] Since when the number of random write requests increases, they are cached in the random write cache. When the random write cache is full, they are stored in the random write backup cache, and then the random write requests are gradually flushed down when the sequential read / write requests regain the advantage. The random write cache reduces the ineffective pre-reading of random read requests and the merging operation of random write requests, reduces the software time consumption, and suppresses the RAID write amplification caused by the instantaneous burst of random write requests in the system, improves the performance of the storage system, and reduces the risk of system freezing and crashing.

[0064] In some optional implementation manners, the above data read / write request processing method further includes: Step a1: If the target data corresponding to the random read request exists in the target cache, the target data is obtained from the target cache and returned to the server to complete the processing of the random read request.

[0065] In some optional implementation manners, the above data read / write request processing method further includes: Step b1, if the logical block address in the first target metadata is not adjacent to and does not coincide with the logical block address in the metadata of the data read / write request, it is determined that the category of the data read / write request is a random read / write request, where the random read / write request includes a random read request and a random write request.

[0066] Among them, if the logical block address in the first target metadata is not adjacent to and does not coincide with the logical block address in the metadata, it is determined that the category of the data read / write request is a random read / write request, that is, a random read request or a random write request.

[0067] The data read / write request processing method provided by the embodiments of the present application can clearly determine whether the data read / write request is a random read / write request or a sequential read / write request by detecting whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the current metadata. This differentiation method can ensure that the system adopts different optimization strategies for different types of data read / write requests, thereby improving the overall performance.

[0068] In some alternative embodiments, the above data read / write request processing method further includes: Step c1, when storing the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket, insert the metadata of the data read / write request into the tail of the target first linked list corresponding to the first redundant array of independent disks (RAID), where each RAID corresponds to multiple first hash bucket linked lists and a first linked list.

[0069] Among them, the first linked list is a least recently used (LRU) linked list.

[0070] Exemplarily, Figure 7 is a schematic diagram of the first linked list corresponding to the RAID with RAID ID 0 provided by the embodiments of the present application. As Figure 7 shown, the first linked list includes the metadata of the data read / write request 1 with stripe number stripe 0, the metadata of the data read / write request 1 with stripe number stripe 5, the metadata of the data read / write request 1 with stripe number stripe 3, and the metadata of the data read / write request 2 with stripe number stripe 5.

[0071] Step c2, every second preset time period, start checking from the head of each first linked list whether there is metadata that exceeds the first preset time.

[0072] Among them, the second preset time period and the first preset time are set by technicians and are not specifically limited here.

[0073] The request classification module sets a timer, and when the timer expires, it starts checking from the head of each first linked list to see if there is metadata that exceeds the set time, i.e., the first preset moment.

[0074] It should be noted that the metadata that exceeds the first preset moment means that the reception time of the data read / write request corresponding to the metadata is before the first preset moment before the current moment. It can be understood that the first preset moment is updated as the current moment is updated.

[0075] Step c3, if there is any, delete the metadata that exceeds the first preset moment from the corresponding first linked list and the first hash bucket linked list.

[0076] Optionally, in the case where there is metadata that exceeds the first preset moment in the first linked list, start checking from the head of the first linked list. If the first number of metadata is checked and all the first number of metadata exceed the first preset moment, stop checking and delete the first number of metadata from the corresponding first linked list and the first hash bucket linked list.

[0077] That is to say, for any first linked list, until a certain number of metadata are checked from the head of the first linked list and all the certain number of metadata exceed the first preset moment, or, when encountering metadata that does not exceed the first preset moment, stop checking and the deletion program ends.

[0078] It can be understood that since the locality of data read / write requests includes spatial locality and temporal locality, the judgment of sequential read / write requests only needs to judge whether there is sequentiality in data transfer requests within a short period of time. Therefore, the request classification module only stores metadata of a small number of data transfer requests, and its storage query time is very fast, and the memory usage and module consumption are very small.

[0079] The data read / write request processing method provided by the embodiments of the present application can ensure that the metadata in the system is always up-to-date through operations of regularly checking and cleaning expired metadata. This is particularly important for data read / write requests that need to be processed in real time, and can avoid incorrect operations caused by using expired metadata.

[0080] In some optional embodiments, the above step S503 includes: Step d1, based on multiple second linked lists in the random write cache, determine a preset number of random write requests. Among them, each redundant array of independent disks corresponds to a second linked list. When the random write cache receives a random write request, based on the redundant array of independent disks corresponding to the random write request, insert the metadata of the random write request into the tail of the corresponding second linked list.

[0081] Among them, the random write cache is established in the memory. When the random write cache receives a random write request, it determines the redundant array of independent disks (RAID) and the stripe number corresponding to the random write request, determines the third hash bucket value corresponding to the stripe number corresponding to the random write request based on the stripe number corresponding to the random write request, and stores the metadata of the random write request into the target third hash bucket linked list corresponding to the target third hash bucket based on the RAID corresponding to the random write request and the third hash bucket value. When storing the metadata of the random write request into the target third hash bucket linked list corresponding to the target third hash bucket, the metadata of the random write request is inserted into the tail of the corresponding second linked list based on the RAID corresponding to the random write request. Each RAID corresponds to multiple third hash bucket linked lists.

[0082] It can be understood that a preset number of random write requests are determined from multiple second linked lists of the random write cache in the order of the reception time of the random write requests corresponding to the metadata from early to late.

[0083] Step d2: Organize the preset number of random write requests into the form of a random write log. The random write log includes the random write data and metadata of the random write request, and one random write request corresponds to one random write log.

[0084] Step d3: Group the random write logs according to the stripe size of the random write backup cache, so that the data volume of the random write logs in each group matches the stripe size of the random write backup cache.

[0085] It should be noted that the random write backup cache is established in a RAID array with a solid state drive (SSD) as the medium. That is to say, the random write backup cache exists in the form of a redundant array of independent disks.

[0086] When the memory amount occupied by the random write cache exceeds the first memory amount threshold, if the random write requests in the random write cache are directly flushed down to the RAID, the write amplification is very large. For example, for RAID6, the write amplification is 6 times. By establishing a random write backup cache, the random write requests sent by the random write cache are organized in the form of logs and organized according to the stripe size of the RAID to which the random write backup cache belongs. Subsequently, the stripe data is sent down to the random write backup cache, that is, the RAID array. Since it is sent down in full stripes, if the random write backup cache is RAID6 with a stripe width of 25, then its write amplification is 27 / 25, that is, 1.08 times, and the write amplification is very small.

[0087] Exemplarily, Figure 8 is a schematic diagram of the random write backup cache storing random write logs provided by the embodiment of the present application. As Figure 8As shown, in the case where the random write request corresponding to the metadata with stripe number stripe 1 in the linked list of the third hash bucket 1 corresponding to the third hash bucket 1 is a random write request sent to the random write backup cache, organize the random write request in the form of a random write log, and when the total amount of data of multiple random write requests to be sent to the random write backup cache matches the stripe size of the random write backup cache, send the random write logs corresponding to the multiple random write requests to the random write backup cache in the form of full stripes, as Figure 8 shown.

[0088] It should be noted that after sending the random write requests in the random write cache to the random write backup cache, the random write cache retains the metadata of the sent random write requests and the storage locations of the random write logs corresponding to the random write requests in the random write backup cache for query by the request classification module.

[0089] Step d4, batch send the grouped random write logs to the random write backup cache.

[0090] The data reading and writing request processing method provided by the embodiments of the present application groups the random write logs according to the stripe size of the random write backup cache, so that the data volume of the random write logs in each group matches the stripe size of the random write backup cache. Send the random write logs to the random write backup cache in the form of full stripes, reducing write amplification and the number of writes, thereby improving the write efficiency.

[0091] In some optional embodiments, the above data reading and writing request processing method includes: Step e1, if the category of the data reading and writing request is a sequential reading and writing request, send the sequential reading and writing request to the sequential reading and writing cache. Among them, in the sequential reading and writing cache, the sequential read request directly uses the user's memory cache (buffer), and the sequential write request allocates the corresponding memory.

[0092] Step e2, if the category of the data reading and writing request is a sequential read request, when there is no data to be read corresponding to the sequential read request in the target cache, send the sequential read request from the sequential reading and writing cache to the corresponding independent disk redundant array for processing, and cache the data to be read and pre-read data of the sequential read request to the sequential reading and writing cache.

[0093] It can be understood that to determine whether there is data to be read corresponding to the sequential read request in the target cache, the logical block address in the metadata of the sequential read request can be compared with the logical block addresses in the metadata of each data request in the target cache. If there is any logical block address in the metadata in the target cache that is the same as the logical block address in the metadata of the sequential read request, it is determined that there is data to be read corresponding to the sequential read request in the target cache, and the data to be read is obtained from the target cache and returned to the server to complete the processing of the sequential read request.

[0094] It is understandable that the random write cache stores the metadata of the random write requests corresponding to the random write logs in the random write backup cache and the storage locations of the random write logs corresponding to the random write requests in the random backup cache. If a sequential read request hits the metadata of the random write request corresponding to the random write log in the random write backup cache, based on the storage location of the random write log corresponding to the random write request corresponding to the hit metadata in the random backup cache, the random write data corresponding to the random write request, i.e., the data to be read corresponding to the sequential read request, can be quickly read out from the corresponding location.

[0095] It should be noted that after any data read / write request is sent to the RAID, the metadata corresponding to the data read / write request is deleted from the corresponding second linked list or third linked list or random write backup cache, and the metadata corresponding to the data read / write request is deleted from the corresponding second hash bucket linked list or third hash bucket linked list.

[0096] Furthermore, after any random write request or sequential write request is sent to the RAID, the random write data of the random write request or the sequential write data of the sequential write request is deleted from the corresponding random write cache or random write backup cache or sequential read / write cache. It should be noted that after the random write cache sends a preset number of random write requests to the random write backup cache, the random write data of the sent random write requests is deleted.

[0097] In some alternative embodiments, step e2 includes: Step f1, if the request range of the sequential read request is within a single block of its corresponding redundant array of independent disks (RAID), and the data length of the sequential read request is less than the data length of a single block, then the data in the single block corresponding to the sequential read request other than the data to be read is used as pre-read data and cached in the sequential read / write cache.

[0098] It is understandable that in the case where the data to be read corresponding to the sequential read request does not exist in the target cache, pre-read expansion needs to be performed on the sequential read request.

[0099] Perform pre-read operation on the sequential read request. If the request range of the sequential read request is within a certain RAID block, and the data length of the sequential read request, i.e., the data length of the data to be read, is less than the RAID block, then pre-read is extended to the entire block.

[0100] Step f2, if the request range of the sequential read request is within two data blocks of its corresponding redundant array of independent disks (RAID), and the data length of the sequential read request is less than the data length of the two data blocks, then the data in the two data blocks corresponding to the sequential read request other than the data to be read is used as pre-read data and cached in the sequential read / write cache.

[0101] If the request range of the sequential read request is located in two RAID chunks and the IO request length is less than two RAID chunks, then the prefetch is extended to two chunks.

[0102] Step f3, if the request range of the sequential read request exceeds two data chunks of the corresponding redundant array of independent disks (RAID), and the data length of the sequential read request is less than the first ratio of the stripe width of the corresponding RAID, then determine the prefetch length as the difference between the second ratio of the stripe width of the RAID corresponding to the sequential read request and the data length of the sequential read request. Based on the prefetch length and the request range, determine the prefetch data and cache the prefetch data into the sequential read / write cache, where the second ratio is greater than the first ratio.

[0103] Among them, the first ratio can be 1 / 4 and the second ratio can be 1 / 2.

[0104] That is to say, if the request range of the sequential read request is more than two RAID chunks and the data length of the sequential read request is less than 1 / 4 of the RAID stripe width, then the prefetch is extended to 1 / 2 of the RAID stripe width.

[0105] Step f4, if the request range of the sequential read request exceeds the first ratio of the stripe width of the corresponding RAID, then use the data other than the data to be read in the stripe corresponding to the sequential read request as the prefetch data and cache it into the sequential read / write cache.

[0106] That is to say, if the request range of the sequential read request is greater than 1 / 4 of the RAID stripe width, then the prefetch is extended to the entire RAID stripe width.

[0107] The data read / write request processing method provided by the embodiments of the present application improves the cache hit rate by prefetching and loading the data that may be needed into the cache in advance.

[0108] In some optional embodiments, the above data read / write request processing method includes: Step g1, if the category of the data read / write request is a sequential write request, then determine the second hash bucket linked list corresponding to the sequential write request from the sequential read / write cache. Among them, when the sequential read / write cache receives any sequential read / write request, based on the second RAID corresponding to the sequential read / write request and the second stripe number, store the metadata of the sequential read / write request into the target second hash bucket linked list, and each RAID corresponds to multiple second hash bucket linked lists.

[0109] Among them, when the sequential read / write cache receives any sequential read / write request, based on the second independent disk redundant array and the second stripe number corresponding to the sequential read / write request, the specific implementation of storing the metadata of the sequential read / write request into the target second hash bucket linked list is similar to the descriptions in the foregoing steps S5011 to S5013, and will not be elaborated herein.

[0110] Step g2: Determine whether there is second target metadata in the second hash bucket linked list corresponding to the sequential write request, where the corresponding stripe number is the same as the target second stripe number of the metadata of the sequential write request.

[0111] Step g3: If it exists, merge the target sequential write request corresponding to the second target metadata with the sequential write request to obtain a merged sequential write request.

[0112] It can be understood that first, it is determined whether the target sequential write request corresponding to the second target metadata can be merged with the sequential write request. If it can be merged, the target sequential write request corresponding to the second target metadata is merged with the sequential write request to obtain a merged sequential write request. Among them, it can be determined whether it can be merged by whether the logical block addresses in the second target metadata and the metadata of the sequential write request are adjacent or coincident.

[0113] It should be noted that if it can be merged, after obtaining the merged sequential write request, the second target metadata and the metadata of the sequential write request are merged into one metadata, corresponding to the merged sequential write request, the merged metadata is re-inserted into the second hash bucket linked list and the third linked list corresponding to the sequential write request, and the second target metadata and the metadata of the sequential write request before merging are deleted from the corresponding second hash bucket linked list and the third linked list.

[0114] Step g4: When the data length of the merged sequential write request is full of blocks or full of stripes, send the merged sequential write request to the corresponding independent disk redundant array for processing.

[0115] It can be understood that the data length of the merged sequential write request being full of blocks means that the data length of the merged sequential write request can form a data block, and the data length of the merged sequential write request being full of stripes means that the data length of the merged sequential write request can form all the data blocks in a stripe.

[0116] The data read / write request processing method provided by the embodiment of the present application, when there is second target metadata in the second hash bucket linked list corresponding to the sequential write request, where the corresponding stripe number is the same as the target second stripe number of the metadata of the sequential write request, merges the sequential write request and the target sequential write request into a merged sequential write request, and when the data length of the merged sequential write request is full of blocks or full of stripes, sends it to the corresponding RAID for processing, reducing disk operations, reducing write amplification, and improving the performance of the storage system.

[0117] In some optional embodiments, the above data read / write request processing method includes: Step h1, at every third preset time period interval, check from the head of each third linked list whether there is metadata exceeding the second preset time. Among them, when the sequential read / write cache receives a sequential read / write request, based on the second independent disk redundant array corresponding to the sequential read / write request, insert the metadata of the sequential read / write request into the target third linked list, and each independent disk redundant array corresponds to a third linked list.

[0118] Step h2, if there is any, send the sequential write request corresponding to the metadata exceeding the second preset time to the corresponding independent disk redundant array for processing.

[0119] In some optional embodiments, the above data read / write request processing method includes: Step i1, if the memory amount occupied by the sequential read / write cache exceeds the second memory amount threshold, send the sequential write requests in the sequential read / write cache to the corresponding independent disk redundant array for processing.

[0120] In some optional embodiments, the above data read / write request processing method includes: Step j1, within the first preset time period, when the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold, send the newly received random read / write requests to the sequential read / write cache as sequential read / write requests.

[0121] Step j2, within the first preset time period, when the proportion of sequential read / write requests in the received data read / write requests is less than the preset proportion threshold, process the newly received sequential read / write requests as random read / write requests.

[0122] That is to say, count the proportion of random read / write requests and sequential read / write requests within a certain set time interval. If the proportion of random read / write requests is less than 10%, then the random write requests are no longer sent to the random write cache, but are put into the sequential read / write cache for processing, and the random read requests are also put into the sequential read / write cache for processing. If the proportion of sequential read / write requests is less than 10%, put the sequential write requests into the random write cache. When there is no corresponding data for the sequential read requests in the target cache, directly send the sequential read requests to the corresponding RAID and no longer perform prefetch expansion.

[0123] The data read / write request processing method provided by the embodiments of the present application classifies data read / write requests and distributes the classified data read / write requests to different cache modules of the cache array to adopt different processing strategies, so that the optimal processing method can be adopted for sequential read / write requests, random read / write requests, and mixed scenarios, thereby improving the performance of the storage system.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0125] The embodiments of the present application also provide an electronic device, such as Figure 9 shown, including a processor 901 and a memory 902. The memory 902 stores a computer program, and the processor 901 is configured to run the computer program to execute the steps in any of the above embodiments of the data read / write request processing method.

[0126] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the data read / write request processing method when running.

[0127] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs and other various media that can store computer programs.

[0128] The embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the data read / write request processing method.

[0129] The embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the data read / write request processing method.

[0130] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered as exceeding the scope of this application.

[0131] The above has introduced in detail a data read / write request processing method, an electronic device, a storage medium, and a program product provided by this application. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing data read / write requests, characterized in that, Including: Receiving a data read / write request and determining the category of the data read / write request; If the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, sending the random read request to the corresponding redundant array of independent disks (RAID) for processing; If the category of the data read / write request is a random write request, sending the random write request to the random write cache, and when the memory amount occupied by the random write cache exceeds the first memory amount threshold, sending a preset number of random write requests in the random write cache to the random write backup cache; When the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold within the first preset time period, sending the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks (RAID) for processing.

2. The method according to claim 1, wherein The method further includes: If the target data corresponding to the random read request exists in the target cache, obtaining the target data from the target cache and returning it to the server to complete the processing of the random read request.

3. The method according to claim 1, characterized in that The determining the category of the data read / write request includes: For any data read / write request, determining the corresponding first redundant array of independent disks (RAID) and the first stripe number; Based on the first stripe number, determining the first hash bucket value corresponding to the first stripe number; Based on the first redundant array of independent disks (RAID) and the first hash bucket value, storing the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket; Detecting whether there is a first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number of the metadata; If it exists, determining whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata; If the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata, determining that the data read / write request is a sequential read / write request, where the sequential read / write request includes a sequential read request and a sequential write request.

4. The method according to claim 3, characterized in that, The method further includes: If the logical block address in the first target metadata is not adjacent to and does not coincide with the logical block address in the metadata, determining that the category of the data read / write request is a random read / write request, where the random read / write request includes a random read request and a random write request.

5. The method according to claim 3, characterized in that, The method further includes: When storing the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket, inserting the metadata into the tail of the target first linked list corresponding to the first redundant array of independent disks (RAID), where each redundant array of independent disks (RAID) corresponds to multiple first hash bucket linked lists and one first linked list; Checking every second preset time period starting from the head of each first linked list whether there is metadata exceeding the first preset time; If it exists, deleting the metadata exceeding the first preset time from the corresponding first linked list and first hash bucket linked list.

6. The method according to claim 1, wherein The sending a preset number of random write requests in the random write cache to the random write backup cache includes: Determine a preset number of random write requests based on multiple second linked lists in the random write cache, where each redundant array of independent disks corresponds to a second linked list, and when the random write cache receives a random write request, based on the redundant array of independent disks corresponding to the random write request, insert the metadata of the random write request into the tail of the corresponding second linked list; Organize the preset number of random write requests into the form of a random write log, where the random write log includes the random write data and metadata of the random write request, and one random write request corresponds to one random write log; Group the random write logs according to the stripe size of the random write backup cache, so that the data volume of the random write logs in each group matches the stripe size of the random write backup cache; Batch issue the grouped random write logs to the random write backup cache; Among them, the random write backup cache exists in the form of a redundant array of independent disks.

7. The method according to claim 1, wherein The method further includes: If the category of the data read / write request is a sequential read / write request, issue the sequential read / write request to the sequential read / write cache; If the category of the data read / write request is a sequential read request, in the case that the data to be read corresponding to the sequential read request does not exist in the target cache, issue the sequential read request from the sequential read / write cache to the corresponding redundant array of independent disks for processing, and cache the data to be read and the pre-read data of the sequential read request to the sequential read / write cache.

8. The method according to claim 7, wherein Caching the pre-read data of the sequential read request to the sequential read / write cache includes: If the request range of the sequential read request is within a single block of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the data length of a single block, then use the data in the single block corresponding to the sequential read request except for the data to be read as pre-read data and cache it to the sequential read / write cache; If the request range of the sequential read request is within two blocks of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the data length of two blocks, then use the data in the two blocks corresponding to the sequential read request except for the data to be read as pre-read data and cache it to the sequential read / write cache; If the request range of the sequential read request exceeds two blocks of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the first ratio of the stripe width of the corresponding redundant array of independent disks, determine the pre-read length as the difference between the second ratio of the stripe width of the corresponding redundant array of independent disks and the data length of the sequential read request, determine the pre-read data based on the pre-read length and the request range, and cache the pre-read data to the sequential read / write cache, where the second ratio is greater than the first ratio; If the request range of the sequential read request exceeds the first ratio of the stripe width of its corresponding redundant array of independent disks, then use the data in the stripe corresponding to the sequential read request except for the data to be read as pre-read data and cache it to the sequential read / write cache.

9. The method according to claim 7, characterized in that, The method further includes: If the category of the data read / write request is a sequential write request, determine a second hash bucket linked list corresponding to the sequential write request from the sequential read / write cache. When the sequential read / write cache receives any sequential read / write request, it stores the metadata of the sequential read / write request in the target second hash bucket linked list based on the second independent disk redundant array and the second stripe number corresponding to the sequential read / write request. Each independent disk redundant array corresponds to multiple second hash bucket linked lists; Determine whether there is second target metadata in the second hash bucket linked list corresponding to the sequential write request, where the stripe number corresponding to the second target metadata is the same as the target second stripe number corresponding to the metadata of the sequential write request; If there is, merge the target sequential write request corresponding to the second target metadata with the sequential write request to obtain a merged sequential write request; When the data length of the merged sequential write request reaches the full block or full stripe, send the merged sequential write request to the corresponding independent disk redundant array for processing.

10. The method according to claim 9, characterized in that The method further includes: Every third preset time period, start checking from the head of each third linked list whether there is metadata that exceeds the second preset time. When the sequential read / write cache receives a sequential read / write request, it inserts the metadata of the sequential read / write request into the target third linked list based on the second independent disk redundant array corresponding to the sequential read / write request. Each independent disk redundant array corresponds to one third linked list; If there is, send the sequential write request corresponding to the metadata that exceeds the second preset time to the corresponding independent disk redundant array for processing.

11. The method according to claim 9, characterized in that, The method further includes: If the memory amount occupied by the sequential read / write cache exceeds the second memory amount threshold, send the sequential write requests in the sequential read / write cache to the corresponding independent disk redundant array for processing.

12. The method according to claim 1, characterized in that, The method further includes: In the case that the proportion of random read / write requests in the received data read / write requests within the first preset time period is less than the preset proportion threshold, send the newly received random read / write requests to the sequential read / write cache as sequential write requests; In the case that the proportion of sequential write requests in the received data read / write requests within the first preset time period is less than the preset proportion threshold, process the newly received sequential write requests as random read / write requests.

13. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the data read / write request processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, where the computer program implements the steps of the data read / write request processing method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, comprising a computer program, characterized in that, The computer program implements the steps of the data read / write request processing method according to any one of claims 1 to 12 when executed by a processor.

Images