Data reading and writing request processing method, electronic device, storage medium and program product

Abstract

The present application discloses a method for processing data read / write requests, an electronic device, a storage medium, and a program product, which relates to the field of redundant array of independent disks technology. If the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is directly sent to the corresponding redundant array of independent disks for processing without prefetching; if the category of the data read / write request is a random write request, the random write cache and the random write backup cache are used to store the random write request. When the ratio of random read / write requests is less than the preset ratio threshold, the random write request is sent to the corresponding redundant array of independent disks for processing without merging the random write requests, and the number of centralized random write requests is reduced, solving the technical problem that the method for processing random read / write requests in the related art reduces the performance of the storage system, and achieving the technical effect of improving the performance of the storage system.

Description

Technical Field

[0001] This application relates to the technical field of redundant arrays of independent disks, and particularly to a method for processing data read and write requests, an electronic device, a storage medium, and a program product. Background Art

[0002] In a cloud computing data center, data storage performance is crucial. The redundant array of independent disks (RAID) technology combines multiple physical disks into a logical disk, improving the input / output (IO) performance of the storage system, and improving the data security of the storage system by providing a parity redundancy technology. However, it also introduces the problem of write amplification, which affects the performance of the storage system.

[0003] To optimize the performance of the storage system, related technologies adopt the method of establishing a cache module on the upper layer of RAID, using the cache module to cache IO data, prefetching the IO data, and merging the IO requests to process sequential IO requests. However, for random IO requests, due to the lack of locality and sequentiality of random IO requests, using this method to process them instead reduces the performance of the storage system. Summary of the Invention

[0004] This application provides a method for processing data read and write requests, an electronic device, a storage medium, and a program product, so as to at least solve the problem that the method for processing random read and write requests in related technologies reduces the performance of the storage system.

[0005] This application provides a method for processing data read and write requests, including:

[0006] Receiving a data read and write request, and determining the category of the data read and write request;

[0007] If the category of the data read and write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, sending the random read request to the corresponding redundant array of independent disks for processing;

[0008] If the category of the data read and write request is a random write request, sending the random write request to the random write cache, and when the memory amount occupied by the random write cache exceeds the first memory amount threshold, sending a preset number of random write requests in the random write cache to the random write backup cache;

[0009] When the proportion of random read and write requests in the received data read and write requests is less than the preset proportion threshold within the first preset time period, sending the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing.

[0010] The present application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above data read / write request processing methods when executing the computer program.

[0011] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any of the above data read / write request processing methods when executed by a processor.

[0012] The present application also provides a computer program product including a computer program, which implements the steps of any of the above data read / write request processing methods when executed by a processor.

[0013] Through the present application, since if the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is directly sent to the corresponding redundant array of independent disks for processing without prefetching; if the category of the data read / write request is a random write request, the random write cache and the random write backup cache are used to store the random write request. When the ratio of the random read / write requests is less than the preset ratio threshold, the random write request is sent to the corresponding redundant array of independent disks for processing without merging the random write requests, and the number of centralized random write requests is reduced, solving the technical problem in the related art that the method for processing random read / write requests reduces the performance of the storage system, and achieving the technical effect of improving the performance of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 It is a schematic diagram of data striping of RAID5 provided by an embodiment of the present application;

[0016] Figure 2 It is a schematic structural diagram of a data read / write request processing system provided by an embodiment of the present application;

[0017] Figure 3 It is a schematic flowchart of a data read / write request processing method provided by an embodiment of the present application;

[0018] Figure 4 It is a schematic structural diagram of a storage system provided by an embodiment of the present application;

[0019] Figure 5Schematic flowchart of yet another data read / write request processing method provided by an embodiment of the present application;

[0020] Figure 6 Schematic diagram of the first hash table corresponding to the RAID with RAID ID 0 provided by an embodiment of the present application;

[0021] Figure 7 Schematic diagram of the first linked list corresponding to the RAID with RAID ID 0 provided by an embodiment of the present application;

[0022] Figure 8 Schematic diagram of the random write backup cache storing random write logs provided by an embodiment of the present application;

[0023] Figure 9 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0024] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0025] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0027] In a cloud computing data center, data storage performance is crucial. With the continuous improvement of hardware performance, optimizing the software layer performance of the storage system has become the key to breaking through the overall system performance bottleneck.

[0028] As the core solution for storage systems, RAID technology combines multiple independent physical disks into logical disks in different ways, providing higher reliability and storage performance than a single disk. Through striping technology, the chunks on multiple physical disks are divided into stripes, converting IO operations into parallel operations on multiple physical disks, thus improving the IO performance of the storage system. RAID technology also provides a parity redundancy technique. By calculating a parity block from the data in the data blocks of a stripe, if a data block is damaged, the damaged data block can be recovered through the parity block, enhancing the data security of the storage system.

[0029] However, while the parity redundancy technique improves the data security of the storage system, it also brings factors that affect RAID performance, such as write amplification, which are factors affecting the performance of the storage system. Especially during random small-block data updates, the parity block must be updated synchronously, resulting in a very large write amplification for small-block data. Exemplarily, Figure 1 The data striping schematic diagram of RAID5 provided by the embodiments of this application is as Figure 1 shown. RAID5 combines 4 physical disks into a logical disk, i.e., a RAID. Among them, each physical disk is divided into disk blocks of the same size according to a fixed size, such as data block D1, data block D2, data block D3, and parity block P. The disk blocks with the same number on each physical disk form a stripe. For example, the stripe numbered stripe 0 includes data block D1 of physical disk 1, data block D2 of physical disk 2, data block D3 of physical disk 3, and parity block P of physical disk 4. For the composition of stripe 1, stripe 2, and stripe 3, refer to Figure 1 , and the detailed description is similar to that of stripe 0 and will not be elaborated here.

[0030] For example, to update the D3 data block in stripe 0, i.e., D3 new , it is necessary to first read out the old data of D3, i.e., D3 old , calculate the intermediate parity value with the new data and the old data, i.e., Delta = D3 new XOR D3 old . Then read out the old parity data, i.e., P old , calculate the new parity data, i.e., P new = Delta XOR P old , and then the new D3 new , P newThe entire process of writing to disk involves four disk operations, with a write amplification of 4 times. If the RAID type is RAID6, two disk operations for verifying data Q are added, and the write amplification is 6 times. Such a high write amplification seriously affects the performance of RAID. The disk operation of RAID becomes the bottleneck of the storage system and may cause abnormally high IO latency at a certain moment, resulting in system service interruption or even crash in a short period of time.

[0031] In order to optimize the performance of the storage system, the relevant technology adopts the method of establishing a cache module on the upper layer of RAID, using the cache module to cache IO data, pre-reading IO data and merging IO requests to process sequential IO requests, thereby reducing the write amplification of RAID and optimizing the performance of RAID and storage system.

[0032] For sequential read IO requests, a pre-read strategy is adopted, that is, the read data blocks are predicted in advance and read into the cache to optimize the latency of the read IO requests. For sequential write IO requests, they can be returned after being stored in the cache to reduce write latency, and dirty data (data that has not been written to the disk) is regularly flushed to the disk. At the same time, read IO requests or write IO requests for the same stripe are merged as much as possible to reduce disk operations. This greatly reduces the write amplification of RAID and improves system performance.

[0033] The cache design in the related art is based on the principle of locality of disk data access, that is, after the data on the disk is accessed, its adjacent data will also be accessed within a short period of time. This is called IO locality or IO sequentiality, and this type of IO request can also be called a sequential IO request. However, with the rapid development of the Internet, the number of concurrent IO operations performed by different users on the network is increasing, and different IO data of different users are unrelated to each other. There are more and more access scenarios for different small data by the same user. Many of these IOs do not have locality and sequentiality, and this type of IO request can be called a random IO request. In a certain scenario or a certain period of time, the proportion of random IO requests in the storage system is very large. For random IO requests, the related art adopts the same processing method as sequential IO requests, that is, merging, pre-reading and other strategies for processing. This method not only does not optimize the performance of the storage system, but because it pre-reads a large amount of invalid data, the IO request merging and waiting increases the retention time of the IO request, which greatly reduces the performance of the storage system.

[0034] There is also a situation where random write IO requests are issued in a concentrated manner, which causes the RAID write amplification to be very large and may cause abnormally high IO latency at a certain moment, resulting in system service interruption or even crash in a short period of time.

[0035] To solve the above problems, an embodiment of the present application provides a method for processing data read / write requests, and the method includes: receiving a data read / write request and determining the category of the data read / write request; if the category of the data read / write request is a random read request, then in the case where the target data corresponding to the random read request does not exist in the target cache, sending the random read request to the corresponding redundant array of independent disks for processing; if the category of the data read / write request is a random write request, then sending the random write request to the random write cache, and in the case where the memory amount occupied by the random write cache exceeds the first memory amount threshold, sending a preset number of random write requests in the random write cache to the random write backup cache; within the first preset time period, in the case where the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold, sending the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing. In the method provided by the above solution, if the category of the data read / write request is a random read request, then in the case where the target data corresponding to the random read request does not exist in the target cache, directly sending the random read request to the corresponding redundant array of independent disks for processing without pre-reading; if the category of the data read / write request is a random write request, then using the random write cache and the random write backup cache to store the random write requests, and when the proportion of random read / write requests is less than the preset proportion threshold, sending the random write requests to the corresponding redundant array of independent disks for processing, without the need to merge the random write requests, and reducing the amount of data for centralized sending of random write requests, thus solving the technical problem that the method for processing random read / write requests in the related art reduces the performance of the storage system, and achieving the technical effect of improving the performance of the storage system.

[0036] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the data read / write request processing method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0037] The data read / write request processing method, electronic device, storage medium, and program product provided by the embodiments of the present application are applicable to processing data read / write requests. For example Figure 2As shown in the figure, it is a schematic structural diagram of a data read / write request processing system based on an embodiment of the present application, mainly including a server and a storage system. Among them, the server is used to issue data read / write requests, and the storage system is used to receive data read / write requests and determine the category of data read / write requests. If the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is issued to the corresponding redundant array of independent disks (RAID) for processing. If the category of the data read / write request is a random write request, the random write request is issued to the random write cache. When the memory amount occupied by the random write cache exceeds the first memory amount threshold, a preset number of random write requests in the random write cache are issued to the random write backup cache. When the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold within the first preset time period, the random write requests in the random write cache and the random write backup cache are issued to the corresponding redundant array of independent disks (RAID) for processing.

[0038] An embodiment of the present application provides a data read / write request processing method, which is applied to a storage system. Figure 3 It is a schematic flowchart of the data read / write request processing method provided by the embodiment of the present application. As Figure 3 shown, the process includes the following steps:

[0039] Step S301: Receive a data read / write request and determine the category of the data read / write request.

[0040] Figure 4 It is a schematic structural diagram of the storage system provided by the embodiment of the present application. As Figure 4 shown, the storage system includes a request classification module, a cache array, and a RAID module. Among them, the request classification module is used to classify the received data read / write requests, that is, I / O requests, and issue the data read / write requests to the corresponding cache of the cache array according to the category of the data read / write requests.

[0041] The cache array includes a sequential read / write cache, a random write cache, and a random write backup cache. The RAID module is used to process the data read / write requests issued by the cache array.

[0042] Step S302: If the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, the random read request is issued to the corresponding redundant array of independent disks (RAID) for processing.

[0043] Among them, the target cache includes a sequential read / write cache and a random write cache.

[0044] If the target data corresponding to the random read request does not exist in the target cache, it means that the target cache misses, and the random read request needs to be sent to the RAID module for processing without any caching and processing operations. It can be understood that the RAID module includes multiple RAIDs, and the random read request is sent to the corresponding RAID based on the metadata of the random read request. Among them, the metadata of the random read request includes the RAID identifier (ID) of the RAID corresponding to the random read request.

[0045] It is understandable that, for a random read request, no cache is performed on the target data of the random read request, and no data is pre-read into the target cache.

[0046] Step S303: If the category of the data read and write request is a random write request, the random write request is sent to the random write cache. When the amount of memory occupied by the random write cache exceeds the first memory amount threshold, a preset number of random write requests in the random write cache are sent to the random write backup cache.

[0047] The random write cache is established in the memory. The first memory amount threshold and the preset amount are set by the technicians and are not specifically limited here. It is understandable that a preset amount of memory can be allocated to the random write cache in advance, and the first memory amount threshold can be a preset amount of memory pre-allocated to the random write cache.

[0048] Step S304: within the first preset time period, if the proportion of random read and write requests in the received data read and write requests is less than the preset proportion threshold, the random write requests in the random write cache and the random write backup cache are sent to the corresponding independent disk redundant array for processing.

[0049] The random read and write request includes a random read request and a random write request. The preset ratio threshold is set by a technician and is not specifically limited here. For example, the preset ratio threshold is 10%. The first preset time period is set by a technician and is not specifically limited here. The first preset time period is a preset time period before the current moment and is updated as the current moment is updated.

[0050] Since random read and write requests are more frequent only at a certain stage, and sequential read and write requests are the mainstream in general, when the proportion of random read and write requests received is less than 10% within the first preset time period, the random write requests in the random write cache and the random write backup cache are sent to the corresponding independent disk redundant array for processing.

[0051] The random write cache and the random write backup cache periodically query the ratio of random read / write requests of the request classification module within the first preset time period. When the ratio of random read / write requests is less than 10%, the random write cache batches and issues the random write requests therein. The random write backup cache reads the random write logs in the random write backup cache from front to back in units of stripes, parses the content of the random write logs, and batches and issues the random write requests corresponding to the random write logs to the corresponding RAID.

[0052] Within the first preset time period, when the ratio of sequential read / write requests in the received data read / write requests is less than the preset ratio threshold, the step of issuing the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing is suspended until within the first preset time period, the ratio of random read / write requests in the received data read / write requests is less than the preset ratio threshold again.

[0053] The data read / write request processing method provided by the embodiments of the present application, if the category of the data read / write request is a random read request, directly issues the random read request to the corresponding redundant array of independent disks for processing without pre-reading processing when the target data corresponding to the random read request does not exist in the target cache; if the category of the data read / write request is a random write request, uses the random write cache and the random write backup cache to store the random write requests, and when the ratio of random read / write requests is less than the preset ratio threshold, issues the random write requests to the corresponding redundant array of independent disks for processing, without merging the random write requests, and reduces the number of centralized issued random write requests, inhibits the RAID write amplification caused by the system's instantaneous burst of random write requests, improves the performance of the storage system, and reduces the risk of system freeze and crash. It solves the technical problem that the method of processing random read / write requests in the related art reduces the performance of the storage system, and achieves the technical effect of improving the performance of the storage system.

[0054] An embodiment of the present application provides a data read / write request processing method, which is applied to a storage system. Figure 5 It is a schematic flowchart of the data read / write request processing method provided by the embodiments of the present application, as Figure 5 shown, the process includes the following steps:

[0055] Step S501, receive a data read / write request and determine the category of the data read / write request.

[0056] Specifically, the above step S501 includes:

[0057] Step S5011, for any data read / write request, determine the first redundant array of independent disks and the first stripe number corresponding to the data read / write request.

[0058] In the request classification module, a first hash table and a first linked list corresponding to each redundant array of independent disks (RAID) are established based on the RAID ID of each RAID. Among them, the first hash table includes a plurality of first hash bucket linked lists corresponding to a plurality of first hash buckets.

[0059] It should be noted that through the RAID ID and the logical block address (Logical Block Addressing, abbreviated as: LBA) in the metadata of the data read / write request, the strip number corresponding to the data read / write request can be determined.

[0060] Step S5012: Based on the first strip number, determine the first hash bucket value corresponding to the first strip number.

[0061] Specifically, the first hash bucket value corresponding to the first strip number is determined by the following formula:

[0062] First hash bucket value = first strip number % first hash table size

[0063] Among them, % is the modulo operation. The first hash table size is the number of first hash buckets in the first hash table, generally a power of 2, to optimize the efficiency of the modulo operation.

[0064] It should be noted that the hash table size is usually set in advance according to system memory and performance requirements.

[0065] By determining the hash bucket value corresponding to the strip number in the above manner, it can be ensured that the strip number is evenly mapped to the hash buckets. This uniform distribution helps to avoid hash conflicts and thus improve the search efficiency.

[0066] Step S5013: Based on the first RAID and the first hash bucket value, store the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket.

[0067] Among them, the metadata includes the access logical block address, that is, the logical block address, the data length of the data read / write request, the RAID ID corresponding to the data read / write request, etc.

[0068] It can be understood that based on the first RAID, the corresponding target first hash table is determined. Based on the first hash bucket value, the target first hash bucket linked list corresponding to the target first hash bucket with the first hash bucket value is determined from the target first hash table, and the metadata of the data read / write request is stored at the end of the target first hash bucket linked list.

[0069] Exemplarily, Figure 6 This is a schematic diagram of the first hash table corresponding to the RAID with RAID ID 0 provided by the embodiment of the present application. As Figure 6As shown, the first hash table includes 2048 first hash buckets, i.e., the first hash bucket 0 to the first hash bucket 2047. Among them, the linked list of the first hash bucket 0 corresponding to the first hash bucket 0 includes the metadata of the data read / write request 1 with the stripe number being stripe 0. The linked list of the first hash bucket 3 corresponding to the first hash bucket 3 includes the metadata of the data read / write request 1 with the stripe number being stripe 3. The linked list of the first hash bucket 5 corresponding to the first hash bucket 5 includes the metadata of the data read / write request 1 with the stripe number being stripe 5 and the metadata of the data read / write request 2 with the stripe number being stripe 5. Each node in the linked list corresponds to the metadata of a data read / write request. In the linked list, the pointer field of the last node is set to NULL to indicate the end of the linked list.

[0070] Step S5014, detect whether there is a first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number corresponding to the metadata of this data read / write request.

[0071] Step S5015, if it exists, determine whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata of this data read / write request.

[0072] If it exists, it indicates that there is a data read / write request with the same RAID ID and stripe number as this data read / write request. Then, further determine whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata of this data read / write request.

[0073] Optionally, if there is a first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number corresponding to the metadata of this data read / write request, determine that this data read / write request is a sequential read / write request, i.e., a sequential read request or a sequential write request. If there is no first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number corresponding to the metadata of this data read / write request, determine that this data read / write request is a random read / write request, i.e., a random read request or a random write request.

[0074] Step S5016, if the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata, determine that this data read / write request is a sequential read / write request, where the sequential read / write request includes a sequential read request and a sequential write request.

[0075] If the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata, determine that this data read / write request is a sequential read request or a sequential write request.

[0076] Step S502: If the category of the data read / write request is a random read request, and the target data corresponding to the random read request does not exist in the target cache, then send the random read request to the corresponding redundant array of independent disks (RAID) for processing. For details, please refer to Figure 3 Step S302 of the embodiment shown in

[0077] Step S503: If the category of the data read / write request is a random write request, then send the random write request to the random write cache. When the memory amount occupied by the random write cache exceeds the first memory amount threshold, send a preset number of random write requests in the random write cache to the random write backup cache. For details, please refer to Figure 3 Step S303 of the embodiment shown in

[0078] Step S504: When the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold within the first preset time period, send the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks (RAID) for processing. For details, please refer to Figure 3 Step S304 of the embodiment shown in

[0079] The data read / write request processing method provided by the embodiment of the present application can quickly locate relevant metadata by determining the redundant array of independent disks (RAID) and stripe number corresponding to the data read / write request and classifying and storing them in combination with the hash bucket value. If it is detected that there is metadata adjacent to or overlapping with the logical block address of the data read / write request in the target hash bucket linked list, then the data read / write request is determined as a sequential read / write request. This method can effectively identify the sequential read / write mode, thereby optimizing the data read / write path and reducing the performance overhead caused by random access.

[0080] Since when the number of random write requests increases, they are cached in the random write cache. When the random write cache is full, they are stored in the random write backup cache, and then the random write requests are gradually flushed down when the sequential read / write requests regain the advantage. The random write cache reduces the ineffective pre-reading of random read requests and the merging operation of random write requests, reduces the software time consumption, and suppresses the RAID write amplification caused by the instantaneous burst of random write requests in the system, improves the performance of the storage system, and reduces the risk of system freezing and crashing.

[0081] In some optional implementation manners, the above data read / write request processing method further includes:

[0082] Step a1: If the target data corresponding to the random read request exists in the target cache, then obtain the target data from the target cache and return it to the server to complete the processing of the random read request.

[0083] In some alternative embodiments, the above data read / write request processing method further includes:

[0084] Step b1, if the logical block address in the first target metadata is not adjacent to and does not coincide with the logical block address in the metadata of the data read / write request, determine that the category of the data read / write request is a random read / write request, where the random read / write request includes a random read request and a random write request.

[0085] Wherein, if the logical block address in the first target metadata is not adjacent to and does not coincide with the logical block address in the metadata, determine that the category of the data read / write request is a random read / write request, that is, a random read request or a random write request.

[0086] The data read / write request processing method provided by the embodiments of the present application can clearly determine whether the data read / write request is a random read / write request or a sequential read / write request by detecting whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the current metadata. This differentiation method can ensure that the system adopts different optimization strategies for different types of data read / write requests, thereby improving the overall performance.

[0087] In some alternative embodiments, the above data read / write request processing method further includes:

[0088] Step c1, when storing the metadata of the data read / write request into the target first hash bucket corresponding to the target first hash bucket linked list, insert the metadata of the data read / write request into the tail of the target first linked list corresponding to the first independent disk redundant array, where each independent disk redundant array corresponds to multiple first hash bucket linked lists and a first linked list.

[0089] Wherein, the first linked list is a Least Recently Used (LRU for short) linked list.

[0090] Exemplarily, Figure 7 is a schematic diagram of the first linked list corresponding to the RAID with RAID ID 0 provided by the embodiments of the present application. As Figure 7 shown, the first linked list includes the metadata of data read / write request 1 with stripe number stripe 0, the metadata of data read / write request 1 with stripe number stripe 5, the metadata of data read / write request 1 with stripe number stripe 3, and the metadata of data read / write request 2 with stripe number stripe 5.

[0091] Step c2, every second preset time period, start checking from the head of each first linked list whether there is metadata that exceeds the first preset time.

[0092] Wherein, the second preset time period and the first preset time are set by technicians and are not specifically limited here.

[0093] The request classification module sets a timer, and when the timer expires, it starts checking from the head of each first linked list to see if there is metadata that exceeds the set time, i.e., the first preset time.

[0094] It should be noted that the metadata that exceeds the first preset time means that the reception time of the data read / write request corresponding to the metadata is before the first preset time before the current time. It can be understood that the first preset time is updated as the current time is updated.

[0095] Step c3, if any exists, delete the metadata that exceeds the first preset time from the corresponding first linked list and the first hash bucket linked list.

[0096] Optionally, in the case where there is metadata that exceeds the first preset time in the first linked list, start checking from the head of the first linked list. If the first quantity of metadata is checked and all the first quantity of metadata exceeds the first preset time, stop checking and delete the first quantity of metadata from the corresponding first linked list and the first hash bucket linked list.

[0097] That is to say, for any first linked list, until a certain quantity of metadata is checked from the head of the first linked list and the certain quantity of metadata all exceeds the first preset time, or, when encountering metadata that does not exceed the first preset time, stop checking and the deletion process ends.

[0098] It can be understood that since the locality of data read / write requests includes spatial locality and temporal locality, the determination of sequential read / write requests only needs to determine whether there is sequentiality in data transfer requests within a short period of time. Therefore, the request classification module only stores metadata of a small number of data transfer requests, and its storage query time is very fast, and the memory usage and module consumption time are very small.

[0099] The data read / write request processing method provided by the embodiments of the present application can ensure that the metadata in the system is always up-to-date through the operations of regularly checking and cleaning expired metadata. This is particularly important for data read / write requests that need to be processed in real time, and can avoid incorrect operations caused by using expired metadata.

[0100] In some optional embodiments, the above step S503 includes:

[0101] Step d1, based on multiple second linked lists in the random write cache, determine a preset quantity of random write requests. Each redundant array of independent disks corresponds to a second linked list. When the random write cache receives a random write request, based on the redundant array of independent disks corresponding to the random write request, insert the metadata of the random write request into the tail of the corresponding second linked list.

[0102] Among them, the random write cache is established in the memory. When the random write cache receives a random write request, it determines the redundant array of independent disks (RAID) and the stripe number corresponding to the random write request, determines the third hash bucket value corresponding to the stripe number corresponding to the random write request based on the stripe number corresponding to the random write request, and stores the metadata of the random write request into the target third hash bucket linked list corresponding to the target third hash bucket based on the RAID corresponding to the random write request and the third hash bucket value. When storing the metadata of the random write request into the target third hash bucket linked list corresponding to the target third hash bucket, the metadata of the random write request is inserted into the tail of the corresponding second linked list based on the RAID corresponding to the random write request. Each RAID corresponds to multiple third hash bucket linked lists.

[0103] It can be understood that a preset number of random write requests are determined from multiple second linked lists of the random write cache in the order of the reception time of the random write requests corresponding to the metadata from early to late.

[0104] Step d2: Organize a preset number of random write requests into the form of a random write log. The random write log includes the random write data and metadata of the random write request, and one random write request corresponds to one random write log.

[0105] Step d3: Group the random write logs according to the stripe size of the random write backup cache, so that the data volume of the random write logs in each group matches the stripe size of the random write backup cache.

[0106] It should be noted that the random write backup cache is established in a RAID array with a solid state drive (SSD) as the medium. That is to say, the random write backup cache exists in the form of a redundant array of independent disks.

[0107] When the memory amount occupied by the random write cache exceeds the first memory amount threshold, if the random write requests in the random write cache are directly flushed down to the RAID, the write amplification is very large. For example, for RAID6, the write amplification is 6 times. By establishing a random write backup cache, the random write requests sent by the random write cache are organized in the form of a log and organized according to the stripe size of the RAID to which the random write backup cache belongs, and then the stripe data is sent down to the random write backup cache, that is, the RAID array. Since it is full stripe sending, if the random write backup cache is RAID6 with a stripe width of 25, then its write amplification is 27 / 25, that is, 1.08 times, and the write amplification is very small.

[0108] Exemplarily, Figure 8 It is a schematic diagram of the random write backup cache storing random write logs provided by the embodiment of the present application. As Figure 8As shown, when the random write request corresponding to the metadata with stripe number being stripe 1 in the linked list of the third hash bucket 1 corresponding to the third hash bucket 1 is a random write request to be sent to the random write backup cache, organize the random write request in the form of a random write log, and when the total data volume of multiple random write requests to be sent to the random write backup cache matches the stripe size of the random write backup cache, send the random write logs corresponding to the multiple random write requests to the random write backup cache in the form of full stripes, as Figure 8 shown.

[0109] It should be noted that after sending the random write requests in the random write cache to the random write backup cache, the random write cache retains the metadata of the sent random write requests and the storage positions of the random write logs corresponding to the random write requests in the random backup cache for query by the request classification module.

[0110] Step d4, batch send the grouped random write logs to the random write backup cache.

[0111] The data read and write request processing method provided by the embodiments of this application groups the random write logs according to the stripe size of the random write backup cache, so that the data volume of the random write logs in each group matches the stripe size of the random write backup cache. Sending the random write logs to the random write backup cache in the form of full stripes reduces write amplification and the number of writes, thereby improving the write efficiency.

[0112] In some optional embodiments, the above data read and write request processing method includes:

[0113] Step e1, if the category of the data read and write request is a sequential read and write request, send the sequential read and write request to the sequential read and write cache. Among them, in the sequential read and write cache, the sequential read request directly uses the user's memory cache (buffer), and the sequential write request allocates corresponding memory.

[0114] Step e2, if the category of the data read and write request is a sequential read request, when there is no data to be read corresponding to the sequential read request in the target cache, send the sequential read request from the sequential read and write cache to the corresponding independent disk redundant array for processing, and cache the data to be read and the pre-read data of the sequential read request in the sequential read and write cache.

[0115] It can be understood that to determine whether there is data to be read corresponding to the sequential read request in the target cache, the logical block address in the metadata of the sequential read request can be compared with the logical block addresses in the metadata of each data request in the target cache. If there is any logical block address in the metadata in the target cache that is the same as the logical block address in the metadata of this sequential read request, it is determined that there is data to be read corresponding to the sequential read request in the target cache, and the data to be read is obtained from the target cache and returned to the server to complete the processing of this sequential read request.

[0116] It can be understood that the metadata of the random write request corresponding to the random write log in the random write backup cache and the storage location of the random write log corresponding to the random write request in the random backup cache are retained in the random write cache. If the sequential read request hits the metadata of the random write request corresponding to the random write log in the random write backup cache, based on the storage location of the random write log corresponding to the random write request corresponding to the hit metadata in the random backup cache, the random write data corresponding to this random write request, that is, the data to be read corresponding to the sequential read request, can be quickly read out from the corresponding location.

[0117] It should be noted that after any data read / write request is sent to the RAID, the metadata corresponding to this data read / write request is deleted from the corresponding second linked list or third linked list or random write backup cache, and the metadata corresponding to this data read / write request is deleted from the corresponding second hash bucket linked list or third hash bucket linked list.

[0118] Furthermore, after any random write request or sequential write request is sent to the RAID, the random write data of this random write request or the sequential write data of this sequential write request is deleted from the corresponding random write cache or random write backup cache or sequential read / write cache. It should be noted that after the random write cache sends a preset number of random write requests to the random write backup cache, the random write data of the random write requests sent is deleted.

[0119] In some optional implementation manners, step e2 includes:

[0120] Step f1, if the request range of the sequential read request is located in a single block of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the data length of a single block, then the data in the single block corresponding to the sequential read request except for the data to be read is used as pre-read data and cached in the sequential read / write cache.

[0121] It can be understood that in the case where there is no data to be read corresponding to the sequential read request in the target cache, it is necessary to perform pre-read expansion on this sequential read request.

[0122] For a sequential read request, a prefetch operation is performed. If the request range of the sequential read request is within a certain RAID chunk and the data length of the sequential read request, i.e., the data length of the data to be read, is less than the RAID chunk, then the prefetch is extended to the entire chunk.

[0123] Step f2, if the request range of the sequential read request is within two data chunks of its corresponding redundant array of independent disks (RAID) and the data length of the sequential read request is less than the data length of the two data chunks, then the data other than the data to be read in the two data chunks corresponding to the sequential read request is used as prefetch data and cached in the sequential read / write cache.

[0124] If the request range of the sequential read request is within two RAID chunks and the IO request length is less than the two RAID chunks, then the prefetch is extended to the two chunks.

[0125] Step f3, if the request range of the sequential read request exceeds two data chunks of its corresponding RAID and the data length of the sequential read request is less than the first ratio of the stripe width of the corresponding RAID, then determine the prefetch length as the difference between the second ratio of the stripe width of the corresponding RAID and the data length of the sequential read request. Based on the prefetch length and the request range, determine the prefetch data and cache the prefetch data in the sequential read / write cache, where the second ratio is greater than the first ratio.

[0126] Among them, the first ratio can be 1 / 4 and the second ratio can be 1 / 2.

[0127] That is to say, if the request range of the sequential read request is more than two RAID chunks and the data length of the sequential read request is less than 1 / 4 of the RAID stripe width, then the prefetch is extended to 1 / 2 of the RAID stripe width.

[0128] Step f4, if the request range of the sequential read request exceeds the first ratio of the stripe width of its corresponding RAID, then the data other than the data to be read in the stripe corresponding to the sequential read request is used as prefetch data and cached in the sequential read / write cache.

[0129] That is to say, if the request range of the sequential read request is greater than 1 / 4 of the RAID stripe width, then the prefetch is extended to the entire RAID stripe width.

[0130] The data read / write request processing method provided by the embodiments of the present application improves the cache hit rate by prefetching and loading potentially required data into the cache in advance.

[0131] In some optional embodiments, the above data read / write request processing method includes:

[0132] Step g1, if the category of the data read / write request is a sequential write request, determine the second hash bucket linked list corresponding to the sequential write request from the sequential read / write cache. When the sequential read / write cache receives any sequential read / write request, based on the second independent disk redundant array and the second strip number corresponding to the sequential read / write request, store the metadata of the sequential read / write request into the target second hash bucket linked list. Each independent disk redundant array corresponds to multiple second hash bucket linked lists.

[0133] Among them, when the sequential read / write cache receives any sequential read / write request, based on the second independent disk redundant array and the second strip number corresponding to the sequential read / write request, the specific implementation of storing the metadata of the sequential read / write request into the target second hash bucket linked list is similar to the description in the foregoing steps S5011 to S5013, and will not be elaborated here.

[0134] Step g2, determine whether there is second target metadata in the second hash bucket linked list corresponding to the sequential write request, and the corresponding strip number of which is the same as the target second strip number of the metadata of the sequential write request.

[0135] Step g3, if there is, merge the target sequential write request corresponding to the second target metadata with the sequential write request to obtain a merged sequential write request.

[0136] It can be understood that first determine whether the target sequential write request corresponding to the second target metadata and the sequential write request can be merged. If they can be merged, merge the target sequential write request corresponding to the second target metadata with the sequential write request to obtain a merged sequential write request. Among them, it can be determined whether they can be merged by whether the logical block addresses in the second target metadata and the metadata of the sequential write request are adjacent or coincident.

[0137] It should be noted that if they can be merged, after obtaining the merged sequential write request, merge the second target metadata and the metadata of the sequential write request into one metadata, corresponding to the merged sequential write request, re-insert the merged metadata into the second hash bucket linked list and the third linked list corresponding to the sequential write request, and delete the second target metadata and the metadata of the sequential write request before merging from the corresponding second hash bucket linked list and the third linked list.

[0138] Step g4, when the data length of the merged sequential write request reaches the full block or full strip, send the merged sequential write request to the corresponding independent disk redundant array for processing.

[0139] It can be understood that the data length of the merged sequential write request reaching the full block means that the data length of the merged sequential write request can form a data block, and the data length of the merged sequential write request reaching the full strip means that the data length of the merged sequential write request can form all the data blocks in a strip.

[0140] In the data read / write request processing method provided by the embodiment of the present application, when there is second target metadata in the second hash bucket linked list corresponding to the sequential write request, and the stripe number corresponding to the second target metadata is the same as the target second stripe number in the metadata of the sequential write request, the sequential write request and the target sequential write request are merged into a merged sequential write request. When the data length of the merged sequential write request reaches the full block or full stripe, it is sent to the corresponding RAID for processing, reducing disk operations, reducing write amplification, and improving the performance of the storage system.

[0141] In some optional embodiments, the above data read / write request processing method includes:

[0142] Step h1, every third preset time period, check from the head of each third linked list whether there is metadata exceeding the second preset time. Among them, when the sequential read / write cache receives a sequential read / write request, based on the second independent disk redundant array corresponding to the sequential read / write request, the metadata of the sequential read / write request is inserted into the target third linked list, and each independent disk redundant array corresponds to a third linked list.

[0143] Step h2, if there is, send the sequential write request corresponding to the metadata exceeding the second preset time to the corresponding independent disk redundant array for processing.

[0144] In some optional embodiments, the above data read / write request processing method includes:

[0145] Step i1, if the memory amount occupied by the sequential read / write cache exceeds the second memory amount threshold, send the sequential write requests in the sequential read / write cache to the corresponding independent disk redundant array for processing.

[0146] In some optional embodiments, the above data read / write request processing method includes:

[0147] Step j1, within the first preset time period, when the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold, send the newly received random read / write requests to the sequential read / write cache as sequential read / write requests.

[0148] Step j2, within the first preset time period, when the proportion of sequential read / write requests in the received data read / write requests is less than the preset proportion threshold, process the newly received sequential read / write requests as random read / write requests.

[0149] That is to say, the ratio of random read / write requests and sequential read / write requests within a certain set time interval is counted. If the proportion of random read / write requests is less than 10%, the random write requests will no longer be sent to the random write cache, but put into the sequential read / write cache for processing, and the random read requests will also be put into the sequential read / write cache for processing. If the proportion of sequential read / write requests is less than 10%, the sequential write requests will be put into the random write cache. When there is no corresponding data for the sequential read requests in the target cache, the sequential read requests will be directly sent to the corresponding RAID without prefetching expansion.

[0150] The data read / write request processing method provided by the embodiments of the present application classifies data read / write requests and sends the classified data read / write requests to different cache modules of the cache array to adopt different processing strategies, so that the optimal processing methods can be adopted for sequential read / write requests, random read / write requests and mixed scenarios, and the performance of the storage system can be improved.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation.

[0152] The embodiments of the present application also provide an electronic device, as Figure 9 shown, including a processor 901 and a memory 902. The memory 902 stores a computer program, and the processor 901 is configured to run the computer program to execute the steps in any of the above embodiments of the data read / write request processing method.

[0153] The embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above embodiments of the data read / write request processing method when running.

[0154] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks or optical discs and other media that can store computer programs.

[0155] The embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the data read / write request processing method.

[0156] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the steps in any of the above embodiments of the data read / write request processing method.

[0157] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0158] The above has introduced in detail a data read / write request processing method, an electronic device, a storage medium, and a program product provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for processing data read / write requests, characterized in that, It includes: Receiving a data read / write request and determining the category of the data read / write request; If the category of the data read / write request is a random read request, when the target data corresponding to the random read request does not exist in the target cache, sending the random read request to the corresponding redundant array of independent disks (RAID) for processing; If the category of the data read / write request is a random write request, sending the random write request to the random write cache, and when the memory amount occupied by the random write cache exceeds the first memory amount threshold, sending a preset number of random write requests in the random write cache to the random write backup cache; When the proportion of random read / write requests in the received data read / write requests is less than the preset proportion threshold within the first preset time period, sending the random write requests in the random write cache and the random write backup cache to the corresponding redundant array of independent disks for processing.

2. The method according to claim 1, wherein The method further includes: If the target data corresponding to the random read request exists in the target cache, obtaining the target data from the target cache and returning it to the server to complete the processing of the random read request.

3. The method according to claim 1, characterized in that, The determining the category of the data read / write request includes: For any data read / write request, determining the corresponding first redundant array of independent disks and the first stripe number; Based on the first stripe number, determining the first hash bucket value corresponding to the first stripe number; Based on the first redundant array of independent disks and the first hash bucket value, storing the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket; Detecting whether there is a first target metadata in the target first hash bucket linked list whose corresponding stripe number is the same as the first stripe number of the metadata; If it exists, determining whether the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata; If the logical block address in the first target metadata is adjacent to or coincides with the logical block address in the metadata, determining that the data read / write request is a sequential read / write request, where the sequential read / write request includes a sequential read request and a sequential write request.

4. The method according to claim 3, wherein The method further includes: If the logical block address in the first target metadata is not adjacent to and does not coincide with the logical block address in the metadata, determining that the category of the data read / write request is a random read / write request, where the random read / write request includes a random read request and a random write request.

5. The method according to claim 3, wherein The method further includes: When storing the metadata of the data read / write request into the target first hash bucket linked list corresponding to the target first hash bucket, inserting the metadata into the tail of the target first linked list corresponding to the first redundant array of independent disks, where each redundant array of independent disks corresponds to multiple first hash bucket linked lists and a first linked list; Checking every second preset time period starting from the head of each first linked list whether there is metadata exceeding the first preset time; If it exists, deleting the metadata exceeding the first preset time from the corresponding first linked list and first hash bucket linked list.

6. The method according to claim 1, wherein The sending a preset number of random write requests in the random write cache to the random write backup cache includes: Determine a preset number of random write requests based on multiple second linked lists in the random write cache, where each redundant array of independent disks corresponds to a second linked list, and when the random write cache receives a random write request, based on the redundant array of independent disks corresponding to the random write request, insert the metadata of the random write request into the tail of the corresponding second linked list; Organize the preset number of random write requests into the form of a random write log, where the random write log includes the random write data and metadata of the random write request, and one random write request corresponds to one random write log; Group the random write logs according to the stripe size of the random write backup cache, so that the data volume of the random write logs in each group matches the stripe size of the random write backup cache; Batch issue the grouped random write logs to the random write backup cache; Among them, the random write backup cache exists in the form of a redundant array of independent disks.

7. The method according to claim 1, characterized in that The method further includes: If the category of the data read / write request is a sequential read / write request, issue the sequential read / write request to the sequential read / write cache; If the category of the data read / write request is a sequential read request, in the case where the data to be read corresponding to the sequential read request does not exist in the target cache, issue the sequential read request from the sequential read / write cache to the corresponding redundant array of independent disks for processing, and cache the data to be read and the pre-read data of the sequential read request to the sequential read / write cache.

8. The method according to claim 7, wherein Caching the pre-read data of the sequential read request to the sequential read / write cache includes: If the request range of the sequential read request is within a single block of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the data length of a single block, then use the data in the single block corresponding to the sequential read request except for the data to be read as the pre-read data and cache it to the sequential read / write cache; If the request range of the sequential read request is within two blocks of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the data length of two blocks, then use the data in the two blocks corresponding to the sequential read request except for the data to be read as the pre-read data and cache it to the sequential read / write cache; If the request range of the sequential read request exceeds two blocks of its corresponding redundant array of independent disks, and the data length of the sequential read request is less than the first ratio of the stripe width of the corresponding redundant array of independent disks, determine the pre-read length as the difference between the second ratio of the stripe width of the corresponding redundant array of independent disks and the data length of the sequential read request, determine the pre-read data based on the pre-read length and the request range, and cache the pre-read data to the sequential read / write cache, where the second ratio is greater than the first ratio; If the request range of the sequential read request exceeds the first ratio of the stripe width of its corresponding redundant array of independent disks, then use the data in the stripe corresponding to the sequential read request except for the data to be read as the pre-read data and cache it to the sequential read / write cache.

9. The method according to claim 7, characterized in that, The method further includes: If the category of the data read / write request is a sequential write request, determine a second hash bucket linked list corresponding to the sequential write request from the sequential read / write cache. When the sequential read / write cache receives any sequential read / write request, it stores the metadata of the sequential read / write request in the target second hash bucket linked list based on the second independent disk redundant array and the second stripe number corresponding to the sequential read / write request. Each independent disk redundant array corresponds to multiple second hash bucket linked lists; Determine whether there is second target metadata in the second hash bucket linked list corresponding to the sequential write request, where the stripe number corresponding to the second target metadata is the same as the target second stripe number corresponding to the metadata of the sequential write request; If it exists, merge the target sequential write request corresponding to the second target metadata with the sequential write request to obtain a merged sequential write request; When the data length of the merged sequential write request reaches the full block or full stripe, send the merged sequential write request to the corresponding independent disk redundant array for processing.

10. The method according to claim 9, characterized in that The method further includes: Every third preset time period, start checking from the head of each third linked list to determine whether there is metadata that exceeds the second preset time. When the sequential read / write cache receives a sequential read / write request, it inserts the metadata of the sequential read / write request into the target third linked list based on the second independent disk redundant array corresponding to the sequential read / write request. Each independent disk redundant array corresponds to one third linked list; If it exists, send the sequential write request corresponding to the metadata that exceeds the second preset time to the corresponding independent disk redundant array for processing.

11. The method according to claim 9, wherein The method further includes: If the memory occupied by the sequential read / write cache exceeds the second memory threshold, send the sequential write requests in the sequential read / write cache to the corresponding independent disk redundant array for processing.

12. The method according to claim 1, wherein The method further includes: In the case where the proportion of random read / write requests in the received data read / write requests within the first preset time period is less than the preset proportion threshold, send the newly received random read / write requests to the sequential read / write cache as sequential read / write requests; In the case where the proportion of sequential read / write requests in the received data read / write requests within the first preset time period is less than the preset proportion threshold, process the newly received sequential read / write requests as random read / write requests.

13. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the data read / write request processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, where the computer program implements the steps of the data read / write request processing method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, comprising a computer program, characterized in that, The computer program implements the steps of the data read / write request processing method according to any one of claims 1 to 12 when executed by a processor.

Images