A method for separating requests

Abstract

The present application relates to the technical field of electronic digital data processing, and in particular to a method for separating and processing requests. The method comprises: obtaining a target I / O request; if the target I / O request is a write request, determining the type of the target I / O request based on the type of data to be written by the target I / O request; if the target I / O request is of a performance I / O type, selecting N disks from the disks of the target computer as target disks for the target I / O request, and writing the data to be written in the target I / O request to each of the target disks; if the target I / O request is of a computing I / O type, sending the target I / O request to a target external storage node to implement verification and redundancy of the data to be written in the target I / O request in the target external storage node. The present invention can ensure high performance of storage access and provide high storage utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a method for separating and processing requests. Background Art

[0002] In storage systems, data redundancy schemes are often employed to ensure data reliability. Conventional data redundancy schemes include multiple replicas and checksums. Multiple replicas do not consume significant computing resources and generally offer good storage access performance. However, these schemes are expensive. A two-replica data redundancy scheme corresponds to half the total storage capacity, a three-replica scheme to one-third, and so on. Checksums, on the other hand, do not duplicate data, effectively resolving the low storage utilization issues inherent in multiple replicas. For example, a RAID 5 scheme constructed with five disks contains four-fifths of the total storage capacity, while a RAID 6 scheme constructed with six disks contains four-sixths. However, checksums require parity checks and calculations, consuming significant computing resources within the computer, leading to resource constraints and poor storage access performance. Summary of the Invention

[0003] The present invention aims to provide a method for separating and processing requests to ensure high performance of storage access and provide higher storage utilization.

[0004] According to the present invention, a method for separating and processing requests is provided, comprising the following steps:

[0005] S100, obtaining a target I / O request; the target I / O request is an upper-layer I / O request received by the database file system or an I / O request generated by the database file system itself.

[0006] S200, if the target I / O request is a write request, determine the type of the target I / O request based on the type of data to be written by the target I / O request; the type of the data to be written by the target I / O request is a metadata type or a non-metadata type, and the type of the target I / O request is a performance I / O type or a computing I / O type.

[0007] S300: If the target I / O request type is a performance I / O type, then proceed to S400; if the target I / O request type is a computing I / O type, then proceed to S500.

[0008] S400, select N disks from the disks of the target computer as target disks for the target I / O request, and write the data to be written in the target I / O request to each of the target disks respectively, so as to achieve N copies of redundancy of the data to be written in the target I / O request in the target computer; N is the redundancy number corresponding to the target I / O request; the target computer is the computer running the database file system.

[0009] S500, sending a target I / O request to a target external storage node to implement verification and redundancy of data to be written in the target I / O request in the target external storage node; the target external storage node is an external storage node among several external storage nodes of the target computer.

[0010] Compared with the prior art, the present invention has at least the following beneficial effects:

[0011] The present invention first determines whether an upper-layer I / O request received by a database file system or a request generated by the database file system itself is a write request. If so, the type of the request is determined based on the type of data to be written in the request; if the request is a performance I / O type, the data in the request is stored on the disk of the target computer and multiple copies are performed for redundancy; if the request is a computing I / O type, the request is sent to a target external storage node so that the target external storage node can verify and redundancy the data to be written in the request. The present invention supports two data redundancy schemes in a file system. For I / O requests of the performance I / O type (the storage address of the corresponding data to be written is accessed more frequently), a multi-copy data redundancy scheme is used to ensure higher storage access performance; for I / O requests of the calculation type (the storage address of the corresponding data to be written is accessed less frequently), a checksum data redundancy scheme is used to improve data storage utilization. Checksum redundancy is performed on an external storage node, which can offload computing tasks from the computing node of a target computer to the external storage node, does not consume resources of the computing node of the target computer, and reserves more computing resources for the computing node. At the same time, no tasks in the external storage node compete with the task of calculating the parity checksum, which can improve the efficiency of calculating the parity checksum. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1A flowchart of a method for separating and processing requests provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] According to the present invention, Figure 1 As shown, a method for separating and processing requests is provided, comprising the following steps:

[0016] S100, obtaining a target I / O request; the target I / O request is an upper-layer I / O request received by the database file system or an I / O request generated by the database file system itself.

[0017] As a specific implementation, the upper-layer I / O requests received by the database file system are I / O requests generated when users read or write. The I / O requests generated by the database file system itself include I / O requests for reading metadata triggered by user behavior or I / O requests generated by scheduled tasks that are not triggered by user behavior.

[0018] S200, if the target I / O request is a write request, determine the type of the target I / O request based on the type of data to be written by the target I / O request; the type of the data to be written by the target I / O request is a metadata type or a non-metadata type, and the type of the target I / O request is a performance I / O type or a computing I / O type.

[0019] In this embodiment, the I / O request is an input / output request, where an input request corresponds to reading data, and an output request corresponds to writing data. When the I / O request corresponds to writing data, the I / O request is determined to be a write request; when the I / O request corresponds to reading data, the I / O request is determined to be a read request. As a specific embodiment, the I / O request includes an identifier for indicating whether the I / O request is a write request or a read request, and based on the identifier, it can be determined whether the I / O request is a write request or a read request.

[0020] In this embodiment, the type of data to be written by the target I / O request is known; if the I / O request is a write request, the I / O request also includes an identifier for characterizing the type of data to be written by the I / O request. As a specific implementation, if the identifier for characterizing the type of data to be written by the I / O request is 1, it indicates that the type of data to be written by the I / O request is a metadata type; if the identifier for characterizing the type of data to be written by the I / O request is 0, it indicates that the type of data to be written by the I / O request is a non-metadata type.

[0021] As a specific implementation, determining the type of the target I / O request according to the type of data to be written by the target I / O request includes:

[0022] S210, if the type of data to be written by the target I / O request is a metadata type, then determine that the type of the target I / O request is a performance I / O type; if the type of data to be written by the target I / O request is a non-metadata type, then enter S220.

[0023] S220, if the data to be written by the target I / O request is hot data, determine the type of the target I / O request to be a performance I / O type; if the data to be written by the target I / O request is cold data, determine the type of the target I / O request to be a computing I / O type.

[0024] In this embodiment, hot data in metadata type data and non-metadata type data are data that are frequently accessed, and cold data in non-metadata type data are data that are infrequently accessed.

[0025] In this embodiment, if the target I / O request is a read request, the type of the target I / O request is not determined, and steps S300-S500 are not executed. Instead, the process is performed according to any method for processing a read request in the prior art. As a specific embodiment, S220 also includes a process for determining whether the data to be written by the target I / O request is hot data or cold data, which includes:

[0026] S221 , obtaining a virtual address add1 of data to be written in the target I / O request.

[0027] In this embodiment, if the target I / O request is a write request, the target I / O request also includes the address of the data to be written, which is a virtual address. According to the virtual address and the conversion relationship from the virtual address to the physical address, the actual storage address (i.e., the physical address) of the data to be written in the target I / O request can be obtained.

[0028] S222 , obtaining the frequency f1 of accesses to the physical address corresponding to add1 within the target historical time period.

[0029] In this embodiment, the target historical time period is the historical time period closest to the current moment and having a preset length.

[0030] In this embodiment, regardless of whether data is written or read from the physical address corresponding to add1 within the target historical time period, it is regarded as the physical address corresponding to add1 being accessed; the frequency f1 of the physical address corresponding to add1 being accessed within the target historical time period is the sum of the frequency of the physical address corresponding to add1 being written to within the target historical time period and the frequency of the physical address corresponding to add1 being read from within the target historical time period.

[0031] S223 , if f1 ≥ f0 , then determine that the data to be written by the target I / O request is hot data; otherwise, determine that the data to be written by the target I / O request is cold data.

[0032] In this embodiment, f0 is an empirical value.

[0033] S300: If the target I / O request type is a performance I / O type, then proceed to S400; if the target I / O request type is a computing I / O type, then proceed to S500.

[0034] In this embodiment, if the type of the target I / O request is a performance I / O type, it means that the address to which the target I / O request intends to write data is frequently accessed, and the performance of storage access is more concerned, and the performance requirement for storage access is relatively high; if the type of the target I / O request is a computing I / O type, it means that the address to which the target I / O request intends to write data is not frequently accessed, and the performance of storage access is not much concerned, and the performance requirement for storage access is relatively low.

[0035] S400, select N disks from the disks of the target computer as target disks for the target I / O request, and write the data to be written in the target I / O request to each of the target disks respectively, so as to achieve N copies of redundancy of the data to be written in the target I / O request in the target computer; N is the redundancy number corresponding to the target I / O request; the target computer is the computer running the database file system.

[0036] In this embodiment, the target disks include N disks, where N ≥ 2. The data to be written in the target I / O request is written to each disk in the target disks. Consequently, each disk in the target disks stores the data to be written in the target I / O request, thereby achieving N redundant copies of the data to be written in the target I / O request. Because data redundancy schemes for multiple copies do not require significant computing resources, writing the data to be written in the target I / O request to the target computer's disks in this embodiment does not consume significant computing resources within the target computer, and thus does not significantly impact the speed at which other computing tasks are processed within the computer.

[0037] As an optional specific implementation, the redundancy quantities corresponding to different I / O requests are equal, N is a preset value, and 2≤N≤5.

[0038] As a preferred specific implementation, the redundancy numbers corresponding to different I / O requests may be the same or different. The process of obtaining N includes:

[0039] S410, obtaining the number of times the target address is accessed in the most recent first time period, H, where H=(H1, H2, ..., H i ,…,H n ), H i H is the number of times the i-th target address is accessed in the most recent first time period, where i ranges from 1 to n, and n is the number of current replicas corresponding to the data to be written by the target I / O request; the target address is the storage address corresponding to the data to be written by the target I / O request, and different target addresses correspond to different disks; i =(H i,1 ,H i,2 ,…,H i,j ,…,H i,m ), H i,j is the number of times the i-th target address is accessed in the j-th sub-time period in the most recent first time period, where j ranges from 1 to m, and m is the number of sub-time periods included in the most recent first time period.

[0040] In this embodiment, the duration of the first time period and the duration of the sub-time period are both empirical values. As a specific implementation, the duration of the first time period is 1 minute, and the duration of the sub-time period is 1 second.

[0041] In this embodiment, n is known; the target I / O request includes a virtual address corresponding to the data to be written, and the storage address corresponding to the data to be written by the target I / O request can be obtained based on the virtual address and the conversion relationship from the virtual address to the physical address (the address is a physical address); if the number of copies corresponding to the data stored at the storage address is s, that is, the data stored at the storage address has backups in another s-1 disks, then n=s.

[0042] S420, traverse H and obtain the average number of times PH the i-th target address is accessed in the most recent first time period i , PH i =∑ m j＝1 H i,j / m.

[0043] S430, obtaining the number of times the target disk is accessed in the most recent first time period, D=(D1, D2, ..., D i ,…,Dn ), D i is the number of times the i-th target disk is accessed in the most recent first time period, where the i-th target disk is the disk corresponding to the i-th target address; D i =(D i,1 ,D i,2 ,…,D i,j ,…,D i,m ), D i,j is the number of times the i-th target disk is accessed in the j-th sub-time period within the most recent first time period.

[0044] S440, traverse D and obtain the average number of times PD the i-th target disk is accessed in the most recent first time period i , PD i =∑ m j＝1 D i,j / m.

[0045] S450, according to PH i and PD i Obtain a copy quantity adjustment judgment value F for data to be written by a target I / O request.

[0046] Preferably, F satisfies the following conditions: F = ∑ n i＝1 ((PH i / PD i -a / 1000)×1000+(PD i / TD i -b / 100)×100) / n, where a is the preset heat expansion threshold, b is the preset disk pressure threshold, TD i The upper limit of the number of times the i-th target disk is accessed in a single sub-time period.

[0047] In this embodiment, a, b and TD i These are all empirical values. The larger the value of a is, the higher the access heat requirement for the target address corresponding to the data when the replica capacity is increased; the larger the value of b is, the higher the access pressure requirement for the disk corresponding to the target address when the replica capacity is increased; as a specific implementation method, a=1, b=80.

[0048] S460, if F is less than the lower limit value of the preset copy number adjustment judgment value interval, then determine whether n is greater than the first preset copy number, if so, determine n-1 as N; if F is greater than the upper limit value of the preset copy number adjustment judgment value interval, then determine whether n is less than the second preset copy number, if so, determine n+1 as N; the lower limit value of the preset copy number adjustment judgment value interval is less than 0, and the upper limit value of the preset copy number adjustment judgment value interval is greater than 0.

[0049] In this embodiment, when F is smaller than the lower limit of the preset copy number adjustment judgment value interval, if n is not greater than the first preset copy number, n is determined to be N.

[0050] In this embodiment, when F is greater than the upper limit of the preset copy number adjustment judgment value interval, if n is not less than the second preset copy number, n is determined to be N.

[0051] In this embodiment, if n-1 is determined to be N, n-1 target disks are selected from the n target disks to store the data to be written by the target I / O request; if n+1 is determined to be N, a disk of the target computer is added as the target disk, so that there are n+1 disks to store the data to be written by the target I / O request.

[0052] In this embodiment, if F belongs to the preset replica quantity adjustment judgment value interval, n is determined to be N.

[0053] In this embodiment, the lower limit value of the preset copy quantity adjustment judgment value interval, the upper limit value of the preset copy quantity adjustment judgment value interval, the first preset copy quantity and the second preset copy quantity are all empirical values. Optionally, the first preset copy quantity is 2 and the second preset copy quantity is 5.

[0054] Based on S410-S460, when the access popularity of the target address corresponding to the data and the access pressure of the disk corresponding to the target address are both high, the number of copies corresponding to the data is increased, thereby improving the storage access performance corresponding to the data; when the access popularity of the target address corresponding to the data and the access pressure of the disk corresponding to the target address are both low, the number of copies corresponding to the data is reduced, thereby reducing the storage cost corresponding to the data; thus, this embodiment achieves the purpose of taking into account both the storage access performance and storage cost of the data.

[0055] S500, sending a target I / O request to a target external storage node to implement verification and redundancy of data to be written in the target I / O request in the target external storage node; the target external storage node is an external storage node among several external storage nodes of the target computer.

[0056] Because the checksum data redundancy scheme consumes a large amount of computing resources within the target computer, and the computing resources within the target computer are very limited, the solution of this embodiment preferably sends the target I / O request to the target external storage node through Remote Direct Memory Access (RDMA) technology, thereby offloading the checksum calculation task originally performed within the target computer to the external storage node. As a result, the checksum calculation of this embodiment only consumes the computing resources of the external storage node, and does not consume the computing resources within the target computer. Moreover, since there is no business need to compete with the checksum calculation for computing resources within the external storage node, the efficiency of calculating the parity checksum can be improved, and more computing resources can be reserved for business within the target computer.

[0057] In this embodiment, the external storage nodes can be expanded as needed. As an optional specific implementation, the number of external storage nodes is greater than or equal to 2, and an external storage node is randomly selected as the target external storage node. As a preferred specific implementation, the process of obtaining the target external storage node includes:

[0058] S501, obtain the total computing power E of the external storage node, E=(E1, E2, ..., E k ,…,E q ), E k is the total computing power of the kth external storage node, the value of k ranges from 1 to q, q is the number of external storage nodes, q ≥ 2.

[0059] In this embodiment, E k The following conditions are met: k =(y k,1 ×c k ×u+y k,2 ×z k )×w k , where y k,1 is the main frequency of the CPU of the kth external storage node, y k,2 is the main frequency of the memory of the kth external storage node, c k is the number of CPU cores of the kth external storage node, z k is the ratio of the memory size of the kth external storage node to the unit memory size, u is the preset CPU weight, u>1, w k is the preset weight of the kth external storage node. Thus, the total computing power of each external storage node can be estimated more accurately and quickly.

[0060] In this embodiment, the unit memory size is 1 GB.

[0061] In this embodiment, wk and u are empirical values. As a specific implementation, u=10.

[0062] S502, obtain the computing power ratio R consumed by the external storage node, R=(R1, R2, ..., R k ,…,R q ), R k The percentage of computing power consumed by the k-th external storage node.

[0063] In this embodiment, R k The following conditions are met: R k =(x k,1 ×u+x k,2 ) / (u+1), x k,1 is the CPU usage percentage of the kth external storage node, x k,2 is the memory usage percentage of the kth external storage node.

[0064] S503, traverse E and R, according to E k and R k Get the remaining computing power G of the kth external storage node k .

[0065] In this embodiment, G k Meet the following conditions: G k =E k ×(1-R k ).

[0066] S504: Determine the external storage node with the largest remaining computing power as the target external storage node.

[0067] Based on S501-S504, the external storage node with the largest remaining computing power can be obtained more accurately and quickly, which is conducive to achieving load balancing of the external storage nodes.

[0068] As a specific implementation, implementing the checksum redundancy of the data to be written in the target I / O request in the target external storage node includes:

[0069] S510 , dividing the data to be written in the target I / O request into Q slices; Q is a preset number of slices.

[0070] In this embodiment, Q is an empirical value, and optionally, Q=4.

[0071] S520: Obtain the parity checksums of the Q slices.

[0072] Those skilled in the art know that any method for calculating the parity checksum in the prior art falls within the protection scope of the present invention.

[0073] S530, use Q+M disks in the target external storage node to store Q shards and the parity sums of the Q shards; wherein, Q disks in the Q+M disks are used to store the Q shards, and different disks store different shards; M disks in the Q+M disks are used to store the parity sums of the Q shards, and the parity sums of the Q shards stored by different disks are the same, and M is the preset number of disks for storing parity sums.

[0074] In this embodiment, after the data to be written in the target I / O request is split, the data corresponding to different shards in the Q shards are different, and the data corresponding to different shards are stored on different disks. Using Q disks can realize the storage of the data to be written in the target I / O request.

[0075] In this embodiment, M is an empirical value, and optionally, M = 1 or M = 2. If M = 1, then implementing checksum redundancy for the data to be written in the target I / O request in the target external storage node requires Q+1 disks, where Q disks are used to store the data to be written in the target I / O request, and 1 disk is used to store the parity checksums of the Q shards. If M = 2, then implementing checksum redundancy for the data to be written in the target I / O request in the target external storage node requires Q+2 disks, where Q disks are used to store the data to be written in the target I / O request, and 2 disks are used to store the parity checksums of the Q shards.

[0076] Small-scale experiments have shown that this embodiment reduces the cost by more than 50% compared to a data redundancy solution that only uses multiple copies, improves the storage access performance by more than 70% compared to a data redundancy solution that only uses checksums, and reduces the computing resource consumption within the target computer by approximately 80%.

[0077] This embodiment first determines whether an upper-layer I / O request received by the database file system or a request generated by the database file system itself is a write-type request. If so, the type of the request is determined based on the type of data to be written in the request. If the request is a performance I / O type, the data in the request is stored on the target computer's disk and multiple copies are used for redundancy. If the request is a computing I / O type, the request is sent to the target external storage node, so that the target external storage node can verify and redundancy the data to be written in the request. This embodiment supports two data redundancy schemes in a file system. For performance I / O type I / O requests (the storage address of the corresponding data to be written is accessed more frequently), a multi-copy data redundancy scheme is used to ensure higher storage access performance; for computing type I / O requests (the storage address of the corresponding data to be written is accessed less frequently), a checksum data redundancy scheme is used to improve data storage utilization, and checksum redundancy is performed on the external storage node, which can offload the computing task from the computing node of the target computer to the external storage node, does not consume the resources of the computing node of the target computer, and reserves more computing resources for the computing node. At the same time, there is no task in the external storage node to compete with the task of calculating the parity checksum, which can improve the efficiency of calculating the parity checksum.

[0078] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for separating and processing requests, characterized in that: The following steps are involved: S100, obtaining a target I / O request; the target I / O request is an upper-layer I / O request received by the database file system or an I / O request generated by the database file system itself; S200, if the target I / O request is a write request, determining the type of the target I / O request based on the type of data to be written by the target I / O request; the type of the data to be written by the target I / O request is a metadata type or a non-metadata type, and the type of the target I / O request is a performance I / O type or a computing I / O type; S300: If the target I / O request type is a performance I / O type, then proceed to S400; if the target I / O request type is a computing I / O type, then proceed to S500; S400, selecting N disks from disks of a target computer as target disks for a target I / O request, and writing the data to be written in the target I / O request to each of the target disks, so as to achieve N-copy redundancy of the data to be written in the target I / O request in the target computer; N is the number of redundancies corresponding to the target I / O request; the target computer is the computer running the database file system; S500, sending a target I / O request to a target external storage node, so as to implement verification and redundancy of data to be written in the target I / O request in the target external storage node; The target external storage node is an external storage node among several external storage nodes of the target computer.

2. The method for separating requests according to claim 1, wherein: The type of the target I / O request is determined based on the type of data to be written by the target I / O request, including: S210, if the type of data to be written by the target I / O request is a metadata type, then determine that the type of the target I / O request is a performance I / O type; if the type of data to be written by the target I / O request is a non-metadata type, then proceed to S220; S220, if the data to be written by the target I / O request is hot data, determine the type of the target I / O request to be a performance I / O type; if the data to be written by the target I / O request is cold data, determine the type of the target I / O request to be a computing I / O type.

3. The method for separating requests according to claim 2, wherein: S220 also includes a process of determining whether the data to be written by the target I / O request is hot data or cold data, which includes: S221, obtaining the virtual address add1 of the data to be written in the target I / O request; S222, obtaining the access frequency f1 of the physical address corresponding to add1 within the target historical time period; S223 , if f1 ≥ f0 , then determine that the data to be written by the target I / O request is hot data; otherwise, determine that the data to be written by the target I / O request is cold data.

4. The method for separating requests according to claim 1, wherein: Implementing checksum redundancy of data to be written in a target I / O request in a target external storage node includes: S510, dividing the data to be written in the target I / O request into Q shards, where Q is a preset number of shards; S520, obtaining the parity checksums of the Q slices; S530, use Q+M disks in the target external storage node to store Q shards and the parity sums of the Q shards; wherein, Q disks in the Q+M disks are used to store the Q shards, and different disks store different shards; M disks in the Q+M disks are used to store the parity sums of the Q shards, and the parity sums of the Q shards stored by different disks are the same, and M is the preset number of disks for storing parity sums.

5. The method for separating requests according to claim 4, wherein: M=1 or M=2.

6. The method for separating requests according to claim 1, wherein: 2≤N≤5。 7. The method for separating requests according to claim 1, wherein: The target I / O request is sent to the target external storage node through Remote Direct Memory Access (RDMA) technology.

Images