An end-to-end dual-channel storage IO processing method

Abstract

The present invention discloses an end-to-end dual-channel storage IO processing method, which belongs to the field of computer storage technology. The method comprises the following steps: business identification is performed on the client side, IO is labeled, and then, when creating an RDMA channel, two IO channels with completely isolated resources are created. When the IO is a low-latency IO, the low-latency IO channel is used to send data; if it is a normal IO, the normal IO channel is selected for sending. When processing IO at the back-end storage node, all low-latency IO are preferentially polled from the low-latency channel for processing, and then some IO are taken out from the normal IO channel for processing. After the processing is completed, all low-latency IO are continuously polled from the low-latency channel for processing, thereby ensuring that the end-to-end latency of the low-latency IO is controllable. The method effectively ensures that multi-tenants meet the test performance and latency standards at the same time, and when capacity expansion is not timely, VIP customers and key business resources are limitedly guaranteed, thereby reducing the business losses caused by the untimely capacity expansion.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer storage, and in particular relates to an end-to-end dual-channel storage IO processing method. Background Art

[0002] With the rise of digital transformation, cloud services are gaining increasing popularity. IT systems are increasingly being migrated to the cloud. However, this migration process has encountered numerous challenges. The most typical example is that previously, systems were built or storage devices were purchased in-house, and performance was primarily dependent on the hardware. However, due to the use of shared hardware resources, performance competition between tenants often occurs, causing one tenant's business to be impacted by another tenant's, completely unaware of the impact. To mitigate this impact, most cloud vendors have implemented multi-tenant service isolation. However, resource isolation for multi-tenant cloud storage is challenging because the underlying storage system is a unified storage cluster, making it impossible to allocate unique resources to each tenant. Furthermore, the cloud relies on overselling to reduce overall operating costs and achieve profitability. This makes it impossible to allocate sufficient storage resources to each tenant, forcing preemption to be the only solution.

[0003] Most current systems ensure that user SLAs are met by reserving sufficient resources and then implementing emergency capacity expansion when performance is insufficient. While this approach solves most issues, it still doesn't address the situation where multiple tenants, some verifying latency while others verify IOPS performance, make it difficult for customers to meet latency requirements. There's also the risk that untimely capacity expansion could harm VIP customers and critical services. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the shortcomings of the background technology and provide an end-to-end dual-channel storage IO processing method. In a distributed storage system, in order to ensure the performance of key services, priority sorting or QoS flow control is usually used to limit cluster traffic to ensure the performance of IO processing.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] An end-to-end dual-channel storage IO processing method, implemented based on the LAVA storage system, specifically includes the following steps:

[0007] Step 1: Identify services on the client and label IOs to indicate whether they are low-latency IOs or normal IOs.

[0008] Step 2: When creating an RDMA channel, create two IO channels with completely isolated resources. If the IO is low-latency IO, use the low-latency IO channel to send data; if it is a normal IO, use the normal IO channel to send data.

[0009] Step 3: When the backend storage node processes IO, it polls all low-latency IOs from the low-latency channel for processing, and then takes out some IOs from the normal IO channel for processing; after the processing is completed, it continues to poll all low-latency IOs from the low-latency channel for processing, thereby ensuring that the end-to-end latency of the low-latency IO is controllable.

[0010] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, the LAVA storage system includes three layers: application layer, index layer and persistence layer.

[0011] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, the application layer, index layer and persistence layer are specifically as follows:

[0012] The application layer is deployed on computing nodes to provide cloud disk services.

[0013] The index layer is deployed on storage nodes and manages LBA indexes for service block storage, as well as routing data based on indexes and views when writing data.

[0014] The persistence layer manages the underlying disks, ECs, and replicas to ensure data storage reliability.

[0015] By identifying the block service layer business and establishing RDMA dual channels between the application layer and the index layer, and between the index layer and the persistence layer, end-to-end storage IO dual channels are achieved, ensuring that low latency and high IOPS services do not affect each other.

[0016] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, in step 2, when creating the RDMA channel, the transport layer RDMA isolation channel is as follows:

[0017] In step 2.1, two completely independent RDMA EP channels are established between the application layer compute node and the index layer storage node, including the required message header resources and data transmission memory space to ensure the independence of the transmission channels. After the two channels are established, EP0 is required to be sent first to increase the EP0 data transmission speed. The RDMA EP channel is used for data transmission between nodes.

[0018] In step 2.2, when data needs to be sent, the network protocol layer uses the EP0 channel to send low-latency, high-priority IO data and the EP1 channel to send high IOPS, low-priority IO data based on the priority tag in the header.

[0019] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, a priority tag is added to the header of the RDMA data IO transmission. The transport layer selects the low-latency channel or the high-IOPS channel based on the priority tag. Although the service can obtain IO of different priorities through different polling channels, during processing, the tag in the header is directly used to determine whether the subsequent low-latency channel or the high-IOPS channel will be used.

[0020] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, the application layer is specifically as follows:

[0021] 1) Block service layer, implements IO channel selection and labels IO:

[0022] 2) Monitor the low-latency channel. If the IO volume of the low-latency channel reaches a certain level, strengthen the low-latency channel threshold to ensure the number of low-latency channel IO.

[0023] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, in step 1), the block service layer implements IO channel selection and labels IO, which specifically includes the following steps:

[0024] 1.1, low-concurrency cloud disk, and the cloud disk type is XSSD1 or above;

[0025] 1.2. For top users’ cloud disk IO, if the IO volume is not large, the concurrent channel will be low;

[0026] 1.3. Low-latency cloud disks or cloud disk IO for customers with strict SLA guarantees use low-latency channels.

[0027] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, the index layer is specifically as follows:

[0028] 1) Each time data is processed, it is first obtained from the low-latency channel for processing, and then the data is obtained from the high-IO channel for processing after the processing is completed;

[0029] 2) Data acquisition through high IO channels should be restricted;

[0030] 3) Further reduce the latency of the channel to obtain IO and select a higher-performance resource pool for processing.

[0031] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, the persistence layer is specifically as follows:

[0032] 1) Each time data is processed, it is first obtained from the low-latency channel for processing, and then the data is obtained from the high-IO channel for processing after the processing is completed;

[0033] 2) When obtaining data from high IO channels, restrictions should be imposed;

[0034] 3) Further IO obtained from the low-latency channel is written to higher-performance storage media;

[0035] 4) When SSDs support priority scheduling in the future, IOs obtained through low-latency channels will be marked with priority processing tags.

[0036] As a further preferred solution of the end-to-end dual-channel storage IO processing method of the present invention, the data transmission process is implemented using an isolated channel:

[0037] Step 1), the block layer receives the upper layer IO;

[0038] Step 2) Construct the header for data IO transmission;

[0039] Step 3) Determine which IO to use for data transmission;

[0040] Step 4) Set the priority tag in the header to high priority;

[0041] Step 5) Call the transport layer for data transmission, select EP0 as the data transmission channel according to the priority label, and perform data transmission;

[0042] Step 6) The index layer receives the IO data sent by the block layer by polling the IO in the high-priority channel first, and checks the priority tag in the header for priority processing;

[0043] Step 7), after the processing is completed, select the EP0 channel for data transmission according to the priority tag;

[0044] Step 8) The persistence layer receives IO data from the index layer by polling the IO in the high-priority channel first, and checks the priority tag in the header for priority processing;

[0045] Step 9) After the processing is completed, select the EP0 channel according to the priority tag to reply to the index layer processing completion;

[0046] Step 10) The index layer receives the IO information replied by the persistence layer by polling the IO in the high-priority channel first, performs optimization processing, and selects the EP0 channel to reply to the block layer application according to the priority label;

[0047] Step 11) The block layer receives the IO information replied by the index layer by polling the IO in the high-priority channel first, and completes the processing and replies to the upper-layer application.

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] 1. Compared with existing QoS control and network IO priority solutions, the present invention completely isolates communication resources through service tags + independent IO passage. At the same time, when processing services, it prioritizes polling low-latency (or high-priority) channel tasks, ensuring that low-latency IO can be processed in a timely manner regardless of how busy the service is, avoiding the situation where low-latency IO is blocked by other low-priority services;

[0050] 2. The new dual-channel solution: First, when establishing an RDMA channel, the resources requested by the two channels are completely isolated, and channel connections are not reused, preventing I / O channel blockage from affecting services of different priorities. (Under the previous solution, since channels were not isolated, even prioritized services could still be blocked by low-priority I / O within the channel.) Second, by setting different polling strategies for low-latency channels and standard channels, I / O on the low-latency channel is processed in a timely manner, preventing I / O starvation on the standard I / O channel.

[0051] 3. Due to the deferred channel, the IO traffic is not large. Although it will occupy the resources of the ordinary IO channel, most of it is less than 5% (when verifying the million IOPS and single concurrent latency at the same time, the impact is less than 1.5%). Therefore, customers can verify the latency and IOPS at the same time, achieving the goal of verifying the million IOPS and 100us at the same time.

[0052] 4. This invention can effectively solve the problem that when different tenants verify IOPS and latency at the same time, the latency of a single concurrent connection cannot meet the standard, resulting in SLA failure.

[0053] 5. The present invention can effectively guarantee the IO latency and timeliness of IO processing for key businesses and VIP customers, thereby ensuring that even when the cluster is under high pressure, the key business and VIP customer experience can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 This is a diagram of the LAVA storage system architecture of the present invention;

[0056] Figure 2 Schematic diagram of the transport layer RDMA channel of the present invention;

[0057] Figure 3 This is a schematic diagram of the label in the header of the present invention being transmitted along with the IO;

[0058] Figure 4 It is a business processing flow chart of the present invention;

[0059] Figure 5 This is a flowchart of the implementation of sending data through the isolation channel of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The present invention is described in detail below based on the drawings and preferred embodiments. The purpose and effect of the present invention will become more clear. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0062] In distributed storage systems, to ensure critical business performance, priority sorting or QoS flow control is usually used to limit cluster traffic to ensure IO processing performance.

[0063] The priority sorting method can guarantee the processing performance and latency of some IO modules, but lacks end-to-end latency guarantee. End-to-end performance cannot be guaranteed because other modules or subsystems do not perform priority processing.

[0064] QoS flow control ensures that the entire cluster does not overload, thereby guaranteeing the overall quality of service. This approach shifts the pressure from the cluster side to the client. When the client host machine experiences high IO pressure, this will cause IO queues on the client host machine (due to cluster QoS flow control), increasing latency and affecting the experience of critical business or VIP customers.

[0065] This patent proposes an end-to-end storage IO dual-channel method: based on the LAVA storage system, LAVA is the next-generation storage engine within Tianyi Cloud.

[0066] First, the business is identified on the client side, and the IO is labeled to indicate whether it is low-latency IO or normal IO. Secondly, when creating an RDMA channel, two IO channels with completely isolated resources are created. When the IO is low-latency IO, the low-latency IO channel is used to send data; if it is a normal IO, the normal IO channel is selected for sending. Finally, when processing IO on the back-end storage node, all low-latency IOs are polled from the low-latency channel for processing first, and then some IOs (for example, 5) are taken out from the normal IO channel for processing. After the processing is completed, all low-latency IOs are continued to be polled from the low-latency channel for processing, thereby ensuring that the end-to-end latency of low-latency IO is controllable. This solution can effectively ensure that multiple tenants meet the test performance and latency standards at the same time, and when capacity expansion is not timely, VIP customers and key business resources are limitedly guaranteed, thereby reducing the approximate business losses due to untimely capacity expansion.

[0067] This patent is mainly based on the next-generation LAVA storage system. The architecture of the next-generation LAVA storage system is as follows:

[0068] like Figure 1 As shown in the figure, the LAVA storage system is divided into three layers. The application layer is deployed on the computing nodes to provide cloud hard disk services to the upper layer. The index layer is deployed on the storage nodes and mainly serves the management of block storage LBA indexes and routing data according to indexes and views when writing. The persistence layer mainly manages the underlying disks and EC and replicas to ensure the reliability of data storage. This patent mainly realizes end-to-end storage IO dual channels by identifying block service layer services and establishing RDMA dual channels between the application layer and index layer, and between the index layer and persistence layer, so as to achieve low latency and high IOPS services without affecting each other.

[0069] 1. Transport layer RDMA isolation channel design:

[0070] 1) If Figure 2Figure 2 shows a schematic diagram of the channel connection between two nodes through which EPs pass. Two completely independent RDMA EP channels are established between the application-layer compute node and the index-layer storage node (or between the index-layer storage node and the persistence-layer storage node), including the required message header resources and data transmission memory space, to ensure the independence of the transmission channels. After the two channels are established, EP0 is prioritized for transmission, thereby increasing the EP0 data transmission speed (most network cards and switches support this feature. If they do not, it only means that the channel cannot prioritize data transmission, which does not affect the implementation of the isolated channel).

[0071] 2) When data needs to be sent, the network protocol layer uses the EP0 channel to send low-latency (high priority) IO data and the EP1 channel to send high IOPS (low priority) IO data according to the priority mark in the header.

[0072] 2. Network layer service design:

[0073] A priority tag is added to the header of the RDMA data (IO) transmission (as shown below). The transport layer chooses to use the low-latency channel or the high-IOPS channel based on the priority tag. Although the business can obtain IO of different priorities by polling different channels, during processing, the tag in the header is directly used to determine whether the subsequent channel is a low-latency channel or a high-IOPS channel. For example, the KV Server obtains a low-latency IO by polling the high-priority channel and starts. When sending it to the Chunk Server for processing, it needs to determine whether to use the low-latency channel or the high-IOPS channel by judging the priority tag in the header. For example, after the Chunk Server receives the low-latency IO from the KV Server and completes the processing, when replying to the KV Server, it also needs to use the priority tag in the header to determine which channel to use for data transmission.

[0074] In this design, if Figure 3 As shown in the figure, the label in the header is transmitted along with the IO until the end-to-end IO processing is completed.

[0075] 3. Business processing design:

[0076] As shown below, the patent process is as follows Figure 4 Down:

[0077] Block service layer (application layer):

[0078] 1) Block service layer, implements IO channel selection and labels IO:

[0079] 1.1 Low-concurrency cloud disk, and the cloud disk type is XSSD1 or above; this is Tianyi Cloud's product called XSSD1. Regarding patents, it can be written in a more general way instead of targeting XSSD1.

[0080] 1.2 For top users, if the IO volume of cloud disk is not large, the concurrent channel will be low;

[0081] 1.3 Low-latency cloud disks or cloud disk IO for customers with strict SLA guarantees use low-latency channels.

[0082] 2) Monitor the low-latency channel. If the IO volume of the low-latency channel reaches a certain level, strengthen the low-latency channel threshold to ensure the number of low-latency channel IO.

[0083] KV service layer (index layer):

[0084] 1) Each time processing is performed, data is obtained from the low-latency channel first for processing, and then data is obtained from the high-IO channel for processing after processing is completed;

[0085] 2) Data acquisition through high-IO channels must be limited to no more than 8 IOs and less than 1M data at a time (this can be adjusted based on business needs to ensure that it does not affect the processing of low-latency tasks); 3) For IOs acquired through further low-latency channels, select a higher-performance resource pool (such as the SCM resource pool) for processing.

[0086] Chunk service layer (persistence layer):

[0087] 1) Each time processing is performed, data is obtained from the low-latency channel first for processing, and then data is obtained from the high-IO channel for processing after processing is completed;

[0088] 2) As with KV Server, when obtaining data from high-IO channels, restrictions must be imposed, processing no more than 4 IOs at a time, and less than 128KB of data (this can be adjusted based on business needs to ensure that it does not affect the processing of low-latency tasks);

[0089] 3) Further write the IO obtained from the low-latency channel to higher-performance storage media, such as PMEM;

[0090] 4) When SSDs support priority scheduling in the future, IOs obtained through low-latency channels will be marked with priority processing tags.

[0091] 4. Use isolated channels to send data implementation process, such as Figure 5 As shown:

[0092] 1) The block layer receives the upper layer IO;

[0093] 2) Construct the header for data (IO) transmission;

[0094] 3) Determine which IO to use for data transmission (for the judgment logic, see the business processing design section in the previous chapter);

[0095] 4) Set the priority tag in the header to high priority (assuming it is a high priority IO);

[0096] 5) Call the transport layer for data transmission. According to the priority label, select EP0 as the data transmission channel for data transmission;

[0097] 6) The index layer receives IO data from the block layer by polling the IO in the high-priority channel first, and checks the priority tag in the header for priority processing;

[0098] 7) After the processing is completed, according to the priority tag, select the EP0 channel for data transmission;

[0099] 8) The persistence layer receives IO data from the index layer by polling the IO in the high-priority channel first, and checks the priority tag in the header for priority processing (such as persisting the data to disk).

[0100] 9) After the processing is completed, select the EP0 channel according to the priority tag to reply to the index layer that the processing is completed;

[0101] 10) The index layer receives the IO information replied by the persistence layer by polling the IO in the high-priority channel first, performs optimization processing, and selects the EP0 channel to reply to the block layer application according to the priority label;

[0102] 11) The block layer receives the IO information replied by the index layer by polling the IO in the high-priority channel first, and completes the processing and replies to the upper-layer application.

[0103] In this embodiment, the implementation is mainly for RDMA connection. Similarly, in a TCP network or other modes, multiple channels can also be established for receiving and sending storage IO priority tasks.

[0104] Furthermore, while this embodiment is primarily designed for dual channels, it can also be expanded to include multiple priority channel designs, enabling finer-grained service identification and prioritized transmission for more priority tasks. However, more priorities mean more physical connection channels, which also consumes more resources. Therefore, this patent focuses on the dual-channel implementation.

[0105] Compared with existing QoS control and network IO priority solutions, this solution completely isolates communication resources through service tags and independent IO routing. Furthermore, during service processing, low-latency (or high-priority) channel tasks are prioritized for polling. This ensures that low-latency IO is processed promptly regardless of service load, preventing it from being blocked by other low-priority services.

[0106] The new dual-channel solution completely isolates the resources requested by the two channels when establishing the RDMA channel, and the channel connections are not reused, preventing I / O channel blockage from impacting services of different priorities. (Under the previous solution, since the channels were not isolated, even prioritized services could still be blocked by low-priority I / O within the channel.) Secondly, by setting different polling strategies for the low-latency channel and the standard channel, I / O on the low-latency channel is processed in a timely manner, preventing I / O starvation on the standard channel.

[0107] Due to the deferred channel, the IO communication volume is not large. Although it will occupy ordinary IO channel resources, most of it is less than 5% (when verifying millions of IOPS and single concurrent latency at the same time, the impact is less than 1.5%). Therefore, it can support customers to verify latency and IOPS at the same time, achieving the goal of simultaneous verification of millions of IOPS and 100us.

[0108] The main effects of this program are:

[0109] This solution can effectively solve the problem of different tenants verifying IOPS and latency at the same time, resulting in single concurrent latency failing to meet the standard and SLA failure.

[0110] This solution can effectively guarantee the IO latency and timeliness of IO processing for key businesses and VIP customers, thereby ensuring that even when the cluster is under high pressure, the experience of key businesses and VIP customers can be guaranteed.

[0111] This end-to-end storage IO dual-channel technology is a universal technology that can be used for different storage services such as files, objects, and blocks, to achieve the identification and limited processing of high- and low-priority tasks, and ensure the storage IO latency of key storage services. The main technical protection points are:

[0112] A method to implement IO priority scheduling through business tags + independent dual-channel (or multi-channel) resource isolation;

[0113] A method that uses different polling strategies for low-latency channels and normal channels;

[0114] A method for low-latency service label identification and end-to-end service label delivery.

[0115] Those skilled in the art will understand that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will still be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention. All technical features in this embodiment may be freely combined according to actual needs.

[0116] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end dual-channel storage IO processing method, characterized by: This is implemented based on the LAVA storage system and includes the following steps: Step 1: Identify services on the client and label IOs to indicate whether they are low-latency IOs or normal IOs. Step 2: When creating an RDMA channel, create two IO channels with completely isolated resources. If the IO is a low-latency IO, use the low-latency IO channel to send data; if it is a normal IO, use the normal IO channel to send data. Step 3: When the backend storage node processes IO, it polls all low-latency IOs from the low-latency IO channel for processing, and then polls some ordinary IOs from the ordinary IO channel for processing; After the processing is completed, all low-latency IOs are polled from the low-latency IO channel for processing. Polling is a method of sequentially reading network data packets. The LAVA storage system includes an application layer, an index layer, and a persistence layer; The application layer, index layer, and persistence layer are as follows: The application layer is deployed on computing nodes to provide cloud disk services. The index layer is deployed on the backend storage nodes to manage the service block storage LBA index and route data based on the index and view when writing data; The persistence layer manages the underlying disks, ECs, and replicas to ensure data storage reliability. By identifying the client's business and establishing dual RDMA channels between the application layer and the index layer, and between the index layer and the persistence layer, two IO channels with completely isolated resources are created, ensuring that the low-latency IO channel and the normal IO channel do not affect each other.

Images