Operating system IO processing method supporting random IO merging, operating system and storage system

By merging random I/O requests into pseudo-sequential requests in user space and performing batch operations in kernel space, the problem of excessive kernel overhead under high throughput is solved, achieving more efficient I/O processing and device throughput performance.

CN121785542APending Publication Date: 2026-04-03HUAZHONG UNIV OF SCI & TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202512035191.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-31
Publication Date
2026-04-03

AI Technical Summary

Technical Problem

Existing technologies cannot effectively mitigate the kernel overhead caused by random I/O requests under high throughput conditions, resulting in insufficient utilization of CPU resources and affecting system throughput.

Method used

In user space, multiple random I/O requests are merged into pseudo-sequential requests, and batch operations are performed in kernel space to reduce the number of system calls. In the device driver layer, pseudo-sequential requests are split into random I/O requests, device commands are generated, and I/O stack operations are optimized.

Benefits of technology

It effectively reduces kernel overhead, improves system throughput and CPU resource utilization, and enhances device throughput performance while ensuring protocol compatibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121785542A_ABST
    Figure CN121785542A_ABST
Patent Text Reader

Abstract

The invention discloses an operating system IO processing method supporting random IO merging, an operating system and a storage system, and belongs to the field of computer operating systems. The method comprises the steps that in a user mode, only one parameter copy is reserved for the same parameters of a plurality of random IO requests and recorded in a shared structure A, respectively recording different parameters of each random IO request into the shared structure body B to obtain a pseudo sequence request; the pseudo-sequence requests are issued to a system kernel, corresponding operations are executed on the pseudo-sequence requests in all layers of the system kernel in sequence, the batch operations are only executed once, and the non-batch operations are executed once for all the random IO requests respectively; the batch operation and the non-batch operation are operations executed for parameters in the shared structure body A and the shared structure body B respectively; and splitting the pseudo sequence request into random IO requests in an equipment driving layer of the kernel system, generating a corresponding equipment command, and submitting the equipment command to storage equipment. According to the invention, kernel overhead under the condition of high throughput can be relieved, so that the throughput is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer operating systems, and more specifically, relates to an operating system I / O processing method, operating system, and storage system that support random I / O merging. Background Technology

[0002] With the development of PCIe / NVMe interfaces, storage throughput has been greatly improved, and a single solid-state drive (SSD) can achieve more than 2 million random I / O operations (IOPS) per second with only 4 PCIe 5.0 lanes. However, it is not uncommon for modern SSDs to underutilize their random IOPS in practice. This is due to kernel space I / O stack overhead, especially for random I / O. For an NVMe Gen5 SSD (Samsung PM1743), a single CPU core (Intel Xeon Gold 6430) can fully utilize the device's sequential I / O bandwidth, but for random I / O, eight CPU cores are needed to fully support the same bandwidth. Considering the decreasing rate of CPU performance improvement, finding ways to achieve efficient request submission with fewer CPU resources to support increasingly faster SSDs has become crucial.

[0003] Currently, methods for reducing kernel overhead mainly fall into two categories. The first category involves bypassing all or part of the kernel. For example, SPDK allows user processes to directly access storage devices to reduce I / O stack overhead. Its limitation is that applications need their own block management layer or file system to build useful I / O primitives on top of a simple block-level interface. Some methods propose bypassing certain kernel operations (such as the block layer and file system) to accelerate request submission. While these methods can reduce CPU overhead to some extent, the lack of relevant semantic verification limits their applicability, and some even require code rewriting and hardware modifications. The second category involves modifying the kernel I / O stack. As an alternative, researchers have optimized queue scheduling, reduced context switching overhead, and made the block layer lightweight.

[0004] However, existing methods cannot prevent software stack overhead from increasing proportionally with the number of requests under I / O workloads. Therefore, how to alleviate kernel overhead under high throughput conditions to improve throughput remains a pressing technical challenge. Summary of the Invention

[0005] In response to the shortcomings and improvement needs of existing technologies, this invention provides an operating system I / O processing method, operating system, and storage system that support random I / O merging. The purpose is to alleviate kernel overhead under high throughput conditions, thereby improving throughput.

[0006] To achieve the above objectives, according to one aspect of the present invention, an operating system I / O processing method supporting random I / O merging is provided, comprising: In user space, for the same parameters of the multiple random I / O requests to be merged, only one copy of the parameters is kept and recorded in shared structure A. The different parameters of the multiple random I / O requests to be merged are recorded in shared structure B respectively. Thus, the multiple random I / O requests to be merged are merged into pseudo-ordered requests. The pseudo-ordered requests are then sent to the system kernel. The corresponding operations are executed sequentially on the pseudo-sequential requests at the kernel interface layer, file system layer, and block layer of the system kernel. When executing the corresponding operations at each layer, batch operations are executed only once, while non-batch operations are executed once for each random I / O request. Batch operations are operations performed on the parameters in shared structure A, while non-batch operations are operations performed on the parameters in shared structure B. In the device driver layer of the kernel system, the pseudo-sequential requests processed by the kernel interface layer, file system layer and block layer are split into random I / O requests, and corresponding device commands are generated and submitted to the storage device.

[0007] Furthermore, the operating system I / O processing method supporting random I / O merging provided by the present invention also includes: When the storage device completes a random I / O request After processing and notifying the CPU via the terminal, the device driver layer determines whether the storage device has completed processing all random I / O requests in the pseudo-sequential request. If so, a callback process is executed uniformly for each random I / O request in the pseudo-sequential request; otherwise, the random I / O requests are released. The non-shared resources occupied are removed, and the interrupt handling process is exited.

[0008] Furthermore, the operating system I / O processing method supporting random I / O merging provided by the present invention further includes: using a bitmap in the device driver layer to track the completion status of each random I / O request in the pseudo-sequential request, or using a counter to track the completion status of random I / O requests in the pseudo-sequential request.

[0009] Furthermore, the operating system I / O processing method supporting random I / O merging provided by the present invention also includes: In user space, allocate contiguous shared address buffers for multiple random I / O requests to be merged.

[0010] Furthermore, the operating system I / O processing method supporting random I / O merging provided by this invention splits pseudo-sequential requests into random I / O requests at the device driver layer, including: The shared address buffer is divided into fixed-size cache pages, and one cache page is allocated for each random I / O request.

[0011] Furthermore, the fields in the shared structure B include: request count num, usage status in_use, submission strategy flag, error message error, and offset address array queue; Among them, the request count num is used to record the number of random I / Os merged into pseudo-sequential requests, the use status in_use is used to indicate whether random I / O merging has been performed, the submission policy flag is used to indicate the interface type for submitting requests to the system kernel, the error message error is used to record the error messages returned by the storage device during the execution of random I / O requests, and the offset address array queue is used to record the offset address of each random I / O request.

[0012] Furthermore, the shared structure A inherits from the original iocb structure of libaio.

[0013] According to another aspect of the present invention, an operating system supporting random I / O merging is provided, comprising: The IO interface module is used in user space to retain only one copy of the same parameters for multiple random IO requests to be merged and record them in a shared structure A, and to record the different parameters of the multiple random IO requests to be merged in a shared structure B, thereby merging the multiple random IO requests to be merged into a pseudo-ordered request; and then sending the pseudo-ordered request to the system kernel. The IO stack module is used to perform corresponding operations on pseudo-sequential requests sequentially at the kernel interface layer, file system layer, and block layer of the system kernel. When performing corresponding operations at each layer, batch operations are executed only once, while non-batch operations are executed once for each random IO request. Batch operations are operations performed on parameters in shared structure A, while non-batch operations are operations performed on parameters in shared structure B. And the device driver module, which is used in the device driver layer of the kernel system to split the pseudo-sequential requests processed by the kernel interface layer, file system layer and block layer into random IO requests, generate corresponding device commands, and submit each device command to the storage device.

[0014] According to another aspect of the present invention, a storage system supporting random I / O merging is provided, comprising: a storage device and an operating system supporting random I / O merging provided by the present invention.

[0015] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) In the user space, the present invention merges multiple random IO requests into a pseudo-sequential request by parameter aggregation and sends it to the kernel at once, which can effectively reduce the number of system calls. In the system kernel, the main operations of the IO stack are divided into two categories: operations performed on the same parameters (called batch operations) and operations performed on different parameters (called non-batch operations). Batch operations are executed only once, while non-batch operations are executed once for each random IO. This can reduce the individual processing of each random IO request, reduce CPU usage, and improve the processing efficiency of the kernel. In the device driver layer, the pseudo-sequential request is further distributed into random IO requests, and independent device commands are generated and sent to the device. This can efficiently submit multiple merged requests while ensuring protocol compatibility, reduce system overhead and improve device throughput performance.

[0016] In summary, this invention effectively reduces the number of system calls and random I / O requests in the I / O stack by merging multiple random I / O requests, optimizing the kernel operations of the merged pseudo-sequential requests in the I / O stack, and distributing the pseudo-sequential requests into multiple random I / O requests at the device driver layer. This effectively reduces kernel overhead under high throughput conditions, thereby improving throughput.

[0017] (2) In a preferred embodiment of the present invention, a pseudo-sequential request callback strategy with fast resource release and delayed callback is proposed. Specifically, when any random IO request obtained from the distribution is completed and the device notifies the CPU via an interrupt, it checks whether all random IO requests in the pseudo-sequential request have been completed. If not all random IO requests in the pseudo-sequential request have been completed, only the non-shared resources occupied by the random IO request are released and the interrupt is exited, thus achieving fast resource release and ensuring that the relevant resources can be used by other requests in a timely manner. If all random IO requests in the pseudo-sequential request have been completed, a callback operation is uniformly executed on all random IO requests in the pseudo-sequential request, thus achieving delayed callback and further improving the overall request processing efficiency. In its preferred embodiment, a bitmap or counter is used in the device driver layer to track the completion status of random IO requests in the pseudo-sequential request, which can quickly determine whether all random IO requests in the pseudo-sequential request have been completed.

[0018] (3) In the preferred embodiment of the present invention, when multiple random IO requests are merged into pseudo-sequential IO requests in user mode, continuous shared address buffers are allocated to the multiple random IO requests to be merged, so that the buffer base address of each random IO request is the same, increasing the same parameters of random IO requests and reducing different parameters, which can further reduce CPU overhead and improve the processing efficiency of IO requests.

[0019] (4) In the preferred embodiment of the present invention, the shared structure A used to record the same parameters of random IO is inherited from the original iocb structure of libaio. By inheriting the original structure in the operating system, the processing flow of the operating system kernel can be simplified and the kernel processing efficiency can be further improved. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the I / O stack of the existing Linux 5.12 operating system.

[0021] Figure 2 This is a schematic diagram of an operating system I / O processing method that supports random I / O merging, provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the merging of random I / O requests provided in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of kernel operation provided in an embodiment of the present invention.

[0024] Figure 5 This is a callback diagram of a pseudo-sequence request provided in an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram of an operating system that supports random I / O merging, provided as an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0027] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0028] Before explaining the technical solution of the present invention in detail, let's first combine it with... Figure 1 A brief introduction to the existing operating system I / O stack is as follows: When an I / O request reaches the operating system, the parameters of the I / O request are first encapsulated into the corresponding structure in user space, and then io_submit() is called to send the I / O request to the system kernel.

[0029] Within the system kernel, the kernel interface layer, file system layer, block layer, and device driver layer sequentially perform corresponding kernel operations on the parameters of the I / O request. The main operations of each layer are as follows: Kernel interface layer: Operations include copying request information from user space, retrieving file descriptors, parameter verification, and allocation and initialization of auxiliary structures.

[0030] File system layer: Operations include mapping user space buffers and interacting with block devices, where interacting with block devices requires allocating and initializing auxiliary structures.

[0031] Block layer: Schedules requests in the queue based on structure information, including processing flags, devices, processes, and CPU information.

[0032] Device driver layer: Operations include retrieving cached physical addresses.

[0033] After the request is processed, the corresponding callback operations will be executed according to the set callback paths. The callback paths include callback operation paths at the driver layer, block layer, file system layer, and interface layer to ensure proper release of resources and handling of I / O completion notifications.

[0034] The data blocks accessed by random I / O are located in scattered and discontinuous locations on the physical storage medium. Under random I / O load, when multiple random I / O requests are received, the above I / O processing process needs to be executed for each random I / O request, which will cause serious CPU overhead and affect system throughput.

[0035] Merging sequential I / O can fully utilize SSD bandwidth and significantly reduce CPU overhead. Inspired by this, in order to solve the technical problem of high kernel overhead under random I / O and its impact on system throughput, this invention provides an operating system I / O processing method, operating system and storage system that supports random I / O merging. The overall concept is to merge random I / O requests into sequential I / O requests (pseudo-sequential I / O requests) to reduce kernel overhead and improve system throughput. Under random workloads, the requested offset addresses are not contiguous. Since the Lowest Basic Address (LBA) is a critical parameter involved in almost all structures of the I / O stack, handling non-contiguous offset addresses and other different request parameters during random I / O becomes a problem. To achieve the above technical concept, this invention improves the internal workings of the operating system, including the following improvements: An I / O interface supporting random request merging: This interface aggregates the parameters of multiple random requests, keeping only one copy of identical parameters and storing different parameters independently as arrays or array pointers, thus merging random requests. The merged request parameters are then sent to the kernel as a pseudo-sequential request in one go to reduce the number of system calls.

[0036] An I / O stack supporting pseudo-sequential requests: The main operations of the I / O stack are divided into two types: i) offset-independent operations: mainly performing metadata management and request scheduling; ii) offset-dependent operations: involving file verification and address translation. Kernel operations are divided into multiple layers, including the interface layer, file system layer, block layer, and driver layer. By sharing structures, batch operation processing at different frequencies is implemented in the interface layer, file system layer, and block layer, thereby reducing the need for individual processing of each request and improving kernel processing efficiency.

[0037] A device driver that supports handling pseudo-sequential requests: To ensure compatibility with existing NVMe protocols, the device driver needs to redistribute the merged pseudo-sequential requests, generating independent NVMe commands for each random request and submitting them separately to the device for processing. In this way, the device driver can efficiently submit multiple merged requests while maintaining protocol compatibility, reducing system overhead and improving device throughput.

[0038] The following is an example.

[0039] Example 1: An operating system I / O processing method that supports random I / O merging, such as Figure 2 As shown, it includes: In user space, for the common parameters of the multiple random I / O requests to be merged, only one copy of the parameters is kept and recorded in shared structure A. The different parameters of the multiple random I / O requests to be merged are recorded separately in shared structure B. This merges the multiple random I / O requests into pseudo-sequential requests, such as... Figure 3 As shown; the pseudo-sequence request is sent to the system kernel; The pseudo-sequential requests are processed sequentially at the kernel interface layer, file system layer, and block layer of the system kernel. During the execution of operations at each layer, batch operations are performed only once, while non-batch operations are performed separately for each random I / O request. Figure 4 As shown; batch operations are operations performed on parameters in shared structure A, while non-batch operations are operations performed on parameters in shared structure B; In the device driver layer of the kernel system, the pseudo-sequential requests processed by the kernel interface layer, file system layer and block layer are split into random I / O requests, and corresponding device commands are generated and submitted to the storage device.

[0040] This embodiment merges multiple random I / O requests into a single pseudo-sequential request in user space by aggregating parameters and sends it to the kernel all at once. This implements an I / O interface that supports random request merging, effectively reducing the number of system calls. In the system kernel, the main operations of the I / O stack are divided into two categories: operations performed with the same parameters (called batch operations) and operations performed with different parameters (called non-batch operations). Batch operations are executed only once, while non-batch operations are executed once for each random I / O request. This implements an I / O stack that supports pseudo-sequential requests, thereby reducing the need for individual processing of each random I / O request, reducing CPU usage, and improving kernel processing efficiency. At the device driver layer, the pseudo-sequential requests are further distributed into random I / O requests, and independent device commands are generated and sent to the device. This implements a device driver that supports pseudo-sequential requests, thereby ensuring protocol compatibility while efficiently submitting multiple merged requests, reducing system overhead, and improving device throughput.

[0041] The I / O interface supporting random request merging implemented in this embodiment further includes: in user space, allocating a contiguous shared address buffer for multiple random I / O requests to be merged. This is similar to sequential I / O operations. Compared to discrete buffers, contiguous buffers simplify management and reduce allocation overhead. By allocating a shared contiguous buffer for each request, the problem of different buffer pointers between multiple requests is solved, allowing all requests to share the same base address.

[0042] Random I / O requests typically share many identical parameters. Typical parameters for each request include: offset address, buffer pointer, opcode, data size, file descriptor, priority, and flags. The offset address determines the location from which data is to be read, and the buffer pointer indicates the data's storage location. Optionally, in this embodiment, the offset address and buffer pointer for random requests are different, while the other parameters are identical. Accordingly, in this embodiment, the shared structure is designed as follows: The shared structure A implementation ensures that only one copy of the same parameters (such as file descriptor, priority, opcode, etc.) is needed. Since multiple requests use the same contiguous buffer and have the same base address, only one copy of the base address information needs to be retained.

[0043] Implementation of Shared Structure B: For different offset address parameters, this invention encapsulates the offset address information of multiple requests within a shared structure B. Structure B includes five main fields: num (number of requests), in_use (usage status), flag (commit strategy), error (error message), and addr (file or block device address). During execution, if a specific request encounters an error, it can either return directly or be marked for later processing, allowing the process to continue execution.

[0044] Offset Address Array Queue: A queue storing non-contiguous offset addresses is included in shared structure B. This queue records the offset addresses of merged random requests, and its depth indicates the degree of merging. Considering latency and device processing capabilities, a queue depth between 64 and 128 is recommended for optimal performance.

[0045] The I / O interface supporting random I / O merging implemented in this embodiment is compatible with traditional asynchronous I / O interface designs. For example, in the libaio interface, the io_setup and io_getevents functions remain unchanged, while a new request submission interface, io_submit_new, is added. This new interface adds a parameter hitpp, which points to a shared structure B, while shared structure A inherits from the original libaio iocb structure. By inheriting the original structure, kernel operations can be simplified and kernel processing efficiency improved. It should be noted that the specific implementations of shared structure A and shared structure B here are only optional embodiments of the present invention and should not be construed as the sole limitation of the present invention. In other embodiments of the present invention, shared structure A may also inherit other suitable existing structures in the operating system kernel or adopt other custom structures.

[0046] In this embodiment, the parameters of the random I / O request are encapsulated into shared structure A and shared structure B, so that the parameters of the random I / O request exist in the form of a structure. After being submitted to the kernel, they are linked to the same structure through pointers for shared access.

[0047] In this embodiment, the IO stack in the operating system kernel processes the same and different parameters of random IO requests through batch operations and non-batch operations. During the execution of batch operations and non-batch operations, the parameters in shared structure A and shared structure B will be modified accordingly, and the relevant modifications will also be passed to the device driver layer.

[0048] After the pseudo-sequential request is passed to the driver layer, this embodiment distributes the pseudo-sequential request into random I / O requests at the device driver layer, and generates a corresponding device command for each random I / O request and sends it to the storage device. This ensures compatibility with the underlying device. That is to say, the storage device does not need to be modified. Optionally, in this embodiment, the storage device and the operating system use the NVMe protocol. Accordingly, at the device driver layer, a unique NVMe command is created for each random I / O request. After creating NVMe commands for all random requests contained in the pseudo-sequential request and submitting them, the device driver updates the queue information and notifies the device to process it.

[0049] Block buffers enable the kernel to perform batch processing on the entire buffer. Since the buffer needs to be redistributed to each random I / O request, optionally, in this embodiment, when the pseudo-sequential requests are distributed into random I / O requests at the device driver layer, a PRP (Physical Region Page) mapping using DMA (Direct Memory Access) is used. Each request is allocated a contiguous 4K page using this mapping. The kernel manages the buffer at the page granularity; although contiguous user-space buffers are allocated for pseudo-sequential requests, they will still be divided into separate pages.

[0050] The device driver for submitting merged random requests implemented in this embodiment adopts a pseudo-ordered request callback strategy with fast resource release and delayed callback, specifically including: When the storage device completes a random I / O request After processing and notifying the CPU via the terminal, the device driver layer determines whether the storage device has completed processing all random I / O requests in the pseudo-sequential request. If not, the random I / O requests are released. The non-shared resources occupied are released and the interrupt handling process is exited. This is called the fast resource release phase. In this phase, the pseudo-sequential request will not return immediately, but will wait for other random I / O requests to complete. If so, the callback process will be executed uniformly for each random I / O request in the pseudo-sequential request. This is called the delayed callback phase.

[0051] The pseudo-ordered request callback strategy described above for fast resource release and delayed callback is as follows: Figure 5 As shown. To facilitate tracking the completion status of random I / O requests in pseudo-sequential requests, as an optional implementation method, as follows... Figure 5 As shown, this embodiment uses a bitmap in the device driver layer to track the completion status of each random I / O request in the pseudo-sequential request. In this bitmap, each bit corresponds to a random I / O request. If a random I / O request is completed, the corresponding bit is set to 1; otherwise, the corresponding bit is set to 0. When all bits in the bitmap become 1, it indicates that all random I / O requests in the pseudo-sequential request have been processed.

[0052] In other embodiments of the present invention, a counter can also be used in the device driver layer to track the completion status of random IO requests in the pseudo-sequential request. The counter value is incremented by 1 for each completed random IO request. When the counter value is the same as the number of random IO requests in the pseudo-sequential request, it indicates that all random IO requests in the pseudo-sequential request have been processed.

[0053] In this embodiment, based on the execution frequency and execution steps of the operation, the entire execution flow of the pseudo-sequential request is as follows: The pseudo-ordered request is submitted from user space to the kernel, and the parameters are copied and passed through a shared structure. Batch operations (such as file descriptor retrieval and flag processing) are executed once in all requests to reduce repetitive operations; for non-batch operations (such as operations related to offset addresses), the kernel processes the offset address separately for each request and ensures that the interaction between the file system layer and the block device is completed normally. The device driver layer processes pseudo-sequential requests in a distributed manner, breaking each request down into independent device commands and submitting them to the device. The execution process of the request is completed in stages through rapid resource release and delayed callbacks, ensuring efficient IO processing and resource release.

[0054] Example 2: An operating system that supports random I / O merging, such as Figure 6 As shown, it includes: The IO interface module is used in user space to retain only one copy of the same parameters for multiple random IO requests to be merged and record them in a shared structure A, and to record the different parameters of the multiple random IO requests to be merged in a shared structure B, thereby merging the multiple random IO requests to be merged into a pseudo-ordered request; and then sending the pseudo-ordered request to the system kernel. The IO stack module is used to perform corresponding operations on pseudo-sequential requests sequentially at the kernel interface layer, file system layer, and block layer of the system kernel. When performing corresponding operations at each layer, batch operations are executed only once, while non-batch operations are executed once for each random IO request. Batch operations are operations performed on parameters in shared structure A, while non-batch operations are operations performed on parameters in shared structure B. And the device driver module, which is used in the device driver layer of the kernel system to split the pseudo-sequential requests processed by the kernel interface layer, file system layer and block layer into random IO requests, generate corresponding device commands, and submit each device command to the storage device.

[0055] In this embodiment, the specific implementation methods of each module can be referred to the description in Embodiment 1 above, and will not be repeated here.

[0056] Example 3: A storage system that supports random I / O merging includes: a storage device and an operating system that supports random I / O merging as provided in Embodiment 2 above.

[0057] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An operating system I / O processing method supporting random I / O merging, characterized in that, include: In user space, for the same parameters of the multiple random IO requests to be merged, only one copy of the parameters is kept and recorded in the shared structure A, and the different parameters of the multiple random IO requests to be merged are recorded in the shared structure B respectively, thereby merging the multiple random IO requests to be merged into pseudo-sequential requests. The pseudo-sequence request is sent to the system kernel; The corresponding operations are executed sequentially on the pseudo-sequential requests at the kernel interface layer, file system layer, and block layer of the system kernel. When executing the corresponding operations at each layer, batch operations are executed only once, and non-batch operations are executed once for each random IO request. Batch operations are operations performed on the parameters in the shared structure A, and non-batch operations are operations performed on the parameters in the shared structure B. In the device driver layer of the kernel system, the pseudo-sequential requests processed by the kernel interface layer, the file system layer and the block layer are split into random I / O requests, and corresponding device commands are generated and submitted to the storage device.

2. The operating system I / O processing method supporting random I / O merging as described in claim 1, characterized in that, Also includes: When the storage device completes a random I / O request After the processing is completed and the CPU is notified via the terminal, the device driver layer determines whether the storage device has finished processing all random I / O requests in the pseudo-sequential request. If so, a callback process is executed uniformly for each random I / O request in the pseudo-sequential request; otherwise, the random I / O requests are released. The non-shared resources occupied are removed, and the interrupt handling process is exited.

3. The operating system I / O processing method supporting random I / O merging as described in claim 2, characterized in that, Also includes: In the device driver layer, a bitmap is used to track the completion status of each random I / O request in the pseudo-sequential request, or a counter is used to track the completion status of random I / O requests in the pseudo-sequential request.

4. The operating system I / O processing method supporting random I / O merging as described in any one of claims 1 to 3, characterized in that, Also includes: In user space, allocate contiguous shared address buffers for multiple random I / O requests to be merged.

5. The operating system I / O processing method supporting random I / O merging as described in claim 4, characterized in that, At the device driver layer, pseudo-sequential requests are split into random I / O requests, including: The shared address buffer is divided into fixed-size cache pages, and one cache page is allocated for each random I / O request.

6. The operating system I / O processing method supporting random I / O merging as described in claim 4, characterized in that, The fields in the shared structure B include: request count num, usage status in_use, submission strategy flag, error message error, and offset address array queue; Wherein, the request quantity num is used to record the number of random I / Os merged into pseudo-sequential requests, the use status in_use is used to indicate whether random I / O merging has been performed, the submission policy flag is used to indicate the interface type for submitting requests to the system kernel, the error information error is used to record the error information returned by the storage device during the execution of random I / O requests, and the offset address array queue is used to record the offset address of each random I / O request.

7. The operating system I / O processing method supporting random I / O merging as described in any one of claims 1 to 3, characterized in that, The shared structure A inherits from the original iocb structure of libaio.

8. An operating system supporting random I / O merging, characterized in that, include: The IO interface module is used in user space to retain only one copy of the same parameters for multiple random IO requests to be merged and record them in a shared structure A, and to record the different parameters of the multiple random IO requests to be merged into a shared structure B, thereby merging the multiple random IO requests to be merged into a pseudo-sequential request. The pseudo-sequence request is sent to the system kernel; The IO stack module is used to perform corresponding operations on the pseudo-sequential requests sequentially in the kernel interface layer, file system layer and block layer of the system kernel; when performing corresponding operations in each layer, batch operations are executed only once, and non-batch operations are executed once for each random IO request; batch operations are operations performed on the parameters in the shared structure A, and non-batch operations are operations performed on the parameters in the shared structure B; And a device driver module, used in the device driver layer of the kernel system to split the pseudo-sequential requests processed by the kernel interface layer, the file system layer and the block layer into random IO requests, generate corresponding device commands, and submit each device command to the storage device.

9. A storage system supporting random I / O merging, characterized in that, include: The storage device and the operating system supporting random I / O merging as described in claim 8.

Citation Information

Cited By

  • Block device layer IO scheduling method and system for artificial intelligence model training

    CN122240335A