Parallel read-write acceleration device based on distributed file system client cache
By introducing a parallel read/write acceleration device into the distributed file system client cache, and utilizing the write cache module and parallel thread pool technology, the problems of high latency and low bandwidth in single-stream, small I/O scenarios are solved, achieving efficient data transmission and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TOYOU FEIJI ELECTRONICS
- Filing Date
- 2026-02-03
- Publication Date
- 2026-05-12
AI Technical Summary
Existing distributed file systems suffer from performance bottlenecks in single-stream, low-IO scenarios, including high latency and low bandwidth limits. These bottlenecks are mainly due to the deep coupling between metadata and data paths, network protocol stack overhead, and the lack of intelligent aggregation mechanisms.
A parallel read/write acceleration device based on distributed file system client caching is adopted. Data is written to the chunk linked list and marked as dirty through the write cache module. The write shared thread pool is used to write to the distributed storage cluster in parallel, and the read shared thread pool is used to pre-read data into the chunk. By combining concurrent write and concurrent pre-read, data aggregation and fast transmission are achieved.
It significantly reduces the latency between writing data from the sender to reading it from the receiver, increases effective bandwidth, and improves the IO processing performance of the distributed file system, especially in single-stream, low-IO scenarios.
Smart Images
Figure CN122018809A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed storage technology, specifically to a parallel read / write acceleration device based on distributed file system client caching. Background Technology
[0002] With the explosive growth of data volume and the rapid development of technologies such as cloud computing, big data, and artificial intelligence, traditional centralized storage architectures have gradually revealed bottlenecks in terms of scalability, fault tolerance, and performance. To address the demand for efficient storage and access to massive amounts of data, distributed storage systems have emerged. These systems significantly improve system throughput and storage capacity by distributing data across multiple nodes and utilizing network collaboration, while also enhancing data reliability and availability.
[0003] While distributed storage systems were initially designed to handle large files, data-intensive tasks, and scenarios with concurrent access from multiple clients, emerging applications such as AI training, high-frequency transaction logs, virtualized disks, and database backups exhibit drastically different workload characteristics: primarily single data streams with fine-grained I / O requests, yet demanding extremely high throughput and latency. These requirements fundamentally conflict with the architectural assumptions of traditional distributed file systems, exposing their performance bottlenecks and insufficient adaptability in high-concurrency, low-I / O scenarios.
[0004] As the most mainstream open-source distributed storage system, Ceph's client (especially when accessing object storage directly through librados) handles small I / O operations as follows (i.e., the path of a typical small I / O write request in Ceph):
[0005] Application initiates a request: The application sends a 4KB / 8KB write request to the client library (such as librbd or libcephfs).
[0006] Metadata lookup and calculation:
[0007] For CephFS: The client may need to query the Metadata Server to resolve the file path and obtain the object ID and layout information corresponding to the file.
[0008] For RBD / objects: The client maps data objects to specific PGs through hash calculations.
[0009] CRUSH calculation: The client uses the CRUSH algorithm to calculate which OSD(s) (usually master and slave OSDs) the data object should be stored on based on the PG ID.
[0010] Network communication: The client establishes a network connection with the computed master OSD (or obtains one from the connection pool) and sends IO requests to it.
[0011] Primary OSD Coordination: After receiving a write request, the primary OSD is responsible for replicating it to other replica OSDs and waiting for confirmation from all replicas.
[0012] Confirmation Return: After receiving confirmation from all replicas, the primary OSD returns a signal to the client indicating successful write.
[0013] However, in this process, the following inherent bottlenecks cause a sharp drop in small I / O performance:
[0014] Defect 1: The deep coupling between metadata and data paths leads to high latency.
[0015] Technical Description: Every small I / O request must go through the complete "metadata / CRUSH calculation -> network communication -> primary replica coordination" path. Even if the metadata is cached, the steps of CRUSH calculation and determining the target OSD cannot be omitted.
[0016] Defect 2: Network protocol stack and system call overhead become performance bottlenecks.
[0017] Technical Description: Each small I / O operation requires encapsulation / decapsulation by the kernel network protocol stack, as well as context switching between user mode and kernel mode. While using librados's async messenger provides some improvement, it does not fundamentally solve the packet-level overhead.
[0018] Defect 3: Lack of intelligent aggregation and scheduling mechanism for single-stream small I / O.
[0019] Technical Description: Traditional client-side request processing follows a pipeline pattern of "one request at a time." While Ceph supports light sorting of Op queues, it lacks the ability to perform intelligent aggregation across targets and time periods on the client side.
[0020] In summary, existing distributed file system clients, due to their traditional design, will cause the following problems in single-stream, 4K / 8K, and small I / O application scenarios:
[0021] 1. The problem of excessive latency caused by factors such as the need for metadata calculation and protocol encapsulation / decapsulation for each small I / O operation.
[0022] 2. The lack of a reasonable aggregation mechanism for small I / O leads to the problem of excessively low bandwidth limits. Summary of the Invention
[0023] To address this, this application provides a parallel read / write acceleration device based on a distributed file system client cache, in order to solve the problems of high latency and low bandwidth limit in existing distributed file systems when processing I / O requests.
[0024] To achieve the above objectives, this application provides the following technical solution:
[0025] In a first aspect, a parallel read / write acceleration device based on a distributed file system client cache, the device being capable of simultaneously reading and writing data, comprising:
[0026] The write cache module is used to receive and respond to data write requests initiated by the write client, write data into the corresponding chunk in the chunk list according to the data write request, and mark the corresponding chunk as dirty; when a chunk is full or no data is written for more than a preset time, the chunk is pushed into the write thread in the write shared thread pool.
[0027] The parallel write module is used to write data from multiple chunks to the distributed storage cluster in parallel using multiple write threads in the shared write thread pool, and to mark the corresponding chunks as clean; the distributed storage cluster is used to store data on various storage nodes.
[0028] The parallel read module is used to pre-read data into multiple chunks using multiple read threads in a shared read thread pool;
[0029] The read cache module is used to receive and respond to data read requests sent by the read client, read the corresponding data from multiple chunks according to the data read requests, and return the data to the read client.
[0030] Preferably, the memory size of the chunk in the chunk list is consistent with the memory size of the storage objects in the distributed storage cluster.
[0031] Preferably, the chunk memory size is 1M.
[0032] Preferably, the chunk includes the location information, access time, and status of the data in the chunk.
[0033] Preferably, during dirty page write-back, the parallel write module locks the corresponding chunk.
[0034] Preferably, when the parallel read module pre-reads data into multiple chunks, the pre-read window is initially 2 chunks in size; if active pre-reading is triggered again, the parallel read module sets the pre-read window to 4 chunks in size.
[0035] Preferably, when the read cache module reads the corresponding data from multiple chunks according to the data read request, if there is no corresponding data in the chunk, it obtains an idle chunk, associates the idle chunk with the chunk_space object and index of the file in the data read request, inserts it into the hash table, and initiates a low-level I / O request to obtain data from the distributed storage set into the idle chunk according to the low-level I / O request.
[0036] Preferably, the parallel read module dynamically monitors the access pattern of the read client: when it detects that the read client is accessing sequentially, it will not only request the chunk data of the current access, but also asynchronously pre-read the subsequent chunk data, and dynamically adjust the number of pre-read chunks according to the sequential read rate.
[0037] Preferably, after the read client finishes consuming a chunk, the parallel read module will automatically pre-read a chunk to supplement the end.
[0038] Secondly, a parallel read / write acceleration method based on a distributed file system client cache, the method being capable of simultaneously reading and writing data, including:
[0039] Receive and respond to data write requests initiated by write clients, write data into the corresponding chunk in the chunk list according to the data write request, and mark the corresponding chunk as dirty; when a chunk is full or no data is written for more than a preset time, push the chunk into the write thread in the write shared thread pool;
[0040] Multiple write threads in a shared write thread pool write data from multiple chunks to a distributed storage cluster in parallel, and mark the corresponding chunks as clean; the distributed storage cluster is used to store data on various storage nodes.
[0041] Data is pre-read into multiple chunks using multiple read threads in a shared read thread pool;
[0042] Receive and respond to data read requests sent by the read client, read the corresponding data from multiple chunks according to the data read requests, and return the data to the read client.
[0043] Compared with the prior art, this application has at least the following beneficial effects:
[0044] Based on further analysis and research of existing technical problems, this application provides a parallel read / write acceleration device based on a distributed file system client cache. This device can simultaneously read and write data, comprising: a write cache module for writing data into chunks according to write requests and marking them as dirty; when a chunk is full or no data is written for a preset time, the chunk is pushed into a write shared thread pool; a parallel write module for writing data to the distributed storage cluster in parallel using write threads in the write shared thread pool; a parallel read module for pre-reading data into multiple chunks using multiple read threads in the read shared thread pool; and a read cache module for reading corresponding data from multiple chunks according to read requests and returning it to the read client. This application aggregates small I / O requests through a caching mechanism and combines concurrent write (disk flushing) and concurrent pre-reading, significantly reducing the latency between data writing from the sending end and reading from the receiving end, and improving effective bandwidth. Attached Figure Description
[0045] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).
[0046] Figure 1 A schematic diagram of a parallel read / write acceleration device based on a distributed file system client cache, provided in Embodiment 1 of this application;
[0047] Figure 2 This is a schematic diagram of the structure of the chunk linked list provided in Embodiment 1 of this application;
[0048] Figure 3 The flowchart illustrates a parallel read / write acceleration method based on a distributed file system client cache, as provided in Embodiment 2 of this application. Detailed Implementation
[0049] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0050] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).
[0051] The terms used in this application, such as "upper," "lower," "left," "right," and "middle," are generally used to indicate the general relative positional relationship for the purpose of intuitive understanding by referring to the accompanying drawings, and are not absolute limitations on the positional relationship in the actual product.
[0052] Example 1
[0053] Please see Figure 1 This embodiment provides a parallel read / write acceleration device based on a distributed file system client cache. The device can perform data reading and data writing simultaneously, and includes a write cache module, a parallel write module, a distributed storage cluster, a parallel read module, and a read cache module.
[0054] The write cache module is used to receive and respond to data write requests initiated by write clients, write data to the corresponding chunk in the chunk list according to the data write request, and mark the corresponding chunk as dirty. When a chunk is full or no data is written for more than a preset time, the chunk is pushed into the write thread in the write shared thread pool.
[0055] Specifically, because traditional kernel file systems often use the kernel's page cache as a buffer, which shifts disk I / O latency to the background, when dirty pages accumulate to a certain threshold, the kernel will forcibly add the flush task to the queue, and may even block the application process initiating the write operation until enough dirty pages are written to disk. This can cause unpredictable pauses in applications, which can be fatal for latency-sensitive applications.
[0056] Therefore, this embodiment redesigns the caching mechanism. For example... Figure 2 As shown, the caching mechanism in this embodiment consists of multiple chunk lists, and the memory size of each chunk in the chunk list is consistent with the memory size of the storage objects in the distributed storage cluster. In this embodiment, the memory size of a chunk is set to 1MB.
[0057] In this embodiment, each open file is associated with a Chunk_space object, which is the management controller of the chunk cache. This object includes a hash table for high-speed lookups and a lock structure to protect the data. The lock structure is used to lock the corresponding chunk during dirty page write-backs. Furthermore, the cache for each open file consists of multiple chunks. Each chunk manages a 1MB memory space allocated from the kernel and contains structures describing the location, access time, and status of the data within the chunk.
[0058] Therefore, in this embodiment, after the write cache module writes data into its corresponding chunk, it immediately marks the corresponding chunk as DIRTY. The write operation returns success as soon as the data is written to memory, achieving a low-latency write experience.
[0059] However, dirty chunks are not immediately written back to the distributed storage cluster. Only when a dirty chunk is full or remains unwritten for more than 3ms will it be placed in the write shared thread pool. The write threads are then responsible for sending the data to the distributed storage cluster via the network library. Once the data is successfully persisted, the chunk will be marked as CLEAN.
[0060] The parallel write module is used to write data from multiple chunks to a distributed storage cluster in parallel using multiple write threads in a shared thread pool; the distributed storage cluster is used to store data on various storage nodes.
[0061] Specifically, in distributed storage systems, data is often distributed across multiple storage nodes, and each storage node contains multiple data disks. Traditional file systems typically use a single thread to perform write operations in a single file write scenario. This serial operation cannot fully utilize the multi-node performance of distributed file systems. In this embodiment, a write-shared thread pool is designed, which includes multiple parallel write threads responsible for writing chunks to the distributed storage cluster.
[0062] In this embodiment, the chunk is locked during dirty page write-back to prevent it from being written to by others. Because it is a distributed file system client, each write client will acquire the write lock of the corresponding file, and only one person can acquire the write lock of the same file at the same time.
[0063] The parallel read module is used to pre-read data into multiple chunks using multiple read threads in a shared read thread pool.
[0064] Specifically, in a distributed storage cluster, data is stored in the form of objects across various storage nodes, thus, an object is the smallest unit of storage. Therefore, during pre-reading, a pre-read of one object size is performed, and a parallel processing mechanism is also used here, pre-reading multiple objects simultaneously during sequential read and write operations.
[0065] The pre-read strategy of the parallel read module is as follows: the pre-read window is initially 2 chunks in size. In sequential read scenarios, after each chunk is consumed, one chunk is automatically pre-read. If active pre-reading is triggered again, the pre-read window is set to 4 chunks, and the window size follows the same pattern.
[0066] More specifically, the parallel read module dynamically monitors the access patterns of read clients: when sequential access is detected, it not only requests the chunk data for the current access but also asynchronously pre-reads subsequent chunk data, dynamically adjusting the number of pre-read chunks based on the sequential read rate. After each read client consumes a chunk, the parallel read module automatically pre-reads one chunk to append to the end.
[0067] The read cache module is used to receive and respond to data read requests sent by the read client, read the corresponding data from multiple chunks according to the data read request, and return it to the read client.
[0068] Specifically, when the read cache module reads the corresponding data from multiple chunks according to the data read request, if there is no corresponding data in the chunk, it obtains an idle chunk, associates the idle chunk with the chunk_space object and index of the file in the data read request, inserts it into the hash table, and initiates an underlying I / O request. According to the underlying I / O request, it obtains the data from the distributed storage set into the idle chunk.
[0069] In this embodiment, when the application (i.e., the read client) initiates a file read request, the device first queries the chunk cache of the file area where the requested data is located; if the data is already in the chunk cache, the data in memory is directly copied to the user space, and the process is completely without I / O operations; if the data is not in the cache, a pre-read operation is performed.
[0070] The pre-read operation is as follows: First, a free chunk is obtained, and the chunk is associated with the file's chunk_space object and the corresponding index, and inserted into the hash table; then, an underlying I / O request is initiated to read data from the storage device into the chunk. Once the data is ready, the UPDATE flag is set to complete the request.
[0071] This embodiment integrates an intelligent adaptive prefetch algorithm, which dynamically monitors the application's access patterns. When sequential access is detected, the device not only requests the chunk data for the current access but also asynchronously prefetches subsequent chunk data. Based on the sequential read rate, the number of prefetched chunks is dynamically adjusted. Simultaneously, after the application consumes a chunk, it automatically prefetches one chunk to supplement the end, ensuring that the amount of prefetched data remains above a certain level.
[0072] The parallel read / write acceleration device based on distributed file system client caching provided in this embodiment has the following advantages:
[0073] 1. Significantly improves the read-write performance of the distributed file system: By caching and aggregating small I / O, fast concurrent disk flushing, and concurrent pre-reading, the latency of writing a datagram from end A to reading it from end B can be greatly reduced.
[0074] 2. High-efficiency resource utilization: Through concurrency technology, it supports parallel writing to multiple nodes of distributed storage simultaneously, giving full play to the resource utilization efficiency of distributed storage.
[0075] In summary, the parallel read / write acceleration device based on distributed file system client caching provided in this embodiment fully leverages the parallel read / write advantages of distributed storage by utilizing distributed file system client caching. It aggregates small I / O requests through a caching mechanism and combines concurrent writes (disk flushing) and concurrent pre-reading, significantly reducing the latency between data writing from the sending end and reading from the receiving end, and improving effective bandwidth. This solves the problems of high latency and low bandwidth in traditional distributed file systems under single-stream, small I / O scenarios. Compared to the serial read / write of current mainstream file systems, it greatly improves I / O processing performance in single-stream scenarios for sequential read / write.
[0076] Example 2
[0077] Please see Figure 3 This embodiment provides a parallel read / write acceleration method based on distributed file system client caching. The method can perform data reading and data writing simultaneously, including:
[0078] Receive and respond to data write requests initiated by write clients, write data into the corresponding chunk in the chunk list according to the data write request, and mark the corresponding chunk as dirty; when a chunk is full or no data is written for more than a preset time, push the chunk into the write thread in the write shared thread pool;
[0079] Multiple write threads in a shared write thread pool write data from multiple chunks to a distributed storage cluster in parallel, and mark the corresponding chunks as clean; the distributed storage cluster is used to store data on various storage nodes.
[0080] Data is pre-read into multiple chunks using multiple read threads in a shared read thread pool;
[0081] Receive and respond to data read requests sent by the read client, read the corresponding data from multiple chunks according to the data read requests, and return the data to the read client.
[0082] The method provided in this embodiment will be further explained below using a typical file read-and-write scenario (i.e., parallel read-and-write acceleration).
[0083] Suppose that distributed file system client A continuously writes 4KB of data to the file / data / test.log from position 0; client B reads the data in the file / data / test.log for analysis.
[0084] The write process includes:
[0085] S101: A write request enters the kernel via a call (such as write), and the VFS layer locates the inode corresponding to / data / test.log and its associated chunk_space object;
[0086] S102: Obtain distributed write lock permission;
[0087] S103: Find the corresponding chunk, index = 0 / 1048576 = 0, find the chunk at position 0 in chunk_space;
[0088] S104: If the chunk does not exist, obtain a new one from the free chunk;
[0089] S105: If the chunk is temporarily unwritable, then wait;
[0090] S106: Write data to the chunk, mark the chunk as not dirty. If the chunk is already full, push the chunk directly into the write thread pool and return; otherwise, return directly.
[0091] Write thread processing flow:
[0092] S201: Retrieve the chunk to be processed from the queue;
[0093] S202: Calculate the objects that need to be written to the distributed storage using the inode to which the chunk belongs and the write offset information;
[0094] S203: Construct a write request and send it to the distributed storage cluster.
[0095] Cache processing thread:
[0096] S301: Check for the existence of dirty chunks every 3ms. If a dirty chunk exists, push the chunk into the write thread pool and quickly flush it to disk.
[0097] S302: Detect the access time of the clean chunk. If no one accesses it for more than 10ms, the chunk will be reclaimed.
[0098] Pre-read processing flow:
[0099] S401: Client B's read request enters the system kernel through a system call (such as read), reads data starting from 0, and the VFS locates the inode corresponding to / data / test.log and its associated chunk_space;
[0100] S402: Find the corresponding chunk, index = 0 / 1048576 = 0, find the chunk at position 0 in chunk_space.
[0101] S403: Cache hit: Directly read the data in the cache into the user buffer.
[0102] S404: Cache miss + read-ahead:
[0103] S4041: Construct a read request, pre-read 2 chunks of data (2M) starting from position 0, and put the data into the newly allocated chunk after the data is returned;
[0104] S4042: Read the corresponding data from the chunk into the user buffer;
[0105] S4043: If the chunk at index=0 has been completely read, before consuming the chunk at index=1, construct an asynchronous request to pre-read the data from the chunk at index=2.
[0106] For details on the specific implementation of a parallel read / write acceleration method based on a distributed file system client cache, please refer to the specific limitations of each module in the parallel read / write acceleration device based on a distributed file system client cache mentioned above, which will not be repeated here.
[0107] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.
Claims
1. A parallel read / write acceleration device based on distributed file system client caching, characterized in that, The device is capable of simultaneously reading and writing data, including: The write cache module is used to receive and respond to data write requests initiated by the write client, write data into the corresponding chunk in the chunk list according to the data write request, and mark the corresponding chunk as dirty; when a chunk is full or no data is written for more than a preset time, the chunk is pushed into the write thread in the write shared thread pool. The parallel write module is used to write data from multiple chunks to the distributed storage cluster in parallel using multiple write threads in the shared write thread pool, and to mark the corresponding chunks as clean; the distributed storage cluster is used to store data on various storage nodes. The parallel read module is used to pre-read data into multiple chunks using multiple read threads in a shared read thread pool; The read cache module is used to receive and respond to data read requests sent by the read client, read the corresponding data from multiple chunks according to the data read requests, and return the data to the read client.
2. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, The memory size of a chunk in the chunk list is consistent with the memory size of the storage objects in the distributed storage cluster.
3. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, The chunk memory size is 1M.
4. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, The chunk includes the location information, access time, and status of the data within the chunk.
5. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, During dirty page writeback, the parallel write module locks the corresponding chunk.
6. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, When the parallel read module pre-reads data into multiple chunks, the pre-read window is initially 2 chunks in size; if active pre-reading is triggered again, the parallel read module sets the pre-read window to 4 chunks in size.
7. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, When the read cache module reads corresponding data from multiple chunks according to the data read request, if there is no corresponding data in the chunk, it obtains an idle chunk, associates the idle chunk with the chunk_space object and index of the file in the data read request, inserts it into the hash table, and initiates a low-level I / O request to obtain data from the distributed storage set into the idle chunk according to the low-level I / O request.
8. The parallel read / write acceleration device based on distributed file system client caching according to claim 1, characterized in that, The parallel read module dynamically monitors the access patterns of the read clients: when sequential access by the read clients is detected, it not only requests the chunk data for the current access, but also asynchronously pre-reads the subsequent chunk data, and dynamically adjusts the number of pre-read chunks according to the sequential read rate.
9. The parallel read / write acceleration device based on distributed file system client caching according to claim 8, characterized in that, After each chunk is consumed by the read client, the parallel read module automatically pre-reads a chunk to supplement the end.
10. A method for accelerating parallel read and write operations based on client caching in a distributed file system, characterized in that, The method can simultaneously read and write data, including: Receive and respond to data write requests initiated by write clients, write data into the corresponding chunk in the chunk list according to the data write request, and mark the corresponding chunk as dirty; when a chunk is full or no data is written for more than a preset time, push the chunk into the write thread in the write shared thread pool; Multiple write threads in a shared write thread pool write data from multiple chunks to a distributed storage cluster in parallel, and mark the corresponding chunks as clean; the distributed storage cluster is used to store data on various storage nodes. Data is pre-read into multiple chunks using multiple read threads in a shared read thread pool; Receive and respond to data read requests sent by the read client, read the corresponding data from multiple chunks according to the data read requests, and return the data to the read client.