Distributed file system based on data processing unit (DPU)

Abstract

This invention relates to a distributed file system based on a Data Processing Unit (DPU), comprising: a compute node, a DPU, and a storage node; the compute node includes a client request interface and host shared memory; the DPU includes a file request processing module, a data path management module, a metadata index management module, and a metadata memory management module; the storage node includes remote shared memory; the file request processing module employs a one-sided polling mechanism; the data path management module employs single-operation zero-copy transmission; the metadata index management module employs a core-based index structure; and the metadata memory management module employs a two-level allocation structure. This invention reduces the CPU overhead of the compute node by employing a one-sided polling mechanism, a core-based index structure, and a two-level allocation structure in the DPU, while simultaneously constructing an efficient data path to overcome the DPU memory bottleneck.

Description

Technical Field

[0001] This invention relates to the field of distributed file system technology, and more specifically, to a distributed file system based on a data processing unit (DPU). Background Technology

[0002] Distributed file systems often employ a storage separation architecture. Existing storage separation architectures primarily consist of compute nodes and storage nodes, characterized by the ability to independently scale compute and storage resources, thereby enhancing system flexibility and performance. Compute nodes run file system clients, mainly used to run applications, virtual machines (VMs), and containers. Applications access file system services through these clients. Storage nodes are primarily responsible for running file system storage services, ensuring data is persistently stored on storage devices such as solid-state drives (SSDs). They receive and process file system requests from compute nodes, providing data storage and retrieval services. While this architecture enhances flexibility, as application performance demands increase, compute node CPU resources become increasingly scarce. For example, in some systems, a large number of CPU cores are occupied by file system clients, impacting the deployment of other applications. Currently, there are solutions using Data Processing Units (DPUs) to offload some file system functions, but these solutions also have drawbacks. On the one hand, they do not fully utilize DPU resources, only partially offloading file system functions, failing to fully leverage the DPU's capabilities. This means that cache management by file system clients still increases host CPU overhead. On the other hand, they do not consider the limitations of the DPU itself, failing to address the DPU's limited memory bandwidth. Existing data path designs cannot fully utilize network bandwidth, impacting overall performance. Summary of the Invention

[0003] The purpose of this invention is to overcome the problems of excessive CPU resource consumption on the host by existing distributed file system clients and the limitation of existing data path design by DPU memory bandwidth. It provides a distributed file system based on a data processing unit (DPU), which designs scalable metadata management in the DPU to avoid CPU overhead on the host, while constructing an efficient data path and breaking the memory bottleneck of the DPU.

[0004] The technical solution of the present invention is as follows:

[0005] A distributed file system based on a Data Processing Unit (DPU) includes computing nodes, a DPU, and storage nodes; the computing node includes a client request interface and host shared memory; the DPU includes a file request processing module, a data path management module, a metadata index management module, and a metadata memory management module; the storage node includes remote shared memory.

[0006] The file request processing module adopts a request transmission method based on one-sided polling, reads file requests sent from the client request interface to the host shared memory, processes valid file requests when they are found, and returns the file request processing result to the host shared memory after the file request is processed.

[0007] The data path management module grants the DPU access to the host shared memory during the initialization phase and establishes an RDMA connection between the DPU and the storage node. In subsequent communication processes, it controls the DPU to use the RDMA connection to perform data interaction between the host shared memory and the remote shared memory.

[0008] The metadata index management module adopts a core-based index architecture, allocating an index node (Inode) to each core. When a file request needs to be parsed, the filename is mapped to an Inode address, and the content stored in the Inode is accessed through that address.

[0009] The metadata memory management module adopts a two-layer allocation structure including a global allocator and a local allocator to realize the allocation and reclamation of Inode memory space; each core is allocated a local allocator, and the local allocator requests pre-allocated memory from the global allocator.

[0010] Preferably, in the file request processing module, the file reading request is processed in batch mode, specifically including:

[0011] The DOCA software stack is used to register host shared memory in the compute node memory for DPU access. The client request interface collects file requests into the host shared memory as batch requests. The DPU uses a queue to perform DMA one-sided reads to obtain batch requests from the host shared memory.

[0012] Preferably, the batch processing request includes a flag bit for controlling the one-sided polling mechanism; when the flag bit is 1, the read is valid and the request is processed; when the flag bit is 0, DMA reading continues; when the request is processed in the DPU, the result is written to the compute node via DMA and the flag bit is changed to 0.

[0013] Preferably, during the initialization phase, the host grants the DPU access to shared memory and establishes an RDMA connection between the DPU and the storage node, specifically including:

[0014] During the initialization phase, the DOCA software stack grants the DPU access permissions to the host shared memory; the DPU registers the RDMA region using the host shared memory, and the storage node registers the RDMA region using the remote shared memory; the DPU uses one-sided RDMA through the RDMA queue to perform data interaction between the host shared memory and the remote shared memory.

[0015] Preferably, in the metadata index management module, the request processing core is divided according to the ID generated by hashing the file name.

[0016] Preferably, the request processing core is divided based on the ID generated by hashing the filename, specifically including:

[0017] The core to be processed is determined by taking the remainder of the ID and the number of cores.

[0018] Preferably, the metadata memory management module, specifically the reclamation of Inode memory space, includes:

[0019] When the local allocator releases more memory than the threshold, the local allocator returns the released memory to the global allocator.

[0020] Preferably, after the local allocator releases more memory than a threshold, the local allocator returns the released memory to the global allocator, specifically including:

[0021] When an Inode in the local allocator is freed, it is first kept in the local allocator's recycling list. When an Inode needs to be allocated, the memory blocks in the recycling list are used first. Once the recycling list exceeds the threshold, the local allocator will return the memory blocks in the recycling list to the free list in the global allocator in batches.

[0022] Preferably, the storage node further includes a server-side persistence interface for writing data stored in remote shared memory to a hard disk.

[0023] The beneficial effects of this invention are as follows:

[0024] (1) The file request processing module of the present invention is based on the request transmission design of one-sided polling. The DPU interacts with the host client through this design. One-sided polling uses the DOCA software stack to register a DPU-accessible area (i.e. host shared memory) in the memory of the computing node. The host client will collect file requests into the host shared memory as batch processing requests. The DPU performs DMA one-sided read to obtain batch processing requests from the host shared memory, reducing the CPU overhead of the computing node.

[0025] (2) In order to accelerate the running speed of the index, the metadata index management module of the present invention adopts a core-based index architecture, allocating an index to each core to avoid data competition;

[0026] (3) When a file is created, the Inode needs to be allocated space. The allocation by multiple cores using the global allocator will also generate a lot of overhead. In order to reduce the allocation overhead, the metadata memory management module of this invention adopts a two-layer design, including a global allocator and a local allocator. The global allocator is used to manage a large memory space. The local allocator pre-allocates a large block of memory to the global allocator. The local allocator allocates memory per core. When each core needs to allocate an Inode data block, it obtains memory from the local allocator. This can reduce the concurrency overhead of memory allocation. Once the memory released by the local allocator exceeds the threshold, the local allocator will also return the released memory to the global allocator to realize memory reclamation.

[0027] (4) The existing DPU data path first reads the host data into the DPU memory through DMA, and then writes it to the remote shared memory in the server through RDMA to complete the host-to-server data writing. This writing method is limited by the DPU's memory bandwidth, and the two data transmission operations also increase the DPU's processing overhead. In order to break the memory bandwidth bottleneck, the data path management module of this invention is designed with single-operation zero-copy transmission. With the help of the DOCA software stack, after the DPU and the server establish an RDMA connection, the data transmission between the host shared memory and the remote shared memory can be performed in a single operation, eliminating the overhead of data passing through the DPU memory.

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the distributed file system based on the data processing unit (DPU) of the present invention is not limited to the embodiments. Attached Figure Description

[0029] Figure 1 This is a framework diagram of the distributed file system according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram illustrating the file request format and one-sided polling mechanism of an embodiment of the present invention;

[0031] Figure 3 This is a diagram illustrating the unoptimized data path in existing technologies.

[0032] Figure 4 This is a schematic diagram of the optimized data path according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram illustrating the unoptimized metadata management in existing technologies.

[0034] Figure 6 This is a schematic diagram of the optimized metadata management according to an embodiment of the present invention. Detailed Implementation

[0035] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0036] Terminology Explanation:

[0037] (1) Data Processing Unit (DPU): This is a hardware component that has received widespread attention in the current field of computer technology. This invention can use NVIDIA's BlueField-3 DPU. This DPU contains an ARM SoC (System-on-a-Chip) CPU with a core frequency of 2.0GHz, 16 cores, 16MB of integrated shared L3 cache, and 32GB of DRAM (memory). The DPU also has powerful network and data processing capabilities, with 2×200Gbps ports and some data processing accelerators;

[0038] (2) DOCA software stack is a software library provided by NVIDIA for its DPU series, providing DPU-specific function interfaces for programming languages; DOCA includes many function implementations on the DPU, such as DMA one-sided read and write, RDMA one-sided read and write, memory licensing, data compression, data digest, etc.

[0039] (3) DOCA provides new support for RDMA, mainly enabling the DPU to use both host shared memory and remote shared memory for RDMA. In a typical RDMA hardware and software environment, memory interactions during RDMA can only occur between the two parties establishing the RDMA connection. For example, when the DPU establishes an RDMA connection with the server, data can only be transferred between the DPU's memory and the server's memory. However, DOCA provides a new mechanism to obtain access to host memory via PCIe. Although the RDMA connection is established between the DPU and the server, the DPU can still use host memory to transfer data with the server's memory.

[0040] like Figure 1 As shown, the present invention discloses a distributed file system based on a Data Processing Unit (DPU), comprising: a computing node 1 (host), a data processing unit 2 (DPU), and a storage node 3 (server); the computing node 1 includes a client request interface 11 and host shared memory 12; the data processing unit 2 (DPU) includes a file request processing module 21, a data path management module 22, a metadata index management module 23, and a metadata memory management module 24; the storage node 3 includes remote shared memory 31 and a server persistence interface 32.

[0041] The file request processing module 21 adopts a request transmission method based on one-sided polling, reads the file requests sent from the client request interface 11 to the host shared memory 12, processes the valid file requests when they are found, and returns the file request processing result to the host shared memory 12 after the file request processing is completed.

[0042] The data path management module 22 grants the DPU access rights to the host shared memory 12 during the initialization phase and establishes an RDMA connection between the DPU and the storage node 3; in subsequent communication processes, it controls the DPU to use the RDMA connection to perform data interaction between the host shared memory 12 and the remote shared memory 31.

[0043] The metadata index management module 23 adopts a core-based index architecture, allocating an index node (Inode) to each core. When a file request needs to be parsed, the filename is mapped to an Inode address, and the content stored in the Inode is accessed through that address.

[0044] The metadata memory management module 24 adopts a two-layer allocation structure including a global allocator and a local allocator to realize the allocation and reclamation of Inode memory space; each core is allocated a local allocator, and the local allocator requests pre-allocated memory from the global allocator.

[0045] Specifically, in this invention, the host acts as a computing node, internally running a client request interface and host shared memory. The client request interface handles requests from user programs, while the host shared memory is used to transfer file data. The DPU connects to the host via PCIe, and its four functional modules run within the DPU processor. The server, acting as a storage node, connects to the DPU via a network. The server-side persistence interface ensures data persistence in the file system, preventing data loss, while the remote shared memory is also used for transferring file data.

[0046] like Figure 2The diagram illustrates the specific file request format and one-sided polling mechanism in the file request processing module of this invention. Request format processing is implemented in the client request interface. Each file request consists of a series of fields, including request sequence number, file operation, file name, host address, remote address, and file size. To accelerate file request transmission, this invention packages multiple file requests together to form a batch request. The batch request consists of a flag bit, the number of requests, and the actual stored file requests. The flag bit is mainly used to control the one-sided polling mechanism. First, the DPU uses DMA to read the batch request. When the flag bit is 1, it indicates that the client has prepared the request, the read is valid, and request processing is required. When the flag bit is 0, it indicates that the client has not yet prepared the data, and DMA reading needs to be repeated. When the request is processed within the DPU, the result is written to the client via DMA, and the flag bit is changed to 0. The client returns the result to the application by checking the flag bit. When the next file request is ready, the client modifies the flag bit to 1 and repeats the above process to complete the request transmission. Figure 2 As shown, the DPU's response to a request is recorded using a 32-bit number. Each number represents a specific action to indicate the request's execution status: 0 for success, 1 for a network error, and 2 for a memory error. If execution is successful and data is returned, the data's address is recorded in the host address field of the file request. This invention sets the maximum batch size to 31, and the maximum storage size for both requests and responses is this number; each request and response is corresponding to a specific request.

[0047] The data path management module of this invention is used to realize file data transfer between a host and a server. For example... Figure 4 As shown, existing data transmission relies on two separate transmission links, requiring two transmission operations for each data exchange. Taking a write request as an example, after metadata processing is complete, the DPU first retrieves data from the host's shared memory via DMA into its own memory, and then transfers the data from the DPU's memory to the server's shared memory via RDMA. See also... Figure 4 As shown, the implementation of the single-operation zero-copy data path in the data path management module mainly involves the collaborative work between the host, DPU, and server. Specifically, the DPU and server establish an RDMA connection. Access permissions to the host shared memory are granted to the DPU by the host through the DOCA software stack. The DPU uses the host shared memory to register RDMA-enabled regions, while the server uses remote shared memory to register RDMA-enabled regions. Finally, the DPU can use unilateral RDMA through the RDMA queue to perform data interaction between the host shared memory and the remote shared memory. It is worth noting that... Figure 4The ① "Authorizing access rights" shown only needs to be executed once during the file system initialization phase, while the subsequent ② "Interaction between host and server" can be executed using a single operation. Figure 3 The unoptimized approach shown lacks support from the DPU and DOCA software stacks, making it impossible to register the host shared memory as an RDMA-enabled area. It can only register the DPU memory as an RDMA-enabled area, thus requiring data to be relayed solely through the DPU memory. This unoptimized approach causes the limited memory bandwidth of the DPU to saturate before network bandwidth is fully utilized. In contrast, the single-operation zero-copy data path only controls data between the host shared memory and remote shared memory, without transferring data to the DPU memory. This avoids the DPU memory bandwidth becoming a bottleneck, improving data transmission efficiency.

[0048] The metadata management design of this invention includes a metadata index management module and a metadata memory management module. Metadata management uses a full-path filename index, employing a Masstree data structure as the index to enable range searches using pathnames. When managing files in this way, the key-value pairs stored internally in the index structure are (full-path filename, inode address). When processing file read / write requests, the remote shared memory address in the inode needs to be obtained through the index before file data exchange; when processing file creation or deletion, index insertion or deletion is required to maintain the index. File read / write requests involve reading the index, while file creation or deletion requires modifying the index. This data contention problem necessitates adding a concurrent access mechanism (mutex locks, read-write locks, etc.) to the index, which also incurs significant CPU overhead. To avoid data contention, the index structure of this invention adopts a core-based index architecture, allocating one index to each core. Furthermore, the request processing core is assigned based on the ID generated by hashing the filename; specifically, the core is determined by taking the ID modulo the number of cores. This method effectively avoids conflicts and contention that may arise from multiple cores using the same index structure simultaneously. Figure 5 As shown, when multiple file requests arrive simultaneously, without this core-based partitioning mechanism, multiple cores might simultaneously attempt to access and modify the same index structure, leading to data inconsistency and system performance degradation. However, by using... Figure 6 The management method shown has each core having its own independent index area and processing tasks, which can efficiently and stably handle file requests.

[0049] Memory management refers to the allocation and reclamation of innode memory space. This invention allocates a certain amount of memory space for innode storage. Each core involves innode allocation and reclamation when handling file creation and deletion. If each core shares a global allocator, concurrent access to the global allocator must be guaranteed, which incurs additional overhead. Figure 6 As shown, the memory allocation structure proposed in this invention adopts a two-layer design, including a global allocator and a local allocator. During file creation operations, memory allocation is used to allocate memory space for inodes. The global allocator is primarily responsible for managing larger memory spaces. In practical applications, such as allocating memory for an inode, a 16GB space is mapped using the mmap system call, and this space is managed by the global allocator. The global allocator is implemented using specific memory management libraries, such as Memkind. It manages this large memory space through complex internal algorithms, ensuring reasonable memory allocation and usage. The local allocator pre-allocates a 64MB memory block to the global allocator, and this allocation is done per core. When a core needs to allocate an inode, it obtains memory from the local allocator. This design ensures that each core has its own independent memory acquisition path, reducing mutual interference between different cores in memory allocation. Compared to the global allocator, the local allocator is more lightweight, capable of quickly responding to the memory needs of a core without the complex memory lookup and allocation operations required by the global allocator. Meanwhile, the local allocator interacts with the global allocator through pre-allocation and return operations to manage memory blocks. Memory block reclamation in the local allocator is implemented through a free list, ensuring the rational flow and utilization of memory resources. When an inode's space is released in the local allocator, it is not immediately returned to the global allocator but is first retained in the local allocator's reclamation list. When an inode needs allocation, memory blocks in the reclamation list are used first. Once the reclamation list exceeds a threshold, the local allocator returns the memory blocks in the reclamation list in batches to the global allocator's free list to ensure that other local allocators can still obtain memory from the global allocator.

[0050] In summary, this invention adds a Data Processing Unit (DPU) to the storage-separated architecture. The metadata management of this invention offloads metadata management from the compute nodes, saving CPU resources. Furthermore, addressing the performance limitations of the DPU, we have made specific optimizations to the aforementioned metadata management in terms of index management and memory allocation. The DPU of this invention connects to the host as an external device via a PCIe link, while communication between the DPU and the server is via an InfiniBand (IB) network. The PCIe link supports DMA transfer, while the IB network supports RDMA transfer. In existing technologies, data exchange between external devices and memory relies on the CPU, and data must pass through CPU registers, placing a heavy burden on the CPU. In this invention, the DMA controller plays a crucial role during DMA transfer. The external device, ready to transfer data, requests it from the DMA controller, which then requests bus control from the CPU. Upon successful transfer, the DMA controller directly transfers data between the peripheral device and memory according to set parameters, eliminating the need for CPU data handling. Existing network data transfer requires multiple copies and protocol stack processing between the host and remote computer, while the RDMA of this invention achieves zero-copy through a memory registration mechanism, directly reading and writing data between remote memory. It relies on specific hardware (such as InfiniBand network cards) and software protocols, and requires initialization and negotiation parameters before communication. RDMA bypasses kernel intervention, has extremely low latency, and can fully utilize network bandwidth to improve throughput.

[0051] The above embodiments are merely illustrative of the present invention and are not intended to limit the invention. Any changes or modifications to the above embodiments based on the technical essence of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A distributed file system based on a data processing unit (DPU), characterized in that, It includes compute nodes, data processing units (DPUs), and storage nodes; the compute nodes include client request interfaces and host shared memory; the data processing units (DPUs) include a file request processing module, a data path management module, a metadata index management module, and a metadata memory management module. The storage node includes remote shared memory; The file request processing module adopts a request transmission method based on one-sided polling, reads file requests sent from the client request interface to the host shared memory, and processes valid file requests when they are found. And after the file request is processed, the file request processing result is returned to the host shared memory; The data path management module, during the initialization phase, grants the host shared memory access permission to the DPU and establishes an RDMA connection between the DPU and the storage node. In subsequent communication processes, the DPU is controlled to use RDMA connection to perform data interaction between host shared memory and remote shared memory; The metadata index management module adopts a core-based index architecture, allocating an index node (Inode) to each core. When a file request needs to be parsed, the filename is mapped to an Inode address, and the content stored in the Inode is accessed through that address. The metadata memory management module adopts a two-layer allocation structure including a global allocator and a local allocator to realize the allocation and reclamation of Inode memory space; each core is allocated a local allocator, and the local allocator requests pre-allocated memory from the global allocator.

2. The distributed file system based on a data processing unit (DPU) according to claim 1, characterized in that, In the file request processing module, file read requests are processed in batch mode, specifically including: The DOCA software stack is used to register host shared memory in the compute node memory for DPU access. The client request interface collects file requests into the host shared memory as batch requests. The DPU uses a queue to perform DMA one-sided reads to obtain batch requests from the host shared memory.

3. The distributed file system based on a data processing unit (DPU) according to claim 2, characterized in that, Batch processing requests include flags for controlling the one-sided polling mechanism; when the flag is 1, the read is valid and the request is processed. When the flag is 0, continue DMA reading; when the request is processed in the DPU, write the result to the compute node via DMA and change the flag to 0.

4. The distributed file system based on a data processing unit (DPU) according to claim 1, characterized in that, During the initialization phase, the host grants shared memory access permissions to the DPU and establishes an RDMA connection between the DPU and the storage node, specifically including: During the initialization phase, the DOCA software stack grants the DPU access permissions to the host shared memory; the DPU registers the RDMA region using the host shared memory, and the storage node registers the RDMA region using the remote shared memory; the DPU uses one-sided RDMA through the RDMA queue to perform data interaction between the host shared memory and the remote shared memory.

5. The distributed file system based on a data processing unit (DPU) according to claim 1, characterized in that, In the metadata index management module, the request processing core is divided according to the ID generated by hashing the file name.

6. The distributed file system based on a data processing unit (DPU) according to claim 5, characterized in that, The request processing core is divided based on IDs derived from the hash of filenames, specifically including: The core to be processed is determined by taking the remainder of the ID and the number of cores.

7. The distributed file system based on a data processing unit (DPU) according to claim 1, characterized in that, The metadata memory management module, specifically the reclamation of Inode memory space, includes: When the local allocator releases more memory than the threshold, the local allocator returns the released memory to the global allocator.

8. The distributed file system based on a data processing unit (DPU) according to claim 7, characterized in that, When the local allocator releases more memory than a threshold, the local allocator returns the released memory to the global allocator, specifically including: When an Inode in the local allocator is freed, it is first kept in the local allocator's recycling list. When an Inode needs to be allocated, the memory blocks in the recycling list are used first. Once the recycling list exceeds the threshold, the local allocator will return the memory blocks in the recycling list to the free list in the global allocator in batches.

9. The distributed file system based on a data processing unit (DPU) according to claim 1, characterized in that, The storage node also includes a server-side persistence interface for writing data stored in remote shared memory to the hard disk.

Images