Distributed file system based on data processing unit (DPU)

Abstract

The invention relates to a distributed file system based on a data processing unit (DPU), which comprises a computing node, the DPU and a storage node, wherein the computing node comprises a client request interface and a host shared memory; the DPU comprises a file request processing module, a data path management module, a metadata index management module and a metadata memory management module; the storage node comprises a far-end shared memory; the file request processing module adopts a unilateral polling mechanism; the data path management module adopts single-operation zero-copy transmission; the metadata index management module adopts a core index structure; and the metadata memory management module adopts a double-layer distribution structure. According to the method, a unilateral polling mechanism is adopted in the DPU, the CPU overhead of the computing nodes is reduced according to a core index structure and a double-layer distribution structure, and meanwhile, an efficient data path is constructed for breaking through the memory bottleneck of the DPU.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file systems, and more specifically, to a distributed file system based on a data processing unit (DPU). Background Art

[0002] Distributed file systems often use a storage separation architecture. The existing storage separation architecture is mainly composed of computing nodes and storage nodes. Its feature is that it can achieve independent expansion of computing resources and storage resources, thereby enhancing the flexibility and performance of the system. The computing node runs the file system client, which is mainly used to run applications, virtual machines (VMs), and containers. Applications access the file system service through the file system client. The storage node is mainly responsible for running the file system storage service to ensure that data is persistently stored on storage devices such as solid-state drives (SSDs). It receives and processes file system requests from computing nodes and provides data storage and reading services for computing nodes. Although this architecture enhances flexibility, as application performance requirements grow, the CPU resources of computing nodes become increasingly scarce. For example, a large number of CPU cores in some systems are occupied by file system clients, affecting the deployment of other applications. There are also solutions to use DPU to offload some file system functions, but such solutions also have shortcomings. On the one hand, the DPU resources are not fully utilized, and only some file system functions are offloaded, which fails to fully utilize the capabilities of the DPU, so that the cache management of the file system client still increases the CPU overhead of the host; on the other hand, the limitations of the DPU itself are not considered, and the limited memory bandwidth of the DPU is not solved. The existing data path design cannot fully utilize the network bandwidth, affecting the overall performance. Summary of the invention

[0003] The purpose of the present invention is to overcome the problems that the existing distributed file system client consumes too much CPU resources on the host and the existing data path design is limited by the DPU memory bandwidth, and to provide a distributed file system based on a data processing unit DPU. Scalable metadata management is designed in the DPU to avoid the CPU overhead of the host, while constructing an efficient data path and breaking the memory bottleneck of the DPU.

[0004] The technical solution of the present invention is as follows:

[0005] A distributed file system based on a data processing unit (DPU), comprising a computing node, a data processing unit (DPU) and a storage node; the computing node comprises a client request interface and a host shared memory; the data processing unit (DPU) comprises a file request processing module, a data path management module, a metadata index management module and a metadata memory management module; the storage node comprises a remote shared memory;

[0006] The file request processing module adopts a request transmission method based on unilateral polling to read the file request sent from the client request interface to the host shared memory, and processes it when a valid file request is found; and after the file request is processed, the file request processing result is returned to the host shared memory;

[0007] The data path management module authorizes the host shared memory access rights to the DPU in the initialization phase and establishes an RDMA connection between the DPU and the storage node; in the subsequent communication process, the DPU is controlled to use the RDMA connection to perform data interaction between the host shared memory and the remote shared memory;

[0008] The metadata index management module adopts a core-based indexing architecture and allocates an index node Inode to each core; when a file request needs to be parsed, the file name is mapped to an Inode address through a mapping relationship, and then the content stored in the Inode is accessed through the address;

[0009] The metadata memory management module adopts a double-layer allocation structure including a global allocator and a local allocator to realize the allocation and recovery of Inode memory space; each core is allocated a local allocator, and the local allocator requests pre-allocated memory from the global allocator.

[0010] Preferably, in the file request processing module, the file request is read in batch processing, which specifically includes:

[0011] The DOCA software stack is used to register the host shared memory in the computing node memory for DPU access. The client request interface collects file requests into the host shared memory as batch requests. The DPU uses queue pairs to perform DMA one-way reads to obtain batch requests from the host shared memory.

[0012] Preferably, the batch request includes a flag bit for controlling the unilateral polling mechanism; when the flag bit is 1, the read is valid and the request is processed; when the flag bit is 0, DMA reading continues; when the request is processed in the DPU, the result is written to the computing node via DMA, and the flag bit is changed to 0.

[0013] Preferably, in the initialization phase, the host is authorized to grant access rights to the DPU for shared memory, and an RDMA connection between the DPU and the storage node is established, specifically including:

[0014] During the initialization phase, the DOCA software stack authorizes the DPU to access the host shared memory. The DPU registers the RDMA area using the host shared memory, and the storage node registers the RDMA area using the remote shared memory. The DPU uses one-sided RDMA through the RDMA queue to exchange data between the host shared memory and the remote shared memory.

[0015] Preferably, in the metadata index management module, the request processing core is divided according to the ID formed by hashing the file name.

[0016] Preferably, the request processing core is divided according to the ID formed by the hash of the file name, specifically including:

[0017] The core to be processed is determined by taking the ID modulo the number of cores.

[0018] Preferably, the metadata memory management module, the recovery of the Inode memory space specifically includes:

[0019] When the memory released by the local allocator exceeds the threshold, the local allocator returns the released memory to the global allocator.

[0020] Preferably, when the memory released by the local allocator exceeds a threshold, the local allocator returns the released memory to the global allocator, specifically including:

[0021] When the Inode space in the local allocator is released, it is first retained in the local allocator's recycling list. Once an Inode needs to be allocated, the memory blocks in the recycling list are used first. Once the recycling list exceeds the threshold, the local allocator will return the memory blocks in the recycling list to the free list in the global allocator in batches.

[0022] Preferably, the storage node further includes a server-side persistence interface for writing the data stored in the remote shared memory to the hard disk.

[0023] The beneficial effects of the present invention are as follows:

[0024] (1) The file request processing module of the present invention is based on a request transmission design of unilateral polling. The DPU interacts with the host client through this design. The unilateral polling uses the DOCA software stack to register a DPU-accessible area (i.e., host shared memory) in the computing node memory. The host client collects the file requests into the host shared memory as batch processing requests. The DPU performs DMA unilateral read to obtain the batch processing requests from the host shared memory, thereby reducing the CPU overhead of the computing node.

[0025] (2) In order to speed up the indexing operation, the metadata index management module of the present invention adopts a core-based indexing architecture, assigning an index to each core to avoid data competition;

[0026] (3) When a file is created, Inode needs to be allocated space. Multiple cores using a global allocator for allocation will also generate a lot of overhead. In order to reduce the allocation overhead, the metadata memory management module of the present invention adopts a two-layer design, including a global allocator and a local allocator. The global allocator is used to manage a larger memory space. The local allocator pre-allocates a large block of memory to the global allocator, and the local allocator is allocated according to each core. When each core needs to allocate an Inode data block, it obtains memory from the local allocator. This can reduce the concurrent overhead of memory allocation. Once the memory released by the local allocator exceeds the threshold, the local allocator will also return the released memory to the global allocator to achieve memory recovery.

[0027] (4) The DPU data path of the prior art first reads the host data into the DPU memory through a DMA, and then writes it to the remote shared memory in the server through RDMA to complete the data writing from the host to the server. This writing method is limited by the memory bandwidth of the DPU. At the same time, the two data transmission operations also increase the processing overhead of the DPU. In order to break the memory bandwidth bottleneck, the data path management module of the present invention designs a single-operation zero-copy transmission. The single-operation zero-copy transmission uses the DOCA software stack. After the DPU and the server establish an RDMA connection, the data between the host shared memory and the remote shared memory can be transmitted in a single operation, eliminating the overhead of data passing through the DPU memory.

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the distributed file system based on the data processing unit DPU of the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A framework diagram of a distributed file system according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a file request format and a unilateral polling mechanism according to an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a non-optimized data path in the prior art;

[0032] Figure 4 This is a schematic diagram of an optimized data path according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of non-optimized metadata management in the prior art;

[0034] Figure 6 This is a schematic diagram of optimized metadata management according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0036] Terminology explanation:

[0037] (1) Data Processing Unit (DPU): It is a hardware component that has received widespread attention in the current computer technology field. The present invention can use NVIDIA's BlueField-3 DPU. The DPU includes an ARM SoC (system-on-chip) CPU with a core frequency of 2.0 GHz, 16 cores, an integrated 16MB shared L3 cache, and 32GB of DRAM (memory). At the same time, the DPU also has powerful network and data processing capabilities, with 2×200 Gbps ports and some data processing accelerators;

[0038] (2) DOCA software stack is a set of software libraries provided by NVIDIA for its DPU series, which provides DPU-specific function interfaces for programming languages. DOCA includes many function implementations on DPU, such as DMA unilateral read and write, RDMA unilateral read and write, memory authorization, data compression, data summary, etc.

[0039] (3) DOCA provides new support for RDMA, mainly that the DPU can use the host shared memory and remote shared memory for RDMA; in the common RDMA hardware and software environment, the memory interaction performed by RDMA can only be carried out between the two parties who establish the RDMA connection. For example, when the DPU establishes an RDMA connection with the server, data can only be transferred between the DPU memory and the server memory. DOCA provides a new mechanism to obtain access rights to the host memory through PCIe. Although the RDMA connection is established between the DPU and the server, the DPU can still use the host memory to transfer data with the server memory.

[0040] like Figure 1 As shown, the present invention is a distributed file system based on a data processing unit DPU, comprising: a computing node 1 (host), a data processing unit 2 (DPU) and a storage node 3 (server); the computing node 1 comprises a client request interface 11 and a host shared memory 12; the data processing unit 2 (DPU) comprises a file request processing module 21, a data path management module 22, a metadata index management module 23 and a metadata memory management module 24; the storage node 3 comprises a remote shared memory 31 and a server persistence interface 32;

[0041] The file request processing module 21 adopts a request transmission method based on unilateral polling to read the file request sent from the client request interface 11 to the host shared memory 12, and processes it when a valid file request is found; and after the file request is processed, the file request processing result is returned to the host shared memory 12;

[0042] The data path management module 22 authorizes the access rights of the host shared memory 12 to the DPU in the initialization phase, and establishes an RDMA connection between the DPU and the storage node 3; in the subsequent communication process, the DPU is controlled to use the RDMA connection to perform data interaction between the host shared memory 12 and the remote shared memory 31;

[0043] The metadata index management module 23 adopts a core-based indexing architecture and allocates an index node Inode to each core; when a file request needs to be parsed, the file name is mapped to an Inode address through a mapping relationship, and then the content stored in the Inode is accessed through the address;

[0044] The metadata memory management module 24 uses a double-layer allocation structure including a global allocator and a local allocator to implement the allocation and recovery of Inode memory space; each core is allocated a local allocator, and the local allocator requests pre-allocated memory from the global allocator.

[0045] Specifically, the host in the present invention acts as a computing node, and runs the client request interface and host shared memory internally. The client request interface is used to process requests from user programs, and the host shared memory is used to transmit file data. The DPU is connected to the host via PCIe, and the four functional modules of the DPU run in the DPU processor. The server is connected to the DPU through the network as a storage node, where the server-side persistence interface is for the data persistence of the file system to ensure that the data is not lost, and the remote shared memory is also used to transmit file data.

[0046] like Figure 2The figure shows a schematic diagram of a specific file request format and a unilateral polling mechanism in the file request processing module of the present invention. The processing of the request format is implemented in the client request interface, and each file request is composed of a series of fields, including a request sequence number, a file operation, a file name, a host address, a remote address, and a file size. In order to speed up the transmission of file requests, the present invention packages multiple file requests together to form a batch request. The batch request is composed of a flag bit, a number of requests, and a file request that is actually stored, wherein the flag bit is mainly used for the control of the unilateral polling mechanism. First, the DPU uses DMA to read the batch request. When the flag bit is 1, it means that the client has prepared the request, the reading is valid, and the request processing needs to be performed next; when the flag bit is 0, it means that the client has not prepared the data, and the DMA reading needs to be repeated. When the request is processed in the DPU, the result is written to the client through DMA, and the flag bit is changed to 0. The client returns the result to the application by judging the flag bit. When the next file request is ready, the client modifies the flag bit to 1, and repeats the above process to complete the request transmission. As shown in FIG. Figure 2 As shown, the request reply returned by the DPU is recorded using a 32-bit number, and the execution status of the request is indicated by defining the meaning of each number, for example, 0 indicates successful execution, 1 indicates a network error, and 2 indicates a memory error. At the same time, if the execution is successful and there is data returned, the address of the data return is recorded in the host address field of the file request. The present invention sets the maximum number of batches to 31, and the maximum number of requests and replies to be stored is this number, and each request and reply is corresponding.

[0047] The data path management module of the present invention is used to realize file data transmission between the host and the server. Figure 4 As shown in the figure, the existing data transmission is based on two links, and each data exchange requires two transmission operations. Taking the write request as an example, after the metadata processing is completed, the DPU first obtains the data from the host shared memory to the DPU memory through DMA, and then transfers the data in the DPU memory to the server shared memory through RDMA. Figure 4 As shown, the implementation of the single-operation zero-copy data path of the data path management module mainly involves the collaborative work between the host, DPU and server. Specifically, the DPU and the server establish an RDMA connection. The host authorizes the DPU to access the host shared memory through the DOCA software stack. The DPU uses the host shared memory to register the RDMA area, and the server uses the remote shared memory to register the RDMA area. Finally, the DPU can use one-sided RDMA through the RDMA queue to exchange data between the host shared memory and the remote shared memory. It is worth noting that Figure 4The ① “authorizing access rights” shown only needs to be performed once during the file system initialization phase, while the subsequent ② “interaction between the host and the server” can be performed using a single operation. Figure 3 The unoptimized method shown in the figure does not have the support of DPU and DOCA software stack, and cannot register the host shared memory as an RDMA-capable area. It can only register the DPU memory as an RDMA-capable area, so data can only be transferred through the DPU memory. The unoptimized method will saturate the limited memory bandwidth of the DPU before the network bandwidth is fully utilized. The single-operation zero-copy data path only controls the data between the host shared memory and the remote shared memory, and does not transfer data to the DPU memory, thereby avoiding the DPU memory bandwidth from becoming a bottleneck and improving data transmission efficiency.

[0048] The metadata management design of the present invention includes a metadata index management module and a metadata memory management module. Metadata management uses a full-path file name index, and uses a Masstree data structure as an index to facilitate range searches using path names. When managing files in this way, the key-value pairs stored inside the index structure are (full-path file name, Inode address). When processing file read and write requests, it is necessary to obtain the remote shared memory address in the Inode through the index, and then exchange file data; when processing file creation or deletion, it is necessary to insert or delete the index to maintain the index. File read and write requests will result in reading the index, while file creation or deletion requires modifying the index. This data competition problem requires adding a concurrent access mechanism to the index (mutual exclusion locks, read-write locks, etc. are optional solutions), and it will also generate huge CPU overhead. In order to avoid data competition problems, the index structure of the present invention adopts a core-indexed architecture and assigns an index to each core. At the same time, the request processing cores are divided according to the ID formed by hashing the file name, and specifically, which core is determined to process by taking the modulus of the number of cores. This method effectively avoids conflicts and competition that may be caused by multiple cores using an index structure at the same time. Figure 5 As shown in Figure 1, when multiple file requests arrive at the same time, without this per-core partitioning mechanism, multiple cores may try to access and modify the same index structure at the same time, resulting in data inconsistency and system performance degradation. Figure 6 As shown in the management method, each core has its own independent index area and processing tasks, which can process file requests efficiently and stably.

[0049] Memory management refers to the allocation and recycling of Inode memory space. The present invention sets a certain memory space for Inode storage. When each core processes the creation and deletion of files, it will involve the allocation and recycling of Inode. If each core shares a global allocator, it is also necessary to ensure concurrent access to the global allocator, which brings additional overhead. Figure 6 As shown, the memory allocation structure proposed by the present invention adopts a two-layer design, including a global allocator and a local allocator. When processing a file creation operation, memory allocation is used to allocate the memory space of the Inode. The global allocator is mainly responsible for managing a larger memory space. In practical applications, for example, when allocating memory for an Inode, a 16GB space will be mapped through the mmap system call, and this space is managed by the global allocator. The implementation of the global allocator relies on some specific memory management libraries, such as Memkind. It manages this larger memory space through complex internal algorithms to ensure the reasonable allocation and use of memory. The local allocator pre-allocates 64MB of memory blocks to the global allocator, and allocates them according to each core. When the core needs to allocate an Inode, it obtains memory from the local allocator. This design method enables each core to have its own independent memory acquisition path, reducing the mutual interference between different cores in memory allocation. Compared with the global allocator, the local allocator is more lightweight, and it can quickly respond to the memory needs of the core without the need to perform complex memory search and allocation operations like the global allocator. At the same time, the local allocator interacts with the global allocator through pre-allocation and return operations to perform memory block interactions. The memory block recycling in the local allocator is implemented through the free list to ensure the reasonable flow and utilization of memory resources. When the Inode space in the local allocator is released, it will not be returned to the global allocator immediately, but will be retained in the recycling list of the local allocator first. Once an Inode needs to be allocated, the memory blocks in the recycling list will be used first. Once the recycling list exceeds the threshold, the local allocator will return the memory blocks in the recycling list to the free list in the global allocator in batches to ensure that other local allocators can still obtain memory from the global allocator.

[0050] In summary, the present invention adds a data processing unit DPU under the architecture of storage separation. The metadata management of the present invention is used to unload the metadata management in the computing node, saving the CPU resources in the computing node. At the same time, in view of the insufficient performance of the DPU, we have made specific optimizations in index management and memory allocation for the above metadata management. The data processing unit DPU of the present invention is connected to the host as an external device through a PCIe link, and the DPU communicates with the server through the InfiniBand (IB) network. The PCIe link supports DMA transmission, while the IB network supports RDMA transmission. In the prior art, the data exchange between the external device and the memory depends on the CPU, and the data must pass through the CPU register, which puts a heavy burden on the CPU. In the present invention, the DMA controller plays a key role in DMA transmission. The external device is ready to transmit data and requests the DMA controller. The DMA controller then applies for bus control rights from the CPU. After success, the peripheral device and the memory are controlled to directly transmit data according to the set parameters without the need for CPU handling. The network transmission data of the prior art requires multiple copies and protocol stack processing between the host and the remote computer, while the RDMA of the present invention realizes zero copy through the memory registration mechanism, and directly reads and writes data between remote memories. It relies on specific hardware (such as InfiniBand network cards) and software protocols, and requires initialization and negotiation parameters before communication. RDMA bypasses kernel intervention, has extremely low latency, and can fully utilize network bandwidth to improve throughput.

[0051] The above embodiments are only used to illustrate the present invention, and are not used to limit the present invention. As long as they are based on the technical essence of the present invention, changes and modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A distributed file system based on a data processing unit DPU, characterized in that: It includes a computing node, a data processing unit DPU and a storage node; the computing node includes a client request interface and a host shared memory; the data processing unit DPU includes a file request processing module, a data path management module, a metadata index management module and a metadata memory management module; The storage node includes a remote shared memory; The file request processing module adopts a request transmission method based on unilateral polling to read the file request sent from the client request interface to the host shared memory, and processes it when a valid file request is found; After the file request is processed, the file request processing result is returned to the host shared memory; The data path management module authorizes the host shared memory access rights to the DPU during the initialization phase, and establishes an RDMA connection between the DPU and the storage node; In the subsequent communication process, the DPU is controlled to use RDMA connection to exchange data between the host shared memory and the remote shared memory; The metadata index management module adopts a core-based indexing architecture and allocates an index node Inode to each core; when a file request needs to be parsed, the file name is mapped to an Inode address through a mapping relationship, and then the content stored in the Inode is accessed through the address; The metadata memory management module adopts a double-layer allocation structure including a global allocator and a local allocator to realize the allocation and recovery of Inode memory space; each core is allocated a local allocator, and the local allocator requests pre-allocated memory from the global allocator.

2. The distributed file system based on the data processing unit DPU according to claim 1 is characterized in that: In the file request processing module, the file request is read in batch mode, which specifically includes: The DOCA software stack is used to register the host shared memory in the computing node memory for DPU access. The client request interface collects file requests into the host shared memory as batch requests. The DPU uses queue pairs to perform DMA one-way reads to obtain batch requests from the host shared memory.

3. The distributed file system based on the data processing unit DPU according to claim 2 is characterized in that: The batch request includes a flag bit for controlling the unilateral polling mechanism; when the flag bit is 1, the read is valid and the request is processed; When the flag is 0, DMA reading continues; when the request is processed in the DPU, the result is written to the computing node through DMA and the flag is changed to 0.

4. The distributed file system based on the data processing unit DPU according to claim 1, characterized in that: During the initialization phase, the host authorizes the DPU to access the shared memory and establishes an RDMA connection between the DPU and the storage node, including: During the initialization phase, the DOCA software stack authorizes the DPU to access the host shared memory. The DPU registers the RDMA area using the host shared memory, and the storage node registers the RDMA area using the remote shared memory. The DPU uses one-sided RDMA through the RDMA queue to exchange data between the host shared memory and the remote shared memory.

5. The distributed file system based on the data processing unit DPU according to claim 1, characterized in that: In the metadata index management module, the request processing core is divided according to the ID formed by the hash of the file name.

6. The distributed file system based on the data processing unit DPU according to claim 5, characterized in that: The request processing core is divided according to the ID formed by the hash of the file name, including: The core to be processed is determined by taking the ID modulo the number of cores.

7. The distributed file system based on the data processing unit DPU according to claim 1, characterized in that: The metadata memory management module, the recovery of the Inode memory space, specifically includes: When the memory released by the local allocator exceeds the threshold, the local allocator returns the released memory to the global allocator.

8. The distributed file system based on the data processing unit DPU according to claim 7, characterized in that: When the memory released by the local allocator exceeds the threshold, the local allocator returns the released memory to the global allocator, including: When the Inode space in the local allocator is released, it is first retained in the local allocator's recycling list. Once an Inode needs to be allocated, the memory blocks in the recycling list are used first. Once the recycling list exceeds the threshold, the local allocator will return the memory blocks in the recycling list to the free list in the global allocator in batches.

9. The distributed file system based on the data processing unit DPU according to claim 1, characterized in that: The storage node also includes a server-side persistence interface for writing the data stored in the remote shared memory into the hard disk.

Images