A CephFS kernel client communication method, device and medium
By introducing the RDMA protocol and optimizing connection management in the CephFS kernel client, the performance bottleneck of the CephFS kernel client in AI training scenarios is resolved, achieving efficient data transmission and low latency, and flexible protocol switching to adapt to different hardware environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNICOM DIGITAL TECNOLOGY CO LTD
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-19
Smart Images
Figure CN121880050B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication technology, specifically to a CephFS kernel client communication method, device, and medium. Background Technology
[0002] In today's cloud services, especially in AI training and push scenarios, the selection and performance optimization of file storage systems are crucial. CephFS, as a commonly used file storage system, provides two mounting methods: kernel-mode client and fuse user-mode client. Kernel-mode client mounting performs better in terms of performance, and compared to fuse user-mode client, it can more efficiently meet the data read and write speed requirements of AI training and push scenarios.
[0003] When mounted using a kernel-mode client, the open-source implementation of CephFS uses TCP and is applied to both Messengerv1 and v2 Ceph protocol scenarios to communicate with the distributed Ceph storage cluster. However, with the development of hardware technology, hardware network cards with RDMA capabilities have become increasingly common. RDMA technology can significantly improve network transmission performance, achieving higher throughput and lower latency.
[0004] Currently, CephFS kernel client mounting only has one mode, defaulting to using the TCP protocol to communicate with the distributed Ceph storage cluster. In scenarios involving large-scale read / write throughput, such as AI training, this presents several drawbacks:
[0005] High CPU resource consumption: The TCP protocol is complex, and the CPU needs to perform a lot of calculations during data processing and connection management, resulting in excessive CPU overhead.
[0006] High data transmission latency: While the acknowledgment and retransmission mechanism of the TCP protocol ensures data accuracy, it also increases data transmission latency.
[0007] Low bandwidth utilization: TCP's congestion control algorithm makes it difficult to fully utilize network bandwidth.
[0008] High memory overhead: TCP connections require the kernel to maintain a large amount of connection state information, and memory overhead increases significantly as the number of concurrent connections increases. Summary of the Invention
[0009] This invention provides a CephFS kernel client communication method. By introducing the RDMA protocol and specifically optimizing the CephFS kernel client's messengerv1 and v2 protocols, it aims to meet the data transmission requirements of high-performance, low-latency application scenarios, reduce CPU overhead, decrease latency, improve bandwidth utilization, and reduce memory overhead, thereby solving the problems in the background art.
[0010] To achieve the above objectives, the technical solution of the present invention is as follows:
[0011] A CephFS kernel client communication method includes the following steps:
[0012] S1: Perform CephFS kernel-mode mount operation on the compute nodes of the AI training cluster, explicitly enabling the specified RDMA protocol by specifying the -o rdma option;
[0013] S2: Based on the specified RDMA protocol, the CephFS kernel client automatically establishes an RDMA connection with the monitoring node of the Ceph cluster, obtains the storage node mapping information of the Ceph cluster, establishes an RDMA connection pool with all active storage nodes, and builds a dedicated communication link.
[0014] S3: Based on the established RDMA communication link, when the AI training script initiates a request to read the training dataset, the CephFS kernel client reads the dataset file on the Ceph storage node through the RDMA protocol.
[0015] S4: Based on the RDMA communication link, when saving model checkpoints during AI training, the CephFS kernel client writes data to CephFS through the RDMA protocol. At the same time, the CephFS kernel client uses the RDMA confirmation mechanism to ensure the integrity of each data block written, and then updates the file information to the metadata node after completion.
[0016] S5: If an RDMA hardware failure or an incompatible service requirement arises, the operator can directly remove the -o rdma option from the mount command and switch back to the TCP protocol to complete data communication.
[0017] Preferably, the specific steps of the S1 process include:
[0018] S11, retrieves the CephFS kernel-mode mount instruction from the computing nodes of the AI training cluster;
[0019] S12, add the -o rdma option to the mount command to explicitly specify that the RDMA protocol is enabled;
[0020] S13 executes the mount command with added options, completing the CephFS kernel-mode mount operation.
[0021] Preferably, when the CephFS kernel client in S2 automatically establishes an RDMA connection with the monitoring node of the Ceph cluster;
[0022] First, the fs module calls the mount interface to trigger a connection request. After the net module detects the -o rdma option, it calls the dedicated interface to complete the initialization of RDMA resources. Then, it creates an RDMA identifier and resolves the RDMA routing address of the monitoring node. Only after the routing is successfully resolved can the protection domain be created, the queue be completed, the send and receive queues be sent and received and the queue association be completed. Finally, the actual RDMA connection is initiated.
[0023] Preferably, the specific steps of the S3 process include:
[0024] S31, the AI training script sends a request to the CephFS kernel client to read the training dataset;
[0025] S32, the CephFS kernel client distributes read requests to the corresponding Ceph storage nodes based on the established RDMA communication link;
[0026] S33, the CephFS kernel client reads data blocks from the storage node concurrently via the RDMA protocol;
[0027] S34, assemble the data blocks that have been read concurrently to form a complete training dataset file;
[0028] S35 transfers the assembled training dataset file to the AI training script's calling end to complete the reading operation.
[0029] Preferably, in S33, the CephFS kernel client independently initializes the RDMA read context and creates a dedicated RDMA request descriptor for each concurrent read request. After allocating the corresponding receive buffer, it generates a DMA address for the buffer through direct memory access mapping, and then uses the RDMAREAD operation to directly read data from the storage node memory to the client memory. The entire data transfer process is completed in a zero-copy manner.
[0030] Preferably, when the CephFS kernel client in S4 writes model checkpoint data to CephFS via the RDMA protocol;
[0031] First, the checkpoint data generated during training is written to the kernel page cache. When the cached data reaches the preset size of 1MB or the fsync() synchronization instruction is triggered, the RDMA write process is started, and the data is divided into data blocks of a fixed size of 1MB before being transmitted block by block.
[0032] Preferably, in S4, the CephFS kernel client uses the RDMA confirmation mechanism to ensure data write integrity;
[0033] After a storage node completes the writing of a single block of data, it sends a confirmation message to the client via the RDMA protocol to confirm the successful write. Upon detecting this message, the client immediately releases the RDMA memory registration resources for the corresponding data block. If no confirmation message is detected within a preset time, the client will automatically re-initiate the write request for the data block.
[0034] Preferably, when switching back to the TCP protocol in S5 to complete data communication, only the -o rdma option in the mount command is removed and the CephFS kernel-mode mount operation is re-executed. No hardware or software configuration of the Ceph storage cluster is modified, nor is the business code of the AI training script adjusted. The existing TCP management network between the compute node and the storage node is directly reused to complete data communication.
[0035] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.
[0036] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.
[0037] As can be seen from the above technical solution compared with the prior art, the present invention has the following beneficial effects:
[0038] 1. This invention adapts the CephFS kernel client to the RDMA protocol, optimizes the RDMA connection establishment process, adopts a zero-copy transmission mechanism and an OSD connection pool preheating strategy, and achieves high-speed data interaction under CephFS kernel-mode mounting. This significantly reduces data read / write latency and significantly improves throughput in AI training scenarios, effectively solving the problem of wasted computing power caused by GPU waiting for data loading, and matching the core requirements of high-throughput and low-latency data transmission for large-scale AI training and push.
[0039] 2. This invention adopts a dual-protocol compatible design by specifying the protocol through mounting options. It does not require modification of Ceph cluster hardware configuration and AI training business code. Flexible switching between RDMA and TCP protocols can be achieved simply by adjusting the mounting parameters. This enables the solution to be quickly adapted to different hardware environments and different business needs, achieving the goals of reducing cluster operation and maintenance costs and reliably ensuring business continuity. It significantly improves the practicality of the solution and the scope of scenario adaptation. Attached Figure Description
[0040] Figure 1 This is a flowchart of the CephFS kernel client RDMA connection establishment process in an embodiment of the present invention;
[0041] Figure 2This is a flowchart of the CephFS kernel client RDMA data writing process in an embodiment of the present invention;
[0042] Figure 3 This is a flowchart of the CephFS kernel client RDMA data reading process in an embodiment of the present invention. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0044] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but should not be used to limit the scope of the present invention.
[0045] This invention provides a CephFS kernel client communication method, comprising the following steps:
[0046] S1: Perform CephFS kernel-mode mount operation on the compute nodes of the AI training cluster, explicitly enabling the specified RDMA protocol by specifying the -o rdma option;
[0047] The -o rdma refers to the mount parameter combination in the CephFS kernel-mode mount command, which explicitly instructs the client to enable the RDMA protocol for communication by using the standard option -o followed by the parameter rdma.
[0048] S2: Based on the specified RDMA protocol, the CephFS kernel client automatically establishes an RDMA connection with the monitoring node of the Ceph cluster, obtains the storage node mapping information of the Ceph cluster, establishes an RDMA connection pool with all active storage nodes, and builds a dedicated communication link.
[0049] S3: Based on the established RDMA communication link, when the AI training script initiates a request to read the training dataset, the CephFS kernel client reads the dataset file on the Ceph storage node through the RDMA protocol.
[0050] S4: Based on the RDMA communication link, when saving model checkpoints during AI training, the CephFS kernel client writes data to CephFS through the RDMA protocol. At the same time, the CephFS kernel client uses the RDMA confirmation mechanism to ensure the integrity of each data block written, and then updates the file information to the metadata node after completion.
[0051] S5: If an RDMA hardware failure or an incompatible service requirement arises, the operator can directly remove the -o rdma option from the mount command and switch back to the TCP protocol to complete data communication.
[0052] Example 1:
[0053] In this embodiment, in the actual deployment scenario of enterprise-level AI training and inference, in order to support the training and inference tasks of large-scale deep learning models, the business side built an AI training cluster based on Ceph distributed storage. The computing nodes and storage nodes in the cluster are equipped with dedicated RDMA network cards, and the CephFS kernel mode mounting method is used to carry the storage and management of training datasets and model checkpoints.
[0054] To address the issues of high latency and high CPU resource consumption in traditional TCP protocol data transmission, it is necessary to achieve efficient data communication between the CephFS kernel client and the Ceph distributed storage system through the RDMA protocol. Therefore, a CephFS kernel client communication method is introduced in this application.
[0055] This application adds an RDMA option mounted on the client in the fs module to communicate with Ceph distributed storage through the RDMA module, and adds RDMA protocol adaptation to the messengerv1 and v2 modules.
[0056] The core process has three steps: establishing an RDMA connection, sending a request to the Ceph distributed system, and receiving a request from the Ceph distributed storage system.
[0057] CephFS kernel client establishes RDMA connection as follows: Figure 1 As shown:
[0058] 1. The fs module submits a request to the Ceph distributed storage system, triggering the establishment of a connection with the storage system nodes.
[0059] 2. The net module determines the network communication protocol type to be used, i.e., TCP or RDMA protocol, based on the protocol options specified during mounting, and registers the network communication operation interface that matches the protocol type.
[0060] 3. Create an RDMA identifier id to initialize the RDMA context.
[0061] 4. Resolve the route address.
[0062] 5. After receiving a successful route resolution, the system begins to create a protection domain (PD), a completion queue (CQ), and send and receive queues (QP), and associates the corresponding completion queues.
[0063] 6. Finally, initiate the actual RDMA connection.
[0064] The process of sending a request to a Ceph distributed system is as follows: Figure 2 As shown:
[0065] 1. Create an RDMA request to facilitate the mapping and management of write buffers; terminate if it fails.
[0066] 2. Map user request data to the buffer of the RDMA request.
[0067] 3. Map the buf and dma requested by RDMA; if this fails, the process ends.
[0068] 4. Synchronize buf to the RDMA device.
[0069] 5. Submit a data request to the sending queue; if it fails, the process ends.
[0070] 6. Sent successfully.
[0071] The process of receiving requests from the Ceph distributed storage system is as follows: Figure 3 As shown:
[0072] 1. The Messengerv1 and v2 protocols transmit the size of the data to be read and the buf pointer.
[0073] 2. If the connection is closed, an EIO error is returned.
[0074] 3. Check if there is available data in the receive queue. If not, relinquish the CPU and wait for the data to arrive. Then repeat step 2.
[0075] 4. Copy the available data to the passed-in buf pointer.
[0076] 5. Determine if all the data to be read has been read. If all the data has been read, return the size. If not, repeat steps 2, 3, and 4.
[0077] Example 2:
[0078] This embodiment is a specific implementation example of Embodiment 1. The complete implementation process of the present invention will be described in detail below in conjunction with specific application scenarios.
[0079] Scenario Description: An AI training cluster based on Ceph distributed storage specifically includes:
[0080] Compute nodes: 8 units, each equipped with 8 GPUs (NVIDIA A100) and 1 Mellanox RDMA network card (100Gbps);
[0081] Storage nodes: 3 Ceph clusters, each equipped with dual AMD EPYC processors, 256GB of memory, 10 NVMe SSDs as OSDs, and 1 Mellanox RDMA network card.
[0082] Network architecture: Compute nodes and storage nodes are interconnected via a 100Gbps RoCEv2 network, while retaining a 10Gbps TCP management network;
[0083] Requirements: Mount CephFS in kernel mode to store training datasets, totaling 5TB, with each file ranging from 100MB to 1GB, and model checkpoints, with each file ranging from 2GB to 10GB.
[0084] Deployment and configuration process;
[0085] Step 1: Driver and kernel preparation;
[0086] To ensure that the RDMA protocol can run normally in the kernel layer of the compute node, the kernel version confirmation and loading of RDMA-related driver modules are completed first: confirm that the Linux kernel version used by the compute node is 6.6, which has good compatibility with the RDMA protocol;
[0087] The RDMA core module, user-space verbs interface module, RDMA connection management module, and Mellanox network card dedicated driver module are loaded sequentially to complete the adaptation of RDMA hardware and kernel, ensuring that the functions of RDMA network card can be recognized and called by kernel.
[0088] After the module loading is completed, the identification status of the RDMA device is verified to confirm that the system can accurately identify the hardware information of the Mellanox RDMA network card, including node type, firmware version, node globally unique identifier, etc., thereby determining that the RDMA driver environment configuration is complete.
[0089] Step 2: Mount CephFSRDMA;
[0090] On the compute node where the driver and kernel environment preparation is complete, the kernel-mode mount operation of CephFS is performed. The core is to specify the enabling of the RDMA protocol through mount parameters. The specific operation is as follows:
[0091] The mount operation uses the monitoring node address of the Ceph cluster as the connection target and mounts CephFS to the specified directory ( / mnt / cephfs-rdma) on the compute node.
[0092] During the mounting process, add the -o rdma option. This option is the core parameter that triggers the RDMA protocol adaptation logic in this application. By using this option, you can explicitly specify that the CephFS kernel client uses the RDMA protocol instead of the traditional TCP protocol to communicate with the storage cluster.
[0093] This mounting method features flexible switching between dual protocols. If it is necessary to switch back to the TCP protocol later, simply remove the "rdma" option and remount it without adjusting other cluster configurations, ensuring the convenience of protocol switching.
[0094] Step 3: Protocol Negotiation and Connection Establishment;
[0095] During the mounting process, the kernel client automatically performs the following sequence of operations:
[0096] Mon connection established:
[0097] The CephFS file system (FS) module calls the mount interface and first initiates a connection request with the Ceph monitoring node.
[0098] When the NET module detects the RDMA option specified during mounting, it immediately calls the RDMA initialization interface to complete the initialization of the basic resources required for RDMA communication.
[0099] Create an RDMA identifier to perform RDMA route resolution on the Internet Protocol (IP) address of the monitoring node;
[0100] Once the route resolution is complete and the system enters the route resolution success state, the protection domain (PD), completion queue (CQ), and send and receive queue (QP) are created sequentially to complete the core resource configuration for the monitoring node's RDMA connection and establish a stable RDMA connection with the monitoring node.
[0101] OSD mapping acquisition:
[0102] Using the established RDMA connection of the monitoring node, send an authentication request to the monitoring node, as well as a request to obtain metadata node mapping information and OSD mapping information;
[0103] The request message is encapsulated according to the frame format of the Messengerv2 protocol and sent to the monitoring node via the RDMA protocol;
[0104] The response data returned by the monitoring node is received via the RDMA protocol and stored in a pre-registered receive buffer, providing a basis for subsequent location of data storage.
[0105] OSD connection pool preheating:
[0106] Based on the obtained OSD mapping information, all active storage nodes in the cluster are selected, and RDMA connections with these storage nodes are established in advance. The default number of concurrent connections is 32.
[0107] Each storage node's RDMA connection maintains an independent send / receive queue and memory registry (MR) pool, avoiding resource contention issues during multi-threaded access. Pre-setting up the connection pool can significantly reduce connection establishment latency during subsequent data read / write operations.
[0108] Step 4: RDMA protocol reading process for the training dataset;
[0109] When the AI training script initiates a training dataset read request based on the PyTorchDataLoader framework, the CephFS kernel client performs a low-latency, high-throughput read operation based on the established RDMA connection, which is divided into two stages:
[0110] Phase 1: Metadata Acquisition (Interacting with the Metadata Server);
[0111] The client sends a file query request to the metadata server via the RDMA protocol to obtain the inode information and data layout information of the target dataset file (i.e., the storage location of the RADOS object corresponding to the data).
[0112] The response time at this stage can be controlled within 0.5 milliseconds, which is more than 75% lower than the 2-3 millisecond response time of the traditional TCP protocol, greatly improving the efficiency of metadata acquisition.
[0113] Phase 2: Data object reading (interaction with storage nodes);
[0114] Taking reading a 500MB training data file as an example, the file is split into four 128MB RADOS objects. The specific operation process is as follows:
[0115] Request distribution: The client creates four concurrent OSD request structures for the four RADOS objects, and initializes the RDMA read context independently for each request to achieve parallel reading of multiple objects;
[0116] For RDMA data transfer, taking a single 128MB object as an example: An RDMA request descriptor is created, a 128MB receive buffer is allocated and memory registration is completed, and the DMA address of the buffer is generated through Direct Memory Access (DMA) mapping; a work request of type RDMAREAD is constructed. Since Ceph's storage nodes support RDMAREAD operations, data can be directly read from the storage node's memory to the client's memory, achieving zero-copy transfer; this work request is submitted to the send queue, and the transfer status is detected by polling the queue. When a successful transfer flag is detected, the object is marked as read complete.
[0117] Data assembly: After all four RADOS objects have been read in parallel, the data blocks are sorted and assembled according to the original file offsets. The assembled complete data is then copied to the user space buffer for use by the training script.
[0118] Step 5: RDMA protocol writing process for model checkpoints;
[0119] During model training, a model checkpoint file of approximately 8GB needs to be saved every 30 minutes. The client uses the RDMA protocol to efficiently and reliably write this large file. The specific process is as follows:
[0120] Write buffer and mapping: The training framework calls the file saving interface to write the model checkpoint data to the kernel page cache of the computing node; when the amount of data in the cache reaches the preset threshold (1MB) or a synchronization command is triggered, the RDMA write process is automatically started.
[0121] RDMA data transmission: The data in the kernel page cache is divided into 1MB blocks, each data block is registered in memory and a DMA address is generated. After creating a data transmission entry, a transmission request is submitted, and the data block is sent to the target storage node via the RDMA protocol. After receiving the data, the storage node directly writes the data to the NVMe solid-state drive, achieving zero-copy transmission throughout the process.
[0122] Transactional guarantee: After each 1MB data block is written, the storage node sends a confirmation message to the client via the RDMA protocol to confirm the successful write. After the client detects the confirmation message, it releases the memory registration resources corresponding to the data block. After all data blocks of the 8GB checkpoint file have been written and confirmation messages have been received, the client submits a file size update request to the metadata server to ensure that the file metadata is consistent with the actual stored data.
[0123] This application significantly improves storage access performance in AI training and push scenarios by implementing RDMA protocol support in the kernel-mode CephFS client and designing a complete connection management and zero-copy read / write process. The above embodiments detail the complete process from mounting configuration and protocol negotiation to actual data read / write, demonstrating the feasibility and superiority of this technical solution in a real-world production environment. Especially in large-scale AI training scenarios, RDMA mode achieves a 2-3 times throughput increase and a 70% latency reduction compared to traditional TCP mode, effectively solving the problem of wasted computing power on GPUs due to data loading wait times.
[0124] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.
[0125] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.
[0126] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the CephFS kernel client communication methods described in the above embodiments.
[0127] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.
[0128] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0129] For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media.
[0130] The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).
[0131] It should be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0132] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0133] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0134] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A CephFS kernel client communication method, characterized in that, Includes the following steps: S1: Perform CephFS kernel-mode mount operation on the compute nodes of the AI training cluster, explicitly enabling the specified RDMA protocol by specifying the -o rdma option; The -o rdma refers to the combination of mount parameters in the CephFS kernel-mode mount command, which explicitly instructs the client to enable the RDMA protocol for communication by using the standard option -o followed by the parameter rdma. S2: Based on the specified RDMA protocol, the CephFS kernel client automatically establishes an RDMA connection with the monitoring node of the Ceph cluster, obtains the storage node mapping information of the Ceph cluster, establishes an RDMA connection pool with all active storage nodes, and builds a dedicated communication link. When the CephFS kernel client in S2 automatically establishes an RDMA connection with the monitoring node of the Ceph cluster: first, the fs module calls the mount interface to trigger a connection request. After the net module detects the -o rdma option, it calls the dedicated interface to complete the initialization of RDMA resources. Then, it creates an RDMA identifier and resolves the RDMA routing address of the monitoring node. Only after the routing is successfully resolved can the protection domain be created, the queue be completed, the send and receive queues be sent and received, and the queue association be completed. Finally, the actual RDMA connection is initiated. S3: Based on the established RDMA communication link, when the AI training script initiates a request to read the training dataset, the CephFS kernel client reads the dataset file on the Ceph storage node through the RDMA protocol. The specific steps of the S3 process include: S31, the AI training script sends a request to the CephFS kernel client to read the training dataset; S32, the CephFS kernel client distributes read requests to the corresponding Ceph storage nodes based on the established RDMA communication link; S33, the CephFS kernel client reads data blocks from the storage node concurrently via the RDMA protocol; In the S33 CephFS kernel client, for each concurrent read request, an RDMA read context is independently initialized and a dedicated RDMA request descriptor is created. After allocating the corresponding receive buffer, a DMA address is generated for the buffer through direct memory access mapping. Then, the RDMAREAD operation is used to directly read data from the storage node memory to the client memory, and the data transmission is completed in a zero-copy manner throughout the process. S34, assemble the data blocks that have been read concurrently to form a complete training dataset file; S35, the assembled training dataset file is transferred to the calling end of the AI training script to complete the reading operation; S4: Based on the RDMA communication link, when saving model checkpoints during AI training, the CephFS kernel client writes data to CephFS through the RDMA protocol. At the same time, the CephFS kernel client uses the RDMA confirmation mechanism to ensure the integrity of each data block written, and then updates the file information to the metadata node after completion. S5: If an RDMA hardware failure or an incompatible service requirement arises, the operator can directly remove the -o rdma option from the mount command and switch back to the TCP protocol to complete data communication.
2. The CephFS kernel client communication method as described in claim 1, characterized in that: The specific steps of the S1 process include: S11 retrieves the CephFS kernel-mode mount instruction from the computing nodes of the AI training cluster. S12, add the -o rdma option to the mount command to explicitly specify that the RDMA protocol is enabled; S13 executes the mount command with added options, completing the CephFS kernel-mode mount operation.
3. The CephFS kernel client communication method as described in claim 1, characterized in that: When the CephFS kernel client in S4 writes model checkpoint data to CephFS via the RDMA protocol; First, the checkpoint data generated during training is written to the kernel page cache. When the cached data reaches the preset size of 1MB or the fsync() synchronization instruction is triggered, the RDMA write process is started, and the data is divided into data blocks of a fixed size of 1MB before being transmitted block by block.
4. The CephFS kernel client communication method as described in claim 3, characterized in that: In S4, the CephFS kernel client uses the RDMA confirmation mechanism to ensure data write integrity. After a storage node completes the writing of a single block of data, it sends a confirmation message to the client via the RDMA protocol to confirm the successful write. Upon detecting this message, the client immediately releases the RDMA memory registration resources for the corresponding data block. If no confirmation message is detected within a preset time, the client will automatically re-initiate the write request for the data block.
5. The CephFS kernel client communication method as described in claim 1, characterized in that: When switching back to the TCP protocol in S5 to complete data communication, only the -o rdma option in the mount command is removed and the CephFS kernel-mode mount operation is re-executed. No hardware or software configuration of the Ceph storage cluster is modified, nor is the business code of the AI training script adjusted. The existing TCP management network between the compute node and the storage node is directly reused to complete data communication.
6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the processor performs the steps of the method as described in any one of claims 1 to 5.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 5.