An acceleration system and method under a user mode file system offload apparatus

Abstract

The application discloses an acceleration system and method under a user mode file system unloading device, and the system comprises a host end, an FPGA hardware module, a DPU end and a remote storage; the host end initiates a file operation request and transmits the file operation request to the DPU end through the FPGA hardware module; the DPU end comprises a virtioFS service module, a FUSE module, a flow identification module, a flow matching module and a storage side copy module; the FUSE module intercepts read-write requests and extracts key parameters; the flow identification module performs sequential flow identification on the requests and generates flow information records; the flow matching module performs matching verification on read-write sequential flows, sends a copy matching success signal to the FUSE module when a condition is met, and triggers the storage side copy module to send a copy request to the remote storage; the remote storage receives the request and performs internal data copying, and the FUSE module returns preset false data to the host end. The application avoids multiple rounds of data moving during file copying, significantly reduces network and bus bandwidth consumption and host computing power loss, and improves the copying efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing and storage technology, specifically to an acceleration system and method under a user-space file system offloading device. Based on FPGA hardware communication architecture, virtioFS protocol and FUSE module, it is suitable for file copying scenarios that achieve remote storage access through DPU, and can significantly reduce bandwidth consumption and improve file copying efficiency. Background Technology

[0002] The statements in this section are provided only as background information in connection with this disclosure and may not constitute prior art.

[0003] In today's rapidly evolving data center landscape, the pressure on computing power and storage is increasingly evident. With the explosive growth of data volume and the ever-increasing demands for real-time performance and high performance from various applications, traditional CPU processing methods are gradually proving inadequate for handling massive file operations and storage management tasks. Furthermore, in data-compute separation scenarios, files are typically stored on remote storage devices and accessed locally via remote storage access protocols such as NFS, CIFS, or CephFS. This forces the local CPU to handle not only file access requests but also protocol interactions and network transmissions, further exacerbating the CPU's workload.

[0004] Data Processing Units (DPUs), as an emerging technology, possess powerful processing capabilities and dedicated hardware acceleration modules, serving as the core middleware between the host and remote storage. Current technologies have established a complete offloading architecture: "host-side virtioFS driver module + FPGA hardware module + DPU-side virtioFS service module + FUSE module." The FPGA hardware module is soldered onto the DPU and connected to the host via a PCIe interface, enabling memory access and protocol conversion between the host and the DPU. On the host side, the virtioFS driver module encapsulates file requests into the virtio protocol format and transmits them via the FPGA to the virtioFS service module on the DPU side. The FUSE module on the DPU side interfaces with the virtioFS service module to execute specific file system operation logic (such as metadata management and permission verification), effectively offloading host-side file system tasks to the DPU and freeing up host CPU resources.

[0005] VirioFS, a user-space file system based on the virtio protocol, achieves efficient communication and task offloading between the host and DPU through collaboration with FPGA hardware modules and FUSE modules. However, the current architecture only focuses on forwarding and protocol conversion of single read / write requests, and is not optimized for high-frequency, high-bandwidth-consumption scenarios such as file copying. In a data-compute separation architecture, the traditional file copy process involves multiple rounds of data movement: "remote storage → DPU → FPGA → host → FPGA → DPU → remote storage." This not only consumes a large amount of DPU network bandwidth and PCIe channel bandwidth, resulting in resource waste, but also increases data latency due to multiple rounds of transmission and protocol conversion, leading to low copy efficiency and failing to realize the potential of direct data migration from the storage side. Traditional file copying... Figure 2 As shown.

[0006] Furthermore, while existing remote storage systems do not universally support native file copy interfaces, from a technical implementation perspective, the requirements can be met by adding a copy service within the remote storage itself. This provides a feasible basis for the DPU to trigger direct copying on the storage side. Therefore, how to optimize the IO path for file copying scenarios based on the existing "FPGA+vritioFS+FUSE" offloading architecture, avoid invalid data movement, reduce bandwidth consumption and host computing power consumption, and improve file copying efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems in current DPU (Data Processing Unit) computing separation scenarios, where the traditional file copy process involves multiple rounds of invalid data movement (i.e., remote storage → DPU → FPGA → host → FPGA → DPU → remote storage), resulting in wasted DPU network and PCIe channel bandwidth, inefficient FPGA hardware resource usage, computational power loss due to host participation in data reading / writing and memory copying, and low file copy efficiency due to multiple rounds of data transmission and protocol conversion. This invention provides an acceleration system and method based on a user-space file system offloading device. It is built upon the existing "host-side + virtioFS + FPGA + DPU-side + virtioFS + FUSE" offloading architecture, introducing a flow identification module, a flow matching module, and a storage module on the DPU side. The storage-side copy module intercepts read / write requests and extracts key parameters through the FUSE module. Combining stream identification and stream matching rules (such as complete consistency comparison based on offset, size, and MD5 values), it accurately identifies file copying behavior. The storage-side copy module then triggers a direct copy service within the remote storage. Simultaneously, it directly returns preset fake data to read / write requests from the host side to achieve a fast response. This avoids unnecessary multiple rounds of data movement, significantly saves DPU network and PCIe channel bandwidth and FPGA hardware transmission resources, significantly reduces host computing power consumption, and greatly improves the overall efficiency of file copying. Furthermore, this system and method can handle unexpected scenarios such as some file copying and data modification, ensuring data consistency and integrity.

[0008] The technical solution of the present invention is as follows:

[0009] An acceleration system under a user-space file system unloading device includes:

[0010] Host-side, FPGA hardware module, DPU-side, and remote storage;

[0011] The host is used to initiate file operation requests and transmit them to the DPU via the FPGA hardware module.

[0012] The FPGA hardware module is connected to the host via a PCIe interface, serving as a hardware communication bridge between the host and the DPU, and is used to complete protocol conversion and file operation request transmission.

[0013] The DPU includes a virtioFS service module, a FUSE module, a stream identification module, a stream matching module, and a storage-side copy module;

[0014] The virtioFS service module is used to receive file operation requests sent by the host and forward them to the FUSE module;

[0015] The FUSE module interfaces with the virtioFS service module to intercept read and write requests, extract key parameters, and forward them to the stream identification module; and after receiving a copy matching success signal sent by the stream matching module, it directly returns preset fake data to the host for the intercepted read and write requests.

[0016] The stream identification module is connected to the FUSE module and is used to identify the sequence of intercepted read and write requests, determine whether they are read sequence streams or write sequence streams, and generate corresponding stream information records.

[0017] The stream matching module is connected to the stream identification module, the FUSE module, and the storage-side copy module. It is used to match and verify the identified read-order stream and write-order stream. If the matching conditions are met, a copy matching success signal is sent to the FUSE module, and a copy trigger signal and synchronization storage information are sent to the storage-side copy module.

[0018] The storage-side copy module is connected to the stream matching module and the remote storage, and is used to send a copy request to the remote storage according to the copy trigger signal;

[0019] The remote storage is used to receive the copy request and trigger the copy service inside the remote storage to perform a data copy operation.

[0020] Furthermore, the host terminal includes:

[0021] VFS is used to provide a standard file access interface for user applications;

[0022] The FUSE kernel module, which interfaces with the VFS, is used to convert user file operations into FUSE file requests;

[0023] The virtioFS driver module is used to encapsulate the FUSE file request into the virtio protocol format and send it to the DPU through the FPGA hardware module; at the same time, it receives the response message from the DPU and feeds it back to the user application through the FUSE kernel module and VFS.

[0024] Furthermore, the stream identification module performs sequential stream identification on the intercepted read / write requests and generates corresponding stream information records, specifically including:

[0025] If consecutive sequential read operations on the same file starting from a fixed position are detected, it is marked as a read sequential stream; if consecutive sequential write operations on the same file starting from a fixed position are detected, it is marked as a write sequential stream.

[0026] For each read / write request of a sequential stream, a four-tuple record is generated and stored in the stream record cache. The four-tuple record includes: stream type, file path and name, starting offset, data size, and the MD5 checksum of the corresponding data.

[0027] Furthermore, the matching conditions include:

[0028] The starting offset, data size, and MD5 checksum of each corresponding read and write request are completely identical between the read sequential stream and the write sequential stream.

[0029] The number of read / write requests that meet the matching criteria has reached the preset matching threshold;

[0030] The source file and the target file are stored on the same remote storage service.

[0031] Furthermore, the copy service within the remote storage includes:

[0032] The copy request parsing unit is used to receive the copy request issued by the storage-side copy module and parse the source file path and the target file path;

[0033] The data block mapping unit is used to establish a mapping relationship between source file data blocks and target files within the remote storage, enabling direct copying at the data block level;

[0034] The metadata synchronization unit is used to synchronize and update the metadata of the target file to ensure file integrity.

[0035] This invention also proposes a file copying acceleration method based on the above-mentioned acceleration system, comprising the following steps:

[0036] The host initiates a file operation request and transmits it to the DPU via the FPGA hardware module.

[0037] The DPU receives the file operation request, intercepts the read and write requests, extracts key parameters for sequential stream identification, determines whether it is a read sequential stream or a write sequential stream, and generates corresponding stream information records.

[0038] The identified read sequence stream and write sequence stream are matched and verified. If the matching conditions are met, the copy matching is confirmed to be successful.

[0039] After confirming a successful copy match, a copy request is sent to the remote storage to trigger a direct copy within the remote storage. Meanwhile, for any subsequent intercepted read and write requests, preset fake data is returned directly to the host.

[0040] Furthermore, after directly returning preset fake data to the host, a step to confirm the normal file copy result is also included:

[0041] After receiving the preset fake data, the host considers the read / write operation to be complete and sends a file close request to the DPU.

[0042] The DPU processes the file closing request and queries the copy results in the remote storage.

[0043] If the remote storage copy is confirmed to be complete, a successful shutdown message will be sent to the host.

[0044] Furthermore, in the scenario of a partial copy where only a portion of the file data is copied to the host, it also includes:

[0045] Extract the starting offset and data size of the last read / write request in the write sequence stream, and calculate the actual size of the target file;

[0046] Send a file size adjustment request to the remote storage, carrying the target file path and the actual size;

[0047] After receiving the file size adjustment request, the remote storage performs the ftruncate operation to adjust the target file size to the actual size, and sends a successful closure feedback to the host after confirming the adjustment is complete.

[0048] Furthermore, generating the corresponding stream information record includes extracting and recording the MD5 checksum of the data corresponding to the read / write request; if it is a data modification scenario where the host modifies the data and then writes it to the target file, the processing steps include:

[0049] When the MD5 checksum of the data corresponding to a subsequent write request is found to be inconsistent with the MD5 checksum of the read and write requests corresponding to the read sequence stream, it is determined to be a data modification scenario, and a matching failure signal is triggered.

[0050] After receiving a subsequent file closure request from the host, report the closure failure to the host.

[0051] After receiving the shutdown failure message, the host re-executes the file copy operation according to the traditional process, and the DPU forwards subsequent read and write requests according to the traditional process.

[0052] Furthermore, the key parameters include the starting offset, data size, and MD5 checksum; successful copy matching requires meeting certain matching conditions, including a preset matching threshold for the number of read / write requests meeting the conditions; when an exception occurs during processing, the following corresponding exception handling logic is executed:

[0053] If the remote storage copy fails, a read / write request failure is reported to the host, and the host will re-execute the copy according to the traditional process.

[0054] If the starting offset, data size, or corresponding MD5 checksum of a read / write request does not match before reaching the matching threshold, the matching is deemed to have failed, and subsequent read / write requests are forwarded to remote storage for processing according to the traditional process.

[0055] Compared with existing technologies, the advantages of this invention are:

[0056] 1. Extremely saves bandwidth and hardware resources: Based on the existing "FPGA+vritioFS+FUSE" architecture, it triggers direct copying to remote storage by recognizing file copying behavior, avoiding multiple rounds of data movement in "remote storage → DPU → FPGA → host → FPGA → DPU → remote storage", which greatly saves DPU network bandwidth, PCIe channel bandwidth and FPGA hardware transmission resources, freeing up resources for other services.

[0057] 2. Significantly reduce host computing power consumption: The host only receives dummy data returned by the FUSE module for read / write requests, without having to perform actual data reading, memory copying and writing operations, which significantly reduces host computing power consumption and allows it to focus on core business computing.

[0058] 3. Significantly improve copy efficiency: Remote storage achieves direct data block-level copying through internal copy service, avoiding multiple rounds of transmission, protocol conversion, and hardware forwarding delays; at the same time, the host quickly receives fake data responses, shortening the overall copy time and improving data migration efficiency in data-compute separation scenarios.

[0059] 4. Strong compatibility and reusability: It fully reuses the existing "host-side + vritioFS + FPGA + DPU-side + vritioFS + FUSE" architecture without adding new underlying hardware or reconstructing core components; the host-side application does not need to be modified and is compatible with the existing ecosystem; the remote storage achieves functionality by adding an internal copy service without significantly modifying the storage architecture.

[0060] 5. High scenario adaptability and data security: Relying on the native functions of the FUSE module, it handles unexpected scenarios such as partial copying and data modification. MD5 verification ensures the accuracy of copy behavior identification and avoids data inconsistency. The result confirmation mechanism when the file is closed further ensures data integrity.

[0061] 6. Low deployment cost: Only the DPU side needs to add stream identification and stream matching modules to expand the FUSE module functionality, without the need for additional hardware investment; the remote storage copy service is a technically feasible logical extension that does not rely on the native support of existing storage, making it easy to implement. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0063] Figure 1 This is a schematic diagram of an acceleration system under a user-mode file system unloading device.

[0064] Figure 2 This is a diagram illustrating the principle of traditional file copying. Detailed Implementation

[0065] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0067] Example 1

[0068] This embodiment addresses the following three core problems with traditional file copying methods in DPU (Data Processing Unit) computing separation scenarios:

[0069] 1. The problem of wasted DPU network bandwidth and PCIe channel bandwidth caused by repeated data movement, as well as the invalid occupation of FPGA hardware transmission resources;

[0070] 2. The problem of computing power loss caused by the host's participation in data reading and writing and memory copying;

[0071] 3. The problem of low file copying efficiency caused by multiple rounds of data transmission, protocol conversion and hardware forwarding.

[0072] At the same time, it is necessary to address the handling requirements of two unexpected scenarios:

[0073] Scenario 1: The host only copies part of the file data, and it is necessary to ensure that the final size of the target file is consistent with the actual copied data;

[0074] Scenario 2: After the host reads a file, modifies some data, and then writes it to the target file, it is necessary to avoid data inconsistency caused by incorrectly triggering a copy operation on the storage side.

[0075] An acceleration system based on a user-space file system offloading device is proposed, specifically including:

[0076] Host-side, FPGA hardware module, DPU-side, and remote storage;

[0077] The host is used to initiate file operation requests and transmit them to the DPU via the FPGA hardware module.

[0078] The FPGA hardware module is connected to the host via a PCIe interface, serving as a hardware communication bridge between the host and the DPU, and is used to complete protocol conversion and file operation request transmission.

[0079] The DPU includes a virtioFS service module, a FUSE module, a stream identification module, a stream matching module, and a storage-side copy module;

[0080] The virtioFS service module is used to receive file operation requests sent by the host and forward them to the FUSE module;

[0081] The FUSE module interfaces with the virtioFS service module to intercept read and write requests, extract key parameters, and forward them to the stream identification module; and after receiving a copy matching success signal sent by the stream matching module, it directly returns preset fake data to the host for the intercepted read and write requests.

[0082] The stream identification module is connected to the FUSE module and is used to identify the sequence of intercepted read and write requests, determine whether they are read sequence streams or write sequence streams, and generate corresponding stream information records.

[0083] The stream matching module is connected to the stream identification module, the FUSE module, and the storage-side copy module. It is used to match and verify the identified read-order stream and write-order stream. If the matching conditions are met, a copy matching success signal is sent to the FUSE module, and a copy trigger signal and synchronization storage information are sent to the storage-side copy module.

[0084] The storage-side copy module is connected to the stream matching module and the remote storage, and is used to send a copy request to the remote storage according to the copy trigger signal;

[0085] The remote storage is used to receive the copy request and trigger the copy service inside the remote storage to perform a data copy operation.

[0086] In this embodiment, specifically, the host terminal includes:

[0087] VFS is used to provide a standard file access interface for user applications;

[0088] The FUSE kernel module, which interfaces with the VFS, is used to convert user file operations into FUSE file requests;

[0089] The virtioFS driver module is used to encapsulate the FUSE file request into the virtio protocol format and send it to the DPU through the FPGA hardware module; at the same time, it receives the response message from the DPU and feeds it back to the user application through the FUSE kernel module and VFS.

[0090] In this embodiment, specifically, the stream identification module performs sequential stream identification on the intercepted read / write requests and generates corresponding stream information records, specifically including:

[0091] If consecutive sequential read operations on the same file starting from a fixed position are detected, it is marked as a read sequential stream; if consecutive sequential write operations on the same file starting from a fixed position are detected, it is marked as a write sequential stream.

[0092] For each read / write request of a sequential stream, a four-tuple record is generated and stored in the stream record cache. The four-tuple record includes: stream type, file path and name, starting offset, data size, and the MD5 checksum of the corresponding data.

[0093] In this embodiment, the matching conditions specifically include:

[0094] The starting offset, data size, and MD5 checksum of each corresponding read and write request are completely identical between the read sequential stream and the write sequential stream.

[0095] The number of read / write requests that meet the matching criteria has reached the preset matching threshold;

[0096] The source file and the target file are stored on the same remote storage service.

[0097] In this embodiment, specifically, the copy service within the remote storage includes:

[0098] The copy request parsing unit is used to receive the copy request issued by the storage-side copy module and parse the source file path and the target file path;

[0099] The data block mapping unit is used to establish a mapping relationship between source file data blocks and target files within the remote storage, enabling direct copying at the data block level;

[0100] The metadata synchronization unit is used to synchronize and update the metadata of the target file to ensure file integrity.

[0101] Based on the same inventive concept, this embodiment also proposes a file copying acceleration method based on the above-mentioned acceleration system, including the following steps:

[0102] The host initiates a file operation request and transmits it to the DPU via the FPGA hardware module.

[0103] The DPU receives the file operation request, intercepts the read and write requests, extracts key parameters for sequential stream identification, determines whether it is a read sequential stream or a write sequential stream, and generates corresponding stream information records.

[0104] The identified read sequence stream and write sequence stream are matched and verified. If the matching conditions are met, the copy matching is confirmed to be successful.

[0105] After confirming a successful copy match, a copy request is sent to the remote storage to trigger a direct copy within the remote storage. Meanwhile, for any subsequent intercepted read and write requests, preset fake data is returned directly to the host.

[0106] Furthermore, after directly returning preset fake data to the host, a step to confirm the normal file copy result is also included:

[0107] After receiving the preset fake data, the host considers the read / write operation to be complete and sends a file close request to the DPU.

[0108] The DPU processes the file closing request and queries the copy results in the remote storage.

[0109] If the remote storage copy is confirmed to be complete, a successful shutdown message will be sent to the host.

[0110] In this embodiment, specifically, in a scenario where the host only copies a portion of the file data, the method further includes:

[0111] Extract the starting offset and data size of the last read / write request in the write sequence stream, and calculate the actual size of the target file;

[0112] Send a file size adjustment request to the remote storage, carrying the target file path and the actual size;

[0113] After receiving the file size adjustment request, the remote storage performs the ftruncate operation to adjust the target file size to the actual size, and sends a successful closure feedback to the host after confirming the adjustment is complete.

[0114] In this embodiment, specifically, generating the corresponding stream information record includes extracting and recording the MD5 checksum of the data corresponding to the read / write request; if it is a data modification scenario where the host modifies the data and then writes it to the target file, the processing steps include:

[0115] When the MD5 checksum of the data corresponding to a subsequent write request is found to be inconsistent with the MD5 checksum of the read and write requests corresponding to the read sequence stream, it is determined to be a data modification scenario, and a matching failure signal is triggered.

[0116] After receiving a subsequent file closure request from the host, report the closure failure to the host.

[0117] After receiving the shutdown failure message, the host re-executes the file copy operation according to the traditional process, and the DPU forwards subsequent read and write requests according to the traditional process.

[0118] In this embodiment, specifically, the key parameters include the starting offset, data size, and MD5 checksum; confirming a successful copy match requires meeting corresponding matching conditions, including the number of read / write requests meeting a preset matching threshold; when an exception occurs during processing, the following corresponding exception handling logic is executed:

[0119] If the remote storage copy fails, a read / write request failure is reported to the host, and the host will re-execute the copy according to the traditional process.

[0120] If the starting offset, data size, or corresponding MD5 checksum of a read / write request does not match before reaching the matching threshold, the matching is deemed to have failed, and subsequent read / write requests are forwarded to remote storage for processing according to the traditional process.

[0121] Example 2

[0122] This embodiment provides a specific implementation architecture and complete workflow of a file copy acceleration system and method under a user-space file system unloading device. Essentially, this embodiment identifies copy operations on the DPU side, and directly calls the copy capability provided by the remote storage to perform the copy operation, thereby accelerating the copy speed on the host side.

[0123] To achieve the above objectives, the acceleration system in this embodiment is an extension of the existing "host-side + virtioFS + FPGA + DPU-side + virtioFS + FUSE" architecture, working collaboratively with the host and remote storage. The specific system composition and technical solution are as follows:

[0124] I. Detailed Explanation of System Composition

[0125] 1. Host side (HOST)

[0126] (1) VFS: Linux's built-in file system interface, used to provide a standard file access interface for user applications;

[0127] (2) FUSE kernel module: A kernel module that comes with the Linux kernel, which interfaces with the VFS and is used to convert user file operations into FUSE file requests;

[0128] (3) virtioFS driver module: Based on the virtio protocol, it communicates with the FPGA hardware module through the PCIe protocol. It is used to receive file operation requests (such as read / write / close) sent by the user application and forward them to the FPGA hardware module on the DPU side; at the same time, it receives the response message from the DPU side and feeds it back to the user application through the FUSE kernel module and VFS.

[0129] 2. FPGA hardware module

[0130] Soldered onto the DPU, it connects to the host via a PCIe interface, serving as a hardware communication bridge between the host and the DPU. It facilitates PCIe protocol conversion between the host and DPU sides, and can be mapped to user space via Igb_uio (a PCI driver), supporting memory access between the host and DPU and enabling high-speed transmission of file operation request and response messages.

[0131] 3. DPU end

[0132] (1) virtioFS service module: Based on the virtio protocol, it communicates with the host virtioFS driver module through the FPGA hardware module. It is used to receive file operation requests sent by the host, parse the requests and forward them to the FUSE module; at the same time, it receives the processing results of the FUSE module, encapsulates them into a response message in the virtio protocol format, and feeds them back to the host through the FPGA hardware module.

[0133] (2) FUSE module: Based on the FUSE architecture already deployed on the DPU side, it interfaces with the virtioFS service module and undertakes multiple core functions: ① Receive file read, write and close requests forwarded by the virtioFS service module; ② Intercept read and write requests and extract key parameters (including file name, starting offset, data size, read and write type); ③ For read and write requests that are successfully matched, return preset fake data directly to the host (without actual business significance, only used to respond to the host), without accessing the remote storage; ④ Process file close requests and link the storage-side copy module to complete the copy result confirmation and unexpected scenario handling; ⑤ When the match is unsuccessful, forward the read and write requests to the remote storage according to the traditional process.

[0134] (3) Stream identification module: connected to the FUSE module, used to identify and record the intercepted read and write requests in sequence.

[0135] Specifically, this includes: Sequential stream determination rules: If consecutive sequential read operations on the same file starting from a fixed position are detected, it is marked as a "read sequential stream"; if consecutive sequential write operations on the same file starting from a fixed position are detected, it is marked as a "write sequential stream"; Stream information recording: A four-tuple record is generated for each read and write request of each sequential stream. The four-tuple record includes: stream type (read / write stream), file path and name (file_name), starting offset (offset), data size (size), and the MD5 checksum of the corresponding data (the MD5 of the source file data is calculated for read requests, and the MD5 of the data to be written is calculated for write requests).

[0136] (4) Stream matching module: It is connected to the stream identification module, FUSE module and storage-side copy module. It is used to match and verify the identified read sequence stream and write sequence stream to determine whether it is a file copying behavior.

[0137] The matching conditions include: ① The starting offset, data size, and MD5 checksum of each corresponding read and write request in the read sequence stream and the write sequence stream are completely identical; ② The number of read and write requests that meet the matching conditions reaches the preset matching threshold (configurable, such as 5); ③ The remote storage where the source file and the target file are located is the same storage service.

[0138] Result processing: If all conditions are met, a "copy match successful signal" is sent to the FUSE module, and the storage information of the source file and the target file is synchronized to the storage-side copy module, triggering the remote storage-side copy; if not, a "match failure signal" is sent to the FUSE module, which will then handle the process according to the traditional procedure.

[0139] (5) Storage-side copy module: connected to the stream matching module and remote storage, used to trigger and coordinate with the remote storage to complete the direct copy.

[0140] The remote storage copy implementation logic is as follows: A new copy service is added within the remote storage, including a copy request parsing unit, a data block mapping unit, and a metadata synchronization unit. The copy request parsing unit receives copy requests from the DPU and parses the source / target file paths; the data block mapping unit directly establishes a mapping relationship between source file data blocks and target files within the storage, enabling direct data block-level copying (without data migration in or out); the metadata synchronization unit synchronously updates the target file metadata (size, creation time, etc.) to ensure file integrity. Upon receiving a "copy trigger signal" from the stream matching module, a copy request is sent to the remote storage, carrying information such as the source file path and target file path, triggering the remote storage's internal copy service to perform a data copy operation, without requiring data transfer via the DPU.

[0141] II. Detailed Implementation Methods and Workflow

[0142] (a) Initialization process

[0143] 1. When the system starts up, the host-side virtioFS driver module and the DPU-side virtioFS service module establish a PCIe communication connection through the FPGA hardware module, negotiate and establish request queues and response queues based on the virtio protocol, and complete the communication initialization.

[0144] 2. The FPGA hardware module initializes the PCIe protocol conversion logic and enables the host and DPU memory access function to ensure high-speed transmission of request and response messages.

[0145] 3. The DPU-side FUSE module initializes file system metadata and registers the interface with the virtioFS service module; the stream recognition module initializes the stream record cache (using an in-memory hash table, where the key is "file name + stream type" and the value is a list of four-tuple records); the stream matching module loads the preset matching threshold (5 by default, configurable); the storage-side copy module establishes a TCP connection with the copy service inside the remote storage and confirms that the command format and parameter passing rules are compatible.

[0146] 4. Remote storage copy service initialization: The copy request parsing unit loads the path parsing rules, the data block mapping unit initializes the block mapping table, and the metadata synchronization unit establishes an association with the storage metadata management module, completing the internal unit readiness.

[0147] (II) Normal File Copy Process

[0148] Taking a file copy operation performed by a host user (such as executing the command `cp / remote / src / fileA / remote / dst / fileB`) ​​as an example, the processing flow is as follows:

[0149] 1. The host initiates a sequential read request for the source file fileA and a sequential write request for the target file fileB through the VFS interface; the request is converted into a FUSE file request by the FUSE kernel module, encapsulated into the virtio protocol format by the virtioFS driver module, and transmitted to the FPGA hardware module through the PCIe interface.

[0150] 2. After the FPGA hardware module completes the protocol conversion, it forwards the request to the DPU-side virtioFS service module. The virtioFS service module parses the request, extracts key information (read / write type, file path, starting offset, data size), and forwards it to the FUSE module.

[0151] 3. The FUSE module intercepts read and write requests, extracts key parameters (example: read request: file_name=fileA, offset=0, size=4KB; write request: file_name=fileB, offset=0, size=4KB), and forwards them to the stream recognition module.

[0152] 4. Stream identification module determination: The read request for fileA starts from offset=0, and the offsets of subsequent consecutive read requests are 4KB and 8KB respectively (offset+size = next offset), which is determined to be a read sequential stream. The MD5 value of the requested data is calculated (denoted as md5_1), and a quadruple record (read stream, fileA, 0, 4KB, md5_1) is generated and stored in the stream record cache. Similarly, the write request for fileB is determined to be a write sequential stream. The MD5 value of the data to be written is calculated (which is consistent with md5_1 after calculation), and a quadruple record (write stream, fileB, 0, 4KB, md5_1) is generated and stored in the stream record cache, and fed back to the stream matching module.

[0153] 5. Stream matching module verification: The starting offset, data size, and MD5 checksum of the first 5 read and write requests are completely consistent, and it is found that the remote storage where fileA and fileB are located are the same CephFS service, which meets the matching conditions. A "copy matching success signal" is sent to the FUSE module, and a "copy trigger signal" and the storage path of fileA ( / poolA / src / fileA) and the storage path of fileB ( / poolA / dst / fileB) are sent to the storage-side copy module.

[0154] 6. The storage-side copy module sends a copy request to the remote storage, carrying the source / target file path; the copy service in the remote storage executes: ① The copy request parsing unit parses the path, confirms that the source file exists and the target file is writable; ② The data block mapping unit queries the data block distribution of fileA and establishes a data block mapping in the storage area of ​​fileB; ③ The metadata synchronization unit updates the metadata of fileB (size, permissions, etc.).

[0155] 7. After the FUSE module receives the "copy match successful signal", all subsequent intercepted read requests for fileA and write requests for fileB will directly return preset fake data to the host. This data will then be fed back to the host via the virtioFS service module and the FPGA hardware module. After receiving the preset fake data, the host considers the read and write operations to be complete and continues to execute the subsequent copy steps.

[0156] 8. After the host completes all read and write operations, it sends file close requests for fileA and fileB, which are then forwarded to the FUSE module.

[0157] 9. The FUSE module, in conjunction with the storage-side copy module, queries the copy results. After confirming that the remote storage copy is complete and returns a success message, it sends a message to the host indicating that the copy was successfully closed, thus completing the normal copy process.

[0158] (III) Unexpected Situation Handling Procedures

[0159] Scenario 1: Partial copy scenario where only a portion of the file data is copied to the host.

[0160] 1. The host performs a partial copy operation (e.g., the source file fileA is 10G, but the host only copies 5G of it continuously), and then sends a file close request for the target file fileB, which is forwarded to the FUSE module on the DPU side via the transmission link.

[0161] 2. The FUSE module extracts the starting offset (e.g., 36KB) and data size (e.g., 4KB) of the last read / write request in the write sequence stream, and calculates the actual size of the target file as 36KB + 4KB = 40KB.

[0162] 3. The FUSE module sends a file size adjustment request to the remote storage through the storage-side copy module, carrying the fileB path and the calculated actual size (40KB).

[0163] 4. After receiving the request, the remote storage performs the ftruncate operation to adjust the size of fileB to the actual size (40KB) and returns a success message.

[0164] 5. After the FUSE module confirms the adjustment is complete, it sends a message to the host indicating that fileB was successfully closed, ensuring that the target file size is consistent with the actual copied data.

[0165] Scenario 2: Data modification scenario where the host modifies the data and then writes it to the target file.

[0166] 1. The host reads the data corresponding to the first 3 read / write requests of fileA (a total of 12KB), modifies the data content corresponding to the 3rd request in memory (at this time, the MD5 value becomes md5_3', which is inconsistent with the md5_3 of the original read sequence stream), and then sends the write request to the DPU.

[0167] 2. The FUSE module on the DPU side intercepts the write request, extracts the MD5 checksum and forwards it to the stream recognition module. The stream recognition module updates the quadruple record corresponding to the write sequence stream (updating the MD5 value in it to md5_3').

[0168] 3. The stream matching module detects that the MD5 checksum (md5_3') of the data corresponding to the write request is inconsistent with the MD5 checksum (md5_3) of the read-write request corresponding to the read sequence stream. It determines that this is a data modification scenario and triggers the sending of a "match failure signal" to the FUSE module.

[0169] 4. After receiving the fileB file closure request subsequently sent by the host, the FUSE module reports closure failure to the host.

[0170] 5. After receiving the closing failure message, the host side considers the fast copy to have failed and re-executes the file copy operation according to the traditional process; the FUSE module on the DPU side forwards subsequent read and write requests according to the traditional process to ensure data accuracy.

[0171] (iv) Exception handling logic

[0172] When the following exceptions occur during processing, the system will handle them according to the corresponding logic:

[0173] 1. Remote storage copy failure: When the storage-side copy module receives a failure response from the remote storage copy service (such as the source file not existing, insufficient permissions, etc.), it immediately sends a feedback to the FUSE module; the FUSE module sends a feedback to the host that the read / write request failed, and the host re-executes the copy according to the traditional process.

[0174] 2. Stream matching interruption: If the starting offset, data size, or corresponding MD5 checksum of a read / write request does not match before the preset matching threshold is reached, the stream matching module determines that the matching has failed and sends a matching failure signal to the FUSE module; the FUSE module forwards subsequent read / write requests to the remote storage for processing according to the traditional process.

[0175] 3. Communication connection interruption: If the PCIe connection between the host and the DPU, or the TCP connection between the DPU and the remote storage is interrupted, the relevant modules will attempt to reconnect. If the reconnection fails, the FUSE module will report the closing failure to the host to ensure that no data is lost.

[0176] The acceleration system and method of this embodiment can accurately identify file copying behavior based on the existing offloading architecture. By triggering direct copying within remote storage and returning preset fake data by the FUSE module, it significantly reduces network bandwidth consumption, PCIe channel bandwidth consumption, hardware transmission resource occupation, and host-side CPU computing power loss caused by multiple rounds of data migration, thus significantly improving copying efficiency. At the same time, relying on a multi-module linkage processing mechanism to deal with unexpected scenarios such as partial copying and data modification, as well as network anomalies, it fully ensures the accuracy and integrity of data, and has strong practicality and feasibility.

[0177] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

[0178] This background section is provided to generally present the context of the invention. The work of the currently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither expressly nor impliedly acknowledged as prior art to the invention.

Claims

1. An acceleration system under a user-mode file system offload apparatus, characterized in that, include: Host-side, FPGA hardware module, DPU-side, and remote storage; The host is used to initiate file operation requests and transmit them to the DPU via the FPGA hardware module. The FPGA hardware module is connected to the host via a PCIe interface, serving as a hardware communication bridge between the host and the DPU, and is used to complete protocol conversion and file operation request transmission. The DPU includes a virtioFS service module, a FUSE module, a stream identification module, a stream matching module, and a storage-side copy module; The virtioFS service module is used to receive file operation requests sent by the host and forward them to the FUSE module; The FUSE module interfaces with the virtioFS service module to intercept read and write requests, extract key parameters, and forward them to the stream identification module. And after receiving the copy matching success signal sent by the stream matching module, the intercepted read / write request is directly returned to the host with preset fake data; The stream identification module is connected to the FUSE module and is used to identify the sequence of intercepted read and write requests, determine whether they are read sequence streams or write sequence streams, and generate corresponding stream information records. The stream matching module is connected to the stream identification module, the FUSE module, and the storage-side copy module. It is used to match and verify the identified read-order stream and write-order stream. If the matching conditions are met, a copy matching success signal is sent to the FUSE module, and a copy trigger signal and synchronization storage information are sent to the storage-side copy module. The storage-side copy module is connected to the stream matching module and the remote storage, and is used to send a copy request to the remote storage according to the copy trigger signal; The remote storage is used to receive the copy request and trigger the copy service inside the remote storage to perform a data copy operation.

2. The system of claim 1, wherein the user-mode file system offload device is a virtual file system filter driver. The host terminal includes: VFS is used to provide a standard file access interface for user applications; The FUSE kernel module, which interfaces with the VFS, is used to convert user file operations into FUSE file requests; The virtioFS driver module is used to encapsulate the FUSE file request into the virtio protocol format and send it to the DPU through the FPGA hardware module; at the same time, it receives the response message from the DPU and feeds it back to the user application through the FUSE kernel module and VFS.

3. The acceleration system under a user-mode file system unloading device according to claim 1, characterized in that, The stream identification module performs sequential stream identification on the intercepted read and write requests and generates corresponding stream information records, specifically including: If consecutive sequential read operations on the same file starting from a fixed position are detected, it is marked as a read sequential stream; if consecutive sequential write operations on the same file starting from a fixed position are detected, it is marked as a write sequential stream. For each read / write request of a sequential stream, a four-tuple record is generated and stored in the stream record cache. The four-tuple record includes: stream type, file path and name, starting offset, data size, and the MD5 checksum of the corresponding data.

4. The acceleration system under a user-mode file system unloading device according to claim 3, characterized in that, The matching conditions include: The starting offset, data size, and MD5 checksum of each corresponding read and write request are completely identical between the read sequential stream and the write sequential stream. The number of read / write requests that meet the matching criteria has reached the preset matching threshold; The source file and the target file are stored on the same remote storage service.

5. The acceleration system under a user-mode file system unloading device according to claim 1, characterized in that, The copy service within the remote storage includes: The copy request parsing unit is used to receive the copy request issued by the storage-side copy module and parse the source file path and the target file path; The data block mapping unit is used to establish a mapping relationship between source file data blocks and target files within the remote storage, enabling direct copying at the data block level; The metadata synchronization unit is used to synchronize and update the metadata of the target file to ensure file integrity.

6. A method for accelerating file copying based on the acceleration system according to any one of claims 1-5, characterized in that, Includes the following steps: The host initiates a file operation request and transmits it to the DPU via the FPGA hardware module. The DPU receives the file operation request, intercepts the read and write requests, extracts key parameters for sequential stream identification, determines whether it is a read sequential stream or a write sequential stream, and generates corresponding stream information records. The identified read sequence stream and write sequence stream are matched and verified. If the matching conditions are met, the copy matching is confirmed to be successful. After confirming a successful copy match, a copy request is sent to the remote storage to trigger a direct copy within the remote storage. Meanwhile, for any subsequent intercepted read / write requests, preset fake data is returned directly to the host.

7. The file copying acceleration method according to claim 6, characterized in that, After directly returning preset fake data to the host, a normal file copy result confirmation step is also included: After receiving the preset fake data, the host considers the read / write operation to be complete and sends a file close request to the DPU. The DPU processes the file closing request and queries the copy results in the remote storage. If the remote storage copy is confirmed to be complete, a successful shutdown message will be sent to the host.

8. The file copying acceleration method according to claim 7, characterized in that, For partial copy scenarios where only a portion of the file data is copied to the host, the following are also included: Extract the starting offset and data size of the last read / write request in the write sequence stream, and calculate the actual size of the target file; Send a file size adjustment request to the remote storage, carrying the target file path and the actual size; After receiving the file size adjustment request, the remote storage performs the ftruncate operation to adjust the target file size to the actual size, and sends a successful closure feedback to the host after confirming the adjustment is complete.

9. The file copying acceleration method according to claim 6, characterized in that, The generation of the corresponding stream information record includes extracting and recording the MD5 checksum value of the data corresponding to the read / write request; If the data modification scenario involves modifying data on the host and then writing it to the target file, the processing steps include: When the MD5 checksum of the data corresponding to a subsequent write request is found to be inconsistent with the MD5 checksum of the read and write requests corresponding to the read sequence stream, it is determined to be a data modification scenario, and a matching failure signal is triggered. After receiving a subsequent file closure request from the host, report the closure failure to the host. After receiving the shutdown failure message, the host re-executes the file copy operation according to the traditional process, and the DPU forwards subsequent read and write requests according to the traditional process.

10. The file copying acceleration method according to claim 6, characterized in that, The key parameters include the starting offset, data size, and MD5 checksum; successful copy matching requires meeting certain matching conditions, including a preset matching threshold for the number of read / write requests meeting the conditions; when an exception occurs during processing, the following corresponding exception handling logic is executed: If the remote storage copy fails, a read / write request failure is reported to the host, and the host will re-execute the copy according to the traditional process. If the starting offset, data size, or corresponding MD5 checksum of a read / write request does not match before reaching the matching threshold, the matching is deemed to have failed, and subsequent read / write requests are forwarded to remote storage for processing according to the traditional process.

Images