An eBPF-based block-level deduplication method and system

Abstract

This invention discloses a block-level reduced copy method and system based on eBPF, relating to the field of data protection technology. The method includes the following steps: deploying and initializing the eBPF block-level I / O capture service; refreshing the file system cache and creating a copy-on-write (COW) temporary snapshot based on eBPF, and initializing the COW bitmap; during the validity period of the COW snapshot, generating a valid data block bitmap based on file system metadata, with the eBPF program ensuring data read consistency during this process. This invention utilizes eBPF technology to dynamically create a temporary copy-on-write snapshot environment at the application layer, intercepts system read / write calls, accurately captures valid data block information under the file system consistency view, and combines real-time incremental changes to achieve efficient, accurate, and consistent reduced data copying. The entire process requires no modification to the kernel or file system code, exhibiting non-intrusiveness and high versatility, and is applicable to various Linux environments and mainstream file systems.

Description

Technical Field

[0001] This invention relates to the field of data protection technology, specifically to a block-level thin copy method and system based on eBPF. Background Technology

[0002] With the development of information technology, data has become a core asset for enterprises and institutions. Data backup is a key means to ensure business continuity and data security. Block-level backup technology is widely used because of its high backup efficiency, support for full / incremental backup, and better guarantee of data consistency (usually by combining with snapshot technology). Especially in virtualized environments and large data centers, block-level backup is the mainstream backup solution. In recent years, the maturity of eBPF (extended Berkeley Packet Filter) technology has provided a new technical path. eBPF allows developers to dynamically load secure and efficient programs into the system without modifying the kernel source code. By intercepting system calls, it can achieve non-intrusive, high-performance, programmable real-time observation and control of I / O operations, demonstrating powerful system call tracing capabilities.

[0003] However, existing block-level replication technologies still have significant shortcomings, mainly in the following aspects:

[0004] 1) Difficulty in kernel version adaptation: Different versions of the Linux kernel have differences in block device management and file system implementation. Traditional block-level thin copy technology cannot work properly on different kernel versions. Targeted development and adaptation are required. Developers need to invest a lot of time and energy in adaptation work, which increases the cost and difficulty of technology application.

[0005] 2) Insufficient consistency guarantee mechanism: Traditional block-level thin replication technology is insufficient in ensuring data consistency. If the system fails or an abnormal situation occurs (such as a sudden power outage) during the creation of snapshots or data replication, traditional technology cannot ensure the consistency between the replicated data and the unreplicated data, thereby affecting the integrity and availability of the data.

[0006] 3) High invasiveness: Some simplified replication schemes require modification of the file system kernel code or rely on specific storage hardware characteristics, resulting in high deployment and maintenance costs and poor versatility;

[0007] Therefore, there is an urgent need in this field for a method that can achieve efficient and refined block-level simplified data replication without significantly increasing system overhead or being intrusive. Summary of the Invention

[0008] This invention provides a block-level simplified copy method and system based on eBPF, which can effectively solve the problems of kernel version adaptation difficulties, insufficient consistency guarantee mechanism and high intrusiveness in the prior art mentioned in the background.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a block-level thin copy method based on eBPF, which achieves efficient, accurate, and consistent thin data copying, comprising the following steps:

[0010] S1. Deploy and initialize the eBPF block-level I / O capture service;

[0011] S2. Refresh the file system cache and create a copy-on-write (COW) temporary snapshot based on eBPF, and initialize the COW bitmap;

[0012] S3. During the validity period of the COW snapshot, a valid data block bitmap is generated based on the file system metadata, and the consistency of data reading is guaranteed by the eBPF program during this process.

[0013] S4. Continuously capture new write I / O operations, maintain and update the write I / O capture bitmap;

[0014] S5. Merge the valid data block bitmap and the write I / O capture bitmap to generate the final data synchronization bitmap;

[0015] S6. Unload the COW temporary snapshot and release related resources;

[0016] S7. Based on the data synchronization bitmap, start and execute the block-level data replication process.

[0017] According to the above technical solution, step S1 specifically includes the following steps:

[0018] S11. Load the eBPF application framework;

[0019] S12, Attach the eBPF program to the system call hook;

[0020] S13. Initialize user-mode control services;

[0021] S14. Initialize memory and data structures;

[0022] S15. Start the eBPF program to capture I / O requests;

[0023] In step S11, the pre-compiled eBPF program is loaded into the system;

[0024] In step S12, the key functions of the eBPF program are attached to the system call interface via tracepoint or uprobe method;

[0025] In step S13, the user-mode daemon process or service module is started.

[0026] S14 initializes the key data structures required for subsequent steps;

[0027] In step S15, the eBPF parameter configuration is completed, and a start signal is sent through the user-space program to enable the eBPF program to start capturing I / O requests from the block device layer.

[0028] According to the above technical solution, step S2 specifically includes the following steps:

[0029] S21. Refresh the file system cache;

[0030] S22. Start COW mode;

[0031] S23, Intercept copy before write;

[0032] S24. Read and save the original data;

[0033] S25, Update COW bitmap;

[0034] S26. Allow the original write operation;

[0035] In step S21, the file system interface is called to write the data in the file system cache to the disk;

[0036] S22, instructs the eBPF program to enter COW snapshot mode;

[0037] In step S23, the eBPF program intercepts a new write I / O request when it arrives.

[0038] In step S24, the original data at the target address of the write request is read by the eBPF program and temporarily stored in a pre-allocated temporary memory buffer.

[0039] S25 is to mark the position of the target data block of the write request as changed in the COW bitmap;

[0040] In step S26, after the original data is saved, the eBPF program allows the write I / O request to continue to execute normally and write the new data to the target block device location.

[0041] According to the above technical solution, step S3 specifically includes the following steps:

[0042] S31. Scan file system metadata;

[0043] S32. Generate a valid data block bitmap;

[0044] S33 and eBPF programs ensure data read consistency;

[0045] In step S31, the file system metadata is traversed and analyzed to determine which data blocks are valid data blocks that have been allocated and used.

[0046] In S32, each bit of the bitmap corresponds to a data block on the disk;

[0047] In step S33, for each read I / O request, the eBPF program queries the COW bitmap to determine whether the data block to be read was modified during the COW period.

[0048] According to the above technical solution, step S4 specifically includes the following steps:

[0049] S41. Create a write I / O capture bitmap;

[0050] S42. Real-time capture of write I / O operations;

[0051] S43. Update the write I / O capture bitmap;

[0052] S41 adopts the same bit-level storage structure as the COW bitmap, with all bit values ​​initially set to 0.

[0053] In step S42, the eBPF program continuously monitors I / O requests at the block device layer;

[0054] In step S43, for each valid write I / O, the value of the bit corresponding to the physical block address covered by the write I / O is marked as 1.

[0055] According to the above technical solution, step S5 specifically includes the following steps:

[0056] S51, Read the valid data block bitmap and write the I / O capture bitmap;

[0057] S52, Perform bitmap merging operation;

[0058] S53. Generate the final data synchronization bitmap;

[0059] In step S51, the previously generated valid data block bitmap and write I / O capture bitmap are read from memory;

[0060] In step S52, the write I / O capture bitmap is used as the main bitmap, and the valid data block bitmap is merged into the main bitmap;

[0061] In step S53, after the bitmap merging operation, the final data synchronization bitmap is generated.

[0062] According to the above technical solution, step S6 specifically includes the following steps:

[0063] S61. Stop the COW mechanism;

[0064] S62. Delete COW temporary snapshot;

[0065] S63, Release the COW bitmap and related memory;

[0066] In step S61, the copy-on-write mechanism is stopped through the eBPF program or system interface, and no copy operation is performed for new write operations. Subsequent write operations will be performed directly on the original disk block.

[0067] In step S62, the interface provided by the file system or storage device is called to delete the previously created COW temporary snapshot;

[0068] In step S63, the bitmap used to record the COW operation and other memory resources allocated during the COW process are released.

[0069] According to the above technical solution, step S7 specifically includes the following steps:

[0070] S71, Read the data synchronization bitmap;

[0071] S72. Obtain the data block to be copied;

[0072] S73, Initiate block-level data replication;

[0073] S74. Monitor the replication progress and status;

[0074] S71 reads the final generated data synchronization bitmap from memory;

[0075] In step S72, the data synchronization bitmap is traversed, and the addresses of data blocks with corresponding bit values ​​of 1 are recorded to form a list of data blocks that need to be copied.

[0076] In step S73, the data copying program is invoked, and the corresponding data blocks are read from the source disk according to the list of data blocks to be copied, and the data is written to the target disk.

[0077] S74 monitors the progress and status of data replication in real time during the data replication process, and handles any errors and anomalies that occur.

[0078] A block-level thin replication system based on eBPF includes an eBPF application layer capture module, a user space control service module, a COW snapshot management module, a valid data identification module, a bitmap management module, and a data replication engine module.

[0079] According to the above technical solution, the core function of the eBPF application layer capture module is to capture and filter system I / O calls in real time;

[0080] The user space control service module is responsible for scheduling and coordinating the entire replication process, and monitors the system health status and task progress throughout the process.

[0081] The COW snapshot management module is specifically responsible for maintaining a dynamic, temporary, consistent data view at the application layer, and managing the COW bitmap and the corresponding memory buffer.

[0082] The core task of the valid data identification module is to accurately locate and mark all data blocks on the disk that store valid data by scanning and parsing the metadata of the file system;

[0083] The bitmap management module maintains three core bitmaps: the COW bitmap, the write I / O capture bitmap, and the valid data block bitmap, and is responsible for integrating and processing all bitmap information;

[0084] The data replication engine module performs efficient block-level data replication based on the data synchronization bitmap and is responsible for managing the concurrency of replication tasks and handling exceptions during transmission.

[0085] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0086] 1. By leveraging eBPF technology to dynamically create temporary copy-on-write snapshot environments at the application layer, and intercepting system read / write calls, this technology accurately captures valid data block information under the file system consistency view. Combined with real-time incremental changes, it achieves efficient, accurate, and consistent simplified data replication. The entire process requires no modification to the kernel or file system code, making it non-intrusive and highly versatile. It is applicable to various Linux environments and mainstream file systems, solving the problems of low resource utilization, high invasiveness, poor performance, and limited versatility in existing block-level replication technologies. Furthermore, by dynamically constructing a list of changed blocks, it replicates only the changed blocks, greatly improving replication efficiency and reducing the waste of network bandwidth and storage resources. This provides strong technical support for ensuring business continuity and data security.

[0087] 2. Through a three-dimensional graph collaboration mechanism, valid data blocks are accurately identified and real-time incremental changes are captured. Only necessary data is copied, achieving efficient and accurate simplified replication. This significantly reduces network bandwidth consumption and target-end storage consumption, improving replication efficiency. Based on eBPF technology, COW snapshots are dynamically created at the application layer without modifying the kernel or file system, achieving zero-intrusion deployment. It provides a non-intrusive architecture and strong consistency guarantees. This mechanism provides a consistent view for metadata scanning, ensuring the integrity and reliability of the copied data. The core processing logic is efficiently executed by eBPF, eliminating context switching overhead and having minimal impact on production systems. Moreover, this solution is applicable to various Linux environments and mainstream file systems, demonstrating wide applicability and excellent versatility and flexibility. Attached Figure Description

[0088] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0089] In the attached diagram:

[0090] Figure 1 This is a flowchart of the block-level simplified copying method of the present invention;

[0091] Figure 2 This is a structural block diagram of the block-level simplified copy system of the present invention;

[0092] Figure 3 This is a schematic diagram of the COW bitmap identifier of the present invention;

[0093] Figure 4 This is a schematic diagram of the effective data block bitmap identifier of the present invention;

[0094] Figure 5 This is a schematic diagram of the write I / O capture bitmap identifier of the present invention;

[0095] Figure 6 This is a schematic diagram illustrating the merging operation of the effective data block bitmap and the write I / O capture bitmap of the present invention. Detailed Implementation

[0096] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0097] Example: Figure 1 As shown, this invention provides a technical solution: a block-level thin copy method based on eBPF. It utilizes eBPF technology to dynamically create a temporary, copy-on-write (COW) snapshot environment at the application layer. Within this environment, by intercepting and intelligently analyzing I / O operations, it accurately obtains valid data block information under the file system consistency view. Combined with real-time captured incremental changes, it ultimately achieves efficient, accurate, and consistent thin data copying, including the following steps:

[0098] S1. Deploy and initialize the eBPF block-level I / O capture service;

[0099] S2. Refresh the file system cache and create a copy-on-write (COW) temporary snapshot based on eBPF, and initialize the COW bitmap;

[0100] S3. During the validity period of the COW snapshot, a valid data block bitmap is generated based on the file system metadata, and the consistency of data reading is guaranteed by the eBPF program during this process.

[0101] S4. Continuously capture new write I / O operations, maintain and update the write I / O capture bitmap;

[0102] S5. Merge the valid data block bitmap and the write I / O capture bitmap to generate the final data synchronization bitmap;

[0103] S6. Unload the COW temporary snapshot and release related resources;

[0104] S7. Based on the data synchronization bitmap, start and execute the block-level data replication process.

[0105] Based on the above technical solution, S1 aims to build the eBPF runtime environment, load and initialize the core program for monitoring block device I / O, and provide basic support for subsequent operations. Specifically, it includes the following steps:

[0106] S11. Load the eBPF application framework;

[0107] S12, Attach the eBPF program to the system call hook;

[0108] S13. Initialize user-mode control services;

[0109] S14. Initialize memory and data structures;

[0110] S15. Start the eBPF program to capture I / O requests;

[0111] S11, Load the pre-compiled eBPF program into the system, which contains probes for intercepting system I / O calls;

[0112] S12, attach the key functions of the eBPF program to the system call interface using tracepoints to ensure that all pending I / O requests can be captured;

[0113] S13, Start the user-mode daemon process, which is used to receive control commands, manage the state of the eBPF program, and read the I / O event data output by the eBPF program;

[0114] S14, Initialize the key data structures required for subsequent steps to store I / O information, bitmap status and control flags;

[0115] S15, complete the eBPF parameter configuration, and send a start signal through the user-space program to enable the eBPF program to start capturing I / O requests from the block device layer.

[0116] like Figure 3As shown, based on the above technical solution, S2 aims to achieve non-intrusive COW snapshots through eBPF, avoiding the impact of traditional snapshot technology on business interruption, and recording the data change location through COW bitmap. Specifically, it includes the following steps:

[0117] S21. Refresh the file system cache;

[0118] S22. Start COW mode;

[0119] S23, Intercept copy before write;

[0120] S24. Read and save the original data;

[0121] S25, Update COW bitmap;

[0122] S26. Allow the original write operation;

[0123] S21, call the file system interface to write the data in the file system cache to the disk, ensuring that the data in memory is consistent with the data on the disk before the snapshot;

[0124] S22, Notify the eBPF program to enter COW snapshot mode;

[0125] S23, when a new write I / O request arrives, the eBPF program intercepts it;

[0126] S24, the eBPF program reads the original data at the target address of the write request and temporarily stores it in a pre-allocated temporary memory buffer;

[0127] S25 marks the location of the target data block for the write request as changed in the COW bitmap.

[0128] The COW bitmap uses a bit-level storage method, where each bit represents one physical block. A bit value of 1 indicates that a write I / O has occurred for the block and the original data is stored in the COW buffer, while a bit value of 0 indicates that no write I / O has occurred.

[0129] S26. After saving the original data, the eBPF program allows the write I / O request to continue executing normally and write the new data to the target block device location.

[0130] like Figure 4 As shown, based on the above technical solution, S3 is the core of ensuring data consistency. It determines valid data blocks through file system metadata analysis, thereby avoiding the copying of empty blocks, and uses the COW mechanism to return the historically consistent state during the reading process. Specifically, it includes the following steps:

[0131] S31. Scan file system metadata;

[0132] S32. Generate a valid data block bitmap;

[0133] S33 and eBPF programs ensure data read consistency;

[0134] S31, by traversing the file system metadata and analyzing this metadata, determine which data blocks are valid data blocks that have been allocated and used;

[0135] S32, each bit in the bitmap corresponds to a data block on the disk. When the data block is a valid data block, the value of the corresponding bit is set to 1, otherwise it is set to 0.

[0136] S33, For each read I / O request, the eBPF program queries the COW bitmap to determine whether the data block it requests to read has been modified during COW;

[0137] If yes, then the read request is intercepted, and the current data is not read from the disk. Instead, the original data before modification is returned from the COW temporary storage. If no, then the request is allowed to execute normally, and the current data is read from the block device.

[0138] like Figure 5 As shown, based on the above technical solution, S4 ensures that no new write I / O data is missed after the COW snapshot takes effect. Real-time changes are recorded by using a write I / O capture bitmap, which is the main bitmap. The specific steps include the following:

[0139] S41. Create a write I / O capture bitmap;

[0140] S42. Real-time capture of write I / O operations;

[0141] S43. Update the write I / O capture bitmap;

[0142] S41 uses the same bit-level storage structure as the COW bitmap, with all bit values ​​initially set to 0;

[0143] S42, the eBPF program continuously monitors I / O requests at the block device layer. When a new write operation is captured, it records the specific physical block address involved in the write operation.

[0144] S43. For each valid write I / O, mark the value of the bit corresponding to the physical block address covered by the write I / O as 1. If the same physical block is written multiple times, it only needs to be marked once to avoid duplicate recording.

[0145] like Figure 6 As shown, based on the above technical solution, step S5 synthesizes historical valid data with real-time incremental changes to obtain the final complete dataset that needs to be copied. This specifically includes the following steps:

[0146] S51, Read the valid data block bitmap and write the I / O capture bitmap;

[0147] S52, Perform bitmap merging operation;

[0148] S53. Generate the final data synchronization bitmap;

[0149] S51 reads the previously generated valid data block bitmap and write I / O capture bitmap from memory;

[0150] S52 uses the write I / O capture bitmap as the main bitmap and merges the valid data block bitmap into the main bitmap, that is, performs a bitwise OR operation on the two bitmaps, the valid data block bitmap and the write I / O capture bitmap.

[0151] Specifically, for each bit in the bitmap, if the corresponding bit in the valid data block bitmap is 1 or the corresponding bit in the write I / O capture bitmap is 1, then the corresponding bit in the final data synchronization bitmap is set to 1; otherwise, it is set to 0. Thus, a bit of 1 in the final data synchronization bitmap indicates that the data block is a valid data block or that a write operation has occurred during the copying process, requiring data synchronization.

[0152] S53, after the bitmap merging operation, generates the final data synchronization bitmap. This bitmap will serve as the basis for subsequent block-level data copying, guiding the copying program to operate only on the data blocks that need to be synchronized.

[0153] Based on the above technical solution, S6 involves unloading the COW temporary snapshot and releasing related resources after determining the final range of data to be copied. This specifically includes the following steps:

[0154] S61. Stop the COW mechanism;

[0155] S62. Delete COW temporary snapshot;

[0156] S63, Release the COW bitmap and related memory;

[0157] S61, through the eBPF program, stops the copy-on-write mechanism, no longer performs copy operations for new write operations, and subsequent write operations will be performed directly on the original disk block;

[0158] S62 calls the interface provided by the file system to delete the previously created COW temporary snapshot, releasing the disk space and other system resources occupied by the snapshot;

[0159] S63 releases the bitmap used to record COW operations and other memory resources allocated during the COW process, ensuring effective reclamation of system resources and avoiding memory leaks.

[0160] Based on the above technical solution, S7 initiates and executes the block-level data replication process according to the data synchronization bitmap, achieving efficient and streamlined block-level replication, specifically including the following steps:

[0161] S71, Read the data synchronization bitmap;

[0162] S72. Obtain the data block to be copied;

[0163] S73, Initiate block-level data replication;

[0164] S74. Monitor the replication progress and status;

[0165] S71 reads the final generated data synchronization bitmap from memory;

[0166] S72, traverse the data synchronization bitmap, record the addresses of data blocks with corresponding bit values ​​of 1, and form a list of data blocks to be copied;

[0167] S73 calls the data copying program, reads the corresponding data blocks from the source disk according to the list of data blocks to be copied, and writes the data to the target disk. During the copying process, parallel copying technology can be used to improve copying efficiency.

[0168] S74 monitors the progress and status of data replication in real time during the data replication process, and handles errors and anomalies that occur to ensure the reliability and integrity of data replication. Errors and anomalies specifically include disk failures and network interruptions.

[0169] like Figure 2 As shown, a block-level thin replication system based on eBPF includes an eBPF application layer capture module, a user space control service module, a COW snapshot management module, an effective data identification module, a bitmap management module, and a data replication engine module. Through the collaborative work of these modules, block-level thin replication is achieved.

[0170] Based on the above technical solution, the core function of the eBPF application layer capture module is to capture and filter system I / O calls in real time. By attaching to the system call interface, it accurately intercepts read and write operations, executes copy-on-write logic, and provides support for creating consistent snapshots.

[0171] The User Space Control Service module is responsible for scheduling and coordinating the entire replication process, including loading, configuring and starting eBPF programs, sending commands to enable or disable COW snapshots, triggering the generation of valid data bitmaps, bitmap merging and key stages of data replication, and monitoring the system health status and task progress throughout the process.

[0172] The COW snapshot management module is specifically responsible for maintaining a dynamic, temporary, consistent data view at the application layer, managing the COW bitmap and the corresponding memory buffer;

[0173] When a write operation occurs, this module ensures that the original data is properly preserved. When it is necessary to read data at a snapshot time, it can route the read request to a temporary buffer by querying the COW bitmap, thereby providing a consistent data source for file system scanning.

[0174] The core task of the valid data identification module is to accurately locate and mark all data blocks on the disk that store valid data by scanning and parsing the metadata of the file system. With the assistance of COW snapshots, it can obtain a consistent view that is not affected by concurrent writes, thereby accurately generating a valid data block bitmap. This bitmap ensures that the copying process does not transfer unallocated empty blocks, greatly improving the copying efficiency.

[0175] The bitmap management module maintains three core bitmaps: the COW bitmap, the write I / O capture bitmap, and the valid data block bitmap. It is responsible for integrating and processing all bitmap information. Its most critical function is to execute the bitmap merging algorithm, which performs a logical OR operation on the bitmap that identifies historical valid data and the bitmap that records real-time increments to generate a final and complete data synchronization bitmap, which serves as a precise task list for the copy operation.

[0176] The COW bitmap records which physical blocks were written to during the creation of a temporary snapshot and triggers a backup of the original data to ensure the consistency of subsequent metadata readings.

[0177] Bit value meaning: 1 indicates that the data block has undergone a write I / O operation since COW snapshots were enabled. The original data of the data block on the disk has been saved to a dedicated COW cache. Any request that needs to read the data at the snapshot time of the block will retrieve the original data from the cache to ensure the consistency of metadata reading. 0 indicates that the data block has not been written since COW snapshots were enabled. The data on the disk at present is the data at the snapshot time. Read requests can directly access the disk.

[0178] The write I / O capture bitmap continuously and in real time captures all new write I / O operations during and after the COW snapshot takes effect. It records all changes that occur to the data block within the time window from the start of the snapshot to the completion of replication, ensuring that no new data changes generated during the replication preparation process are missed.

[0179] Bit value meaning: 1 indicates that a new write I / O operation has occurred in the replication preparation process for this data block. Regardless of whether the block was a valid data block before, it must be included in the replication scope because its content has been updated to ensure the integrity of the target data. 0 indicates that no new write I / O operation has occurred in the replication preparation process for this data block, and its status is completely determined by the valid data block bitmap.

[0180] The valid data block bitmap is a consistent view based on COW snapshots. By analyzing file system metadata, it identifies data blocks on the disk that have been truly allocated by the file system and store valid user data. It filters logically valid data from the physical full data and avoids copying unallocated empty blocks, thereby improving copying efficiency.

[0181] Bit value meaning: 1 indicates that the physical block has been allocated by the file system and stores valid business data, and should be included in the subsequent copying scope; 0 indicates that the physical block is a free block that has not been allocated by the file system or a zero block that has been allocated but has no valid data, and does not need to be included in the copying scope, and can be directly filtered to achieve simplified copying.

[0182] The data replication engine module performs efficient block-level data replication based on the data synchronization bitmap. By parsing the bitmap, it generates a list of data blocks to be transferred, schedules these data blocks to be read from the source disk, transmitted over the network, and finally written to the target disk. It is also responsible for managing the concurrency of replication tasks, handling exceptions during the transmission process, and ensuring the reliability and integrity of the entire replication process.

[0183] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A block-level reduced copy method based on eBPF, characterized in that: To achieve efficient, accurate, and consistent streamlined data replication, the following steps are involved: S1. Deploy and initialize the eBPF block-level I / O capture service; S2. Refresh the file system cache and create a copy-on-write (COW) temporary snapshot based on eBPF, and initialize the COW bitmap; S3. During the validity period of the COW snapshot, a valid data block bitmap is generated based on the file system metadata, and the eBPF program ensures the consistency of data reading during this process. S4. Continuously capture new write I / O operations, maintain and update the write I / O capture bitmap; S5. Merge the valid data block bitmap and the write I / O capture bitmap to generate the final data synchronization bitmap; S6. Unload the COW temporary snapshot and release related resources; S7. Based on the data synchronization bitmap, start and execute the block-level data replication process.

2. The block-level reduced replication method based on eBPF according to claim 1, characterized in that: S1 specifically includes the following steps: S11. Load the eBPF application framework; S12, Attach the eBPF program to the system call hook; S13. Initialize user-mode control services; S14. Initialize memory and data structures; S15. Start the eBPF program to capture I / O requests; In step S11, the pre-compiled eBPF program is loaded into the system; In step S12, the key functions of the eBPF program are attached to the system call interface via tracepoint or uprobe method; In step S13, a user-mode daemon process or service module is started. S14 initializes the key data structures required for subsequent steps; In step S15, the eBPF parameter configuration is completed, and a start signal is sent through the user-space program to enable the eBPF program to start capturing I / O requests from the block device layer.

3. The block-level reduced replication method based on eBPF according to claim 1, characterized in that: S2 specifically includes the following steps: S21. Refresh the file system cache; S22. Start COW mode; S23, Intercept copy before write; S24. Read and save the original data; S25, Update COW bitmap; S26. Allow the original write operation; In step S21, the file system interface is called to write the data in the file system cache to the disk; S22, instructs the eBPF program to enter COW snapshot mode; In step S23, the eBPF program intercepts a new write I / O request when it arrives. In step S24, the original data at the target address of the write request is read by the eBPF program and temporarily stored in a pre-allocated temporary memory buffer. S25 is to mark the position of the target data block of the write request as changed in the COW bitmap; In step S26, after the original data is saved, the eBPF program allows the write I / O request to continue to execute normally and write the new data to the target block device location.

4. The block-level reduced replication method based on eBPF according to claim 1, characterized in that: S3 specifically includes the following steps: S31. Scan file system metadata; S32. Generate a valid data block bitmap; S33 and eBPF programs ensure data read consistency; In step S31, the file system metadata is traversed and analyzed to determine which data blocks are valid data blocks that have been allocated and used. In S32, each bit of the bitmap corresponds to a data block on the disk; In step S33, for each read I / O request, the eBPF program queries the COW bitmap to determine whether the data block to be read was modified during the COW period.

5. A block-level reduced copy method based on eBPF according to claim 1, characterized in that: S4 specifically includes the following steps: S41. Create a write I / O capture bitmap; S42. Real-time capture of write I / O operations; S43. Update the write I / O capture bitmap; S41 adopts the same bit-level storage structure as the COW bitmap, with all bit values ​​initially set to 0. In step S42, the eBPF program continuously monitors I / O requests at the block device layer; In step S43, for each valid write I / O, the value of the bit corresponding to the physical block address covered by the write I / O is marked as 1.

6. The block-level reduced replication method based on eBPF according to claim 1, characterized in that: S5 specifically includes the following steps: S51, Read the valid data block bitmap and write the I / O capture bitmap; S52, Perform bitmap merging operation; S53. Generate the final data synchronization bitmap; In step S51, the previously generated valid data block bitmap and write I / O capture bitmap are read from memory; In step S52, the write I / O capture bitmap is used as the main bitmap, and the valid data block bitmap is merged into the main bitmap; In step S53, after the bitmap merging operation, the final data synchronization bitmap is generated.

7. The block-level reduced replication method based on eBPF according to claim 1, characterized in that: S6 specifically includes the following steps: S61. Stop the COW mechanism; S62. Delete COW temporary snapshot; S63, Release the COW bitmap and related memory; In step S61, the copy-on-write mechanism is stopped through the eBPF program or system interface, and no copy operation is performed for new write operations. Subsequent write operations will be performed directly on the original disk block. In step S62, the interface provided by the file system or storage device is called to delete the previously created COW temporary snapshot; In step S63, the bitmap used to record the COW operation and other memory resources allocated during the COW process are released.

8. A block-level reduced copy method based on eBPF according to claim 1, characterized in that: S7 specifically includes the following steps: S71, Read the data synchronization bitmap; S72. Obtain the data block to be copied; S73, Initiate block-level data replication; S74. Monitor the replication progress and status; S71 reads the final generated data synchronization bitmap from memory; In step S72, the data synchronization bitmap is traversed, and the addresses of data blocks with corresponding bit values ​​of 1 are recorded to form a list of data blocks that need to be copied. In step S73, the data copying program is invoked, and the corresponding data blocks are read from the source disk according to the list of data blocks to be copied, and the data is written to the target disk. S74 monitors the progress and status of data replication in real time during the data replication process, and handles any errors and anomalies that occur.

9. A block-level thin copy system based on eBPF, comprising a block-level thin copy method based on eBPF according to any one of claims 1-8, characterized in that: It includes the eBPF application layer capture module, user space control service module, COW snapshot management module, valid data identification module, bitmap management module, and data replication engine module.

10. A block-level thin copy system based on eBPF according to claim 9, characterized in that: The core function of the eBPF application layer capture module is to capture and filter system I / O calls in real time. The user space control service module is responsible for scheduling and coordinating the entire replication process, and monitors the system health status and task progress throughout the process. The COW snapshot management module is specifically responsible for maintaining a dynamic, temporary, consistent data view at the application layer, and managing the COW bitmap and the corresponding memory buffer. The core task of the valid data identification module is to accurately locate and mark all data blocks on the disk that store valid data by scanning and parsing the metadata of the file system; The bitmap management module maintains three core bitmaps: the COW bitmap, the write I / O capture bitmap, and the valid data block bitmap, and is responsible for integrating and processing all bitmap information; The data replication engine module performs efficient block-level data replication based on the data synchronization bitmap and is responsible for managing the concurrency of replication tasks and handling exceptions during transmission.

Images