A distributed file remote replication method, device, equipment and medium

Abstract

The application discloses a distributed file remote replication method and device, equipment and medium, applied to the computer field, comprising; determining the target replication file from the file stored in the master cluster according to the replication directory specified information and the file attribute filtering condition, and generating the file snapshot corresponding to the target replication file; obtaining the difference data range between the reference snapshot in the file snapshot and the destination snapshot in the file snapshot; the reference snapshot is a completed replication snapshot, and the destination snapshot is a to-be-replicated snapshot; determining the complete data of each data object according to the object identifier associated with the target replication file, intercepting the difference data segment from the complete data based on the difference data range, summarizing the difference data segments of all data objects, and migrating the corresponding summary result to the slave cluster. The application reduces the data transmission amount in the remote replication process, especially for the large file small change scene, can significantly reduce the data processing and transmission time consumption, and improves the replication efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a distributed file remote copying method, apparatus, device, and medium. Background Technology

[0002] Distributed storage clusters are now being used more and more widely. With the continuous advancement of digitalization, data is gradually becoming the core of enterprise operations, and users are placing increasingly higher demands on the stability of storage systems that hold this data. To ensure the continuity, recoverability, and high availability of data storage, remote disaster recovery solutions have emerged, and remote replication technology is one of the key technologies in these solutions: its core idea is to synchronously replicate data multiple times to different locations, thereby minimizing the risk of data loss due to natural disasters or human-caused damage.

[0003] In current technology, files with changed data require a full copy. This is especially true for large files, where even minor data changes can significantly impact copying efficiency and performance. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a distributed file remote copying method, apparatus, device, and medium, which can reduce the amount of data transmission during the remote copying process. Especially for scenarios involving large files with minor changes, it can significantly reduce data processing and transmission time and improve copying efficiency. The specific solution is as follows:

[0005] In a first aspect, this application discloses a distributed file remote copying method, including:

[0006] Determine the remote replication strategy that associates the remote replication links between the master cluster and the slave cluster; the remote replication strategy includes information on the replication directory and file attribute filtering conditions.

[0007] Based on the specified information of the replication directory and the file attribute filtering conditions, the target replication file is determined from the files stored in the main cluster, and a file snapshot corresponding to the target replication file is generated;

[0008] Obtain the range of difference data between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is the snapshot that has been copied, and the destination snapshot is the snapshot to be copied.

[0009] The complete data of each data object is determined based on the object identifier associated with the target replicated file. Based on the range of difference data, the difference data segments are extracted from the complete data. The difference data segments of all data objects are summarized, and the corresponding summary results are migrated to the slave cluster.

[0010] Optionally, the target file to be replicated can be determined from the files stored in the primary cluster based on the information specified in the replication directory and the file attribute filtering criteria, including:

[0011] If a remote replication link is associated with only one remote replication policy, then any file stored in the primary cluster is located in the directory corresponding to the replication directory specified in the remote replication policy, and the file attributes of any file meet the file attribute filtering conditions in the remote replication policy, then that file will be identified as the target replication file.

[0012] Optionally, the target file to be replicated can be determined from the files stored in the primary cluster based on the information specified in the replication directory and the file attribute filtering criteria, including:

[0013] If a remote replication link is associated with multiple remote replication policies, then any file stored in the primary cluster is located in the directory corresponding to the replication directory specified in any remote replication policy, and the file attributes of any file meet the file attribute filtering conditions in any remote replication policy, then that file will be identified as the target replication file.

[0014] Optionally, the information to be copied includes the directory name, and the file attribute filtering conditions include one or more of the following: file name filtering conditions, file size filtering conditions, file operation time filtering conditions, user ID filtering conditions, or user group ID filtering conditions.

[0015] Optionally, the remote replication strategy that determines the remote replication link association between the master cluster and the slave cluster includes:

[0016] Set the directory name through the remote copy management interface, or use the directory name corresponding to the default directory;

[0017] The remote copy management interface allows you to set one or more target filenames to determine the filename filtering criteria in the file attribute filtering conditions based on one or more target filenames.

[0018] Select the file size comparison type through the remote copy management interface and set the corresponding file size threshold to determine the file size filter conditions in the file attribute filter conditions;

[0019] By selecting the file operation time type through the remote copy management interface and setting the corresponding time size threshold, the file operation time filter condition in the file attribute filter conditions can be determined.

[0020] Set the target user or target user group through the remote copy management interface to determine the user identifier or user group identifier filter conditions in the file attribute filter conditions.

[0021] The remote replication strategy is obtained by integrating the filtering conditions of directory name, file name, file size, file operation time, user ID, or user group ID to obtain the remote replication link association between the master cluster and the slave cluster.

[0022] Optional, distributed remote file copying methods also include:

[0023] The remote replication management interface displays the remote replication policies associated with the remote replication links between the master and slave clusters. The interface includes a list of remote replication links and details of individual remote replication links.

[0024] Optionally, the process of summarizing the difference data segments of all data objects and migrating the corresponding summary results to the slave cluster also includes:

[0025] If the migration fails, the header data corresponding to the target copied file in the cluster will be used as the new baseline snapshot, and the step of obtaining the range of difference data between the baseline snapshot and the target snapshot in the file snapshot will be re-executed based on the new baseline snapshot and the target snapshot.

[0026] Secondly, this application discloses a distributed file remote copying device, comprising:

[0027] The strategy determination module is used to determine the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster; the remote replication strategy includes information on the replication directory and file attribute filtering conditions.

[0028] The snapshot determination module is used to determine the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions, and to generate a file snapshot corresponding to the target copy file;

[0029] The difference data determination module is used to obtain the range of difference data between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is the snapshot that has been copied, and the destination snapshot is the snapshot to be copied.

[0030] The differential data migration module is used to determine the complete data of each data object based on the object identifier associated with the target copied file, extract differential data segments from the complete data based on the differential data range, summarize the differential data segments of all data objects, and migrate the corresponding summary results to the slave cluster.

[0031] Thirdly, this application discloses an electronic device, including:

[0032] Memory is used to store computer programs;

[0033] A processor is used to execute computer programs to implement the aforementioned disclosed distributed remote file copying method.

[0034] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned distributed remote file copying method.

[0035] As can be seen, this application proposes a distributed file remote replication method, including: determining a remote replication strategy that associates the remote replication links between the master cluster and the slave cluster; wherein, the remote replication strategy includes replication directory specification information and file attribute filtering conditions; determining the target replication file from the files stored in the master cluster according to the replication directory specification information and file attribute filtering conditions, and generating a file snapshot corresponding to the target replication file; obtaining the difference data range between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is a snapshot that has been replicated, and the destination snapshot is a snapshot to be replicated; determining the complete data of each data object according to the object identifier associated with the target replication file, extracting difference data segments from the complete data based on the difference data range, summarizing the difference data segments of all data objects, and migrating the corresponding summary results to the slave cluster. In summary, this application accurately determines the target file to be copied in the master cluster by specifying the copy directory information and filtering file attributes in the remote copy strategy. This avoids unnecessary files from occupying the slave cluster's storage resources, reduces the invalid storage overhead of the slave cluster, and improves the efficiency of storage resource utilization. Furthermore, this application obtains the range of difference data based on file snapshots, extracts the difference data segments from the complete data based on the difference data range, summarizes the difference data segments of all data objects, and migrates the corresponding summary results to the slave cluster, instead of copying the entire file. This significantly reduces the amount of data transmission during the remote copy process, especially for scenarios with large files and minor changes, which can significantly reduce data processing and transmission time and improve copy efficiency. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a distributed remote file copying method disclosed in this application;

[0038] Figure 2 This is a schematic diagram of the structure of a distributed file remote copying device disclosed in this application;

[0039] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] In current technology, files with changed data require a full copy. This is especially true for large files, where even minor data changes can significantly impact copying efficiency and performance.

[0042] Therefore, this application proposes a distributed remote file copying scheme, which reduces the amount of data transmission during the remote copying process. Especially for scenarios involving large files with minor changes, it can significantly reduce data processing and transmission time and improve copying efficiency.

[0043] This application discloses a distributed remote file copying method. See also Figure 1 As shown, the method includes:

[0044] Step S11: Determine the remote replication strategy for the remote replication link association between the master cluster and the slave cluster; wherein, the remote replication strategy includes the replication directory specification information and file attribute filtering conditions.

[0045] In this embodiment, the specified information for the copied directory includes the directory name, and the file attribute filtering conditions include one or more of the following: file name filtering conditions, file size filtering conditions, file operation time filtering conditions, user ID filtering conditions, or user group ID filtering conditions.

[0046] In this embodiment, the directory name is set through the remote replication management interface, or the directory name corresponding to the default directory is used; one or more target file names are set through the remote replication management interface to determine the file name filtering condition in the file attribute filtering conditions based on one or more target file names; the file size comparison type is selected through the remote replication management interface, and the corresponding file size threshold is set to determine the file size filtering condition in the file attribute filtering conditions; the file operation time type is selected through the remote replication management interface, and the corresponding time size threshold is set to determine the file operation time filtering condition in the file attribute filtering conditions; the target user or target user group is set through the remote replication management interface to determine the user identifier filtering condition or user group identifier filtering condition in the file attribute filtering conditions; the directory name, file name filtering condition, file size filtering condition, file operation time filtering condition, user identifier filtering condition, or user group identifier filtering condition are integrated to obtain the remote replication strategy that associates the remote replication links between the master cluster and the slave cluster.

[0047] For example, 1. Set a unique policy name for the remote replication policy to be configured through the remote replication management interface (this is a required configuration; the "Policy Name" field in the interface indicates it is mandatory); 2. Set a directory name through the remote replication management interface, or use the directory name corresponding to the default directory (the default directory is the root directory of the replication directory corresponding to the remote replication link, and only one directory entry can be configured for each remote replication policy); 3. Set one or more target file names through the remote replication management interface (e.g., "data_" can match all files starting with "data_") to determine the file name filtering conditions in the file attribute filtering conditions based on one or more target file names; 4. Select file size comparison through the remote replication management interface. 5. Select the file operation time type (including "greater than or equal to" and "less than") through the remote copy management interface, and set the corresponding time comparison type ("greater than or equal to" or "less than") and set the corresponding time size threshold (in days) to determine the file operation time filter condition in the file attribute filter conditions; 6. Set the target user or target user group through the remote copy management interface. When setting, you need to select the "local" type and use the target user's user ID (uid, User ID) to filter the file operation time. Match the user ID (GID) or group ID (GID) of the target user group to determine the user ID or group ID filtering conditions in the file attribute filtering conditions; 7. Integrate the directory name, file name, file size, file operation time, user ID, or group ID filtering conditions to obtain a remote replication policy associated with the remote replication link between the master cluster and the slave cluster; If multiple remote replication policies need to be configured for this remote replication link, repeat steps 1-6 above, and the total number of remote replication policies associated with a single remote replication link shall not exceed 16. If it exceeds this limit, the system will return error code -332 to indicate that the configuration limit has been exceeded.

[0048] Step S12: Determine the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions, and generate a file snapshot corresponding to the target copy file.

[0049] In this embodiment, if a remote replication link is associated with only one remote replication policy, then any file stored in the primary cluster is located in the directory corresponding to the replication directory specified in the remote replication policy, and the file attributes of any file meet the file attribute filtering conditions in the remote replication policy, then that file is determined as the target replication file. For example, when a remote replication link is bound to only one remote replication policy, "directory ownership verification + full attribute matching verification" must be performed on any file stored in the primary cluster; only when both are satisfied can it be determined as the target replication file. Directory attribution verification: Determines whether the file's storage path is entirely within the directory specified in the "Replication Directory Information" section of the remote replication strategy (if the strategy does not manually set a directory, it defaults to matching the root directory of the replication link, and only verifies whether the file is directly or indirectly stored in that root directory); Full attribute matching verification: For each attribute of the file, it matches one by one all the file attribute filtering conditions configured in the remote replication strategy. If the strategy configures "File name filtering condition", the file name must completely match one or more target file names (e.g., "report_" matches all files starting with "report_"); if the strategy configures "File size filtering condition", the actual file size must conform to the preset file size comparison type ("greater than or equal to" or "smaller than"). The file's operation time (btime creation time or mtime modification time) must meet the preset time comparison type ("greater than or equal to" or "less than") and the corresponding time threshold (in bits). If the policy is configured with "user ID filter condition" or "user group ID filter condition", the file's user uid (user ID) or user group gid (user group ID) must be completely consistent with the target uid / gid set in the policy (only local user / user group matching is supported, cross-cluster ID is not supported). The file will only be identified as the target copy file if the above "directory ownership verification passes" and "all file attribute filter conditions configured in the policy are successfully matched".

[0050] In this embodiment, if a remote replication link is associated with multiple remote replication policies, then when any file stored in the primary cluster is located in the directory corresponding to the replication directory specified in any remote replication policy, and the file attributes of any file meet the file attribute filtering conditions in any remote replication policy, that file will be identified as the target replication file. For example, when a remote replication link is bound to two or more remote replication policies, "directory ownership verification + single-policy attribute matching verification" is performed on any file stored in the primary cluster. As long as it meets both conditions of any policy, it can be identified as the target replication file. Single-policy dual-condition judgment: For each remote replication policy, "directory ownership verification (rules are the same as the directory verification above) + full attribute matching verification (rules are the same as the attribute verification above)" is used to determine whether the file meets the policy. Multi-policy "OR logic" matching: If the file meets the "directory ownership + full attribute matching" dual conditions of any policy (i.e., at least one policy determines that the file is a matching file), then there is no need to verify other policies, and the file is directly identified as the target replication file (the file meets the directory and attribute conditions of policy 1, and even if it does not meet the conditions of policies 2 and 3, it is still identified as the target replication file).

[0051] In this embodiment, after completing the screening of all target copy files, the system will automatically generate a corresponding file snapshot for each target copy file. The generation process and characteristics are as follows: (1) Snapshot generation mechanism: The snapshot technology based on COW (Copy On Write) is adopted. Before generating the snapshot, all data objects associated with the target copy file are preprocessed. If the data object may be modified later, the data of the current state of the data object will be copied to generate an independent snapshot object. Then the storage address of the original data object will be marked to ensure that the snapshot data and the original file data are isolated from each other; (2) Snapshot metadata recording: After the snapshot is generated, the system will record the snapshot metadata on the storage node (OSD, Object Storage). The snapshot's metadata information is recorded in the Daemon (object storage daemon process), including but not limited to: the snapshot's unique identifier (used to distinguish between the baseline snapshot and the destination snapshot), the snapshot's generation time, the path and object identifier of the associated target copy file, and the snapshot's clone_overlap information (recording the data part that was not written on the original data object after the snapshot clone data object in the form of "offset~len", providing a basis for obtaining the range of difference data in the future); (3) Snapshot purpose positioning: The file snapshot generated this time will serve as the basis for the subsequent remote replication process. Among them, the snapshot that has been copied to the cluster will be marked as the "baseline snapshot", and the latest snapshot to be copied will be marked as the "destination snapshot". Both will be used together for the subsequent "obtaining the range of difference data" operation.

[0052] Step S13: Obtain the range of difference data between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is the snapshot that has been copied, and the destination snapshot is the snapshot to be copied.

[0053] In this embodiment, the difference data range between the baseline snapshot and the target snapshot in the file snapshot is obtained. The specific process and technical details are as follows:

[0054] I. Conditions for obtaining the range of difference data: Before obtaining the range of difference data, two basic conditions must be met: (1) Snapshot identification is clear: The file snapshot corresponding to the target replication file has been generated through step S12, and a unique identifier (snapshot id, Identifier) ​​has been assigned to each snapshot, and the "base snapshot" and "destination snapshot" are clearly distinguished. The snapshot id of the base snapshot has been associated with the status flag "replication to the slave cluster has been completed" (maintained by the synchronization record module of the remote replication link), and the destination snapshot is the latest snapshot to be replicated generated in step S12, and its snapshot id is bound to the current status of the target replication file; (2) Data object association is completed: All data objects associated with the target replication file (such as obj1, obj2, etc.) have been determined through the metadata of the target replication file, and the storage address and object identifier of each data object in the master cluster OSD have been obtained to ensure that the OSD can accurately locate the data object to be queried.

[0055] II. Specific process for obtaining the range of difference data: 1. Initiate a difference snapshot query request: The client library (libclient) of the main cluster initiates a difference data range query request to the storage node (OSD), calls the newly added ICFS_OSD_OP_DIFF_SNAPS operation in OSD, and passes in three core input parameters: (1) Query object identifier: the unique identifier of each data object associated with the target copy file (such as the object ID of obj1), to ensure that OSD can locate the specific data object to be compared; (2) Base snapshot id: the unique identifier of the base snapshot that has been copied, used by OSD to locate the metadata of the data object in the base state; (3) Destination snapshot id: the unique identifier of the destination snapshot to be copied, used by OSD to locate the metadata of the data object in the latest state. 2. OSD parses clone_overlap metadata to determine the range of differences: After receiving a query request, the OSD parses the clone_overlap metadata of the data object to determine the range of differences between the baseline snapshot and the target snapshot. The clone_overlap metadata is key information recorded in the OSD for each snapshot that has undergone a clone operation. It records the data portion of the original data object that was not written to after the snapshot was cloned (i.e., the data range that has not changed since the snapshot was generated) in the form of "offset~len". The OSD reverse-engineers the range of data changes between the two snapshots by comparing the clone_overlap metadata corresponding to the baseline snapshot and the target snapshot. If a segment of data exists in the clone_overlap of the baseline snapshot but is missing in the clone_overlap of the target snapshot, or if the "offset~len" values ​​of the two clone_overlap records do not overlap, then that segment of data represents the range of differences between the two snapshots. 3. Generates and returns the range of differences: Based on the clone_overlap metadata parsing results, the OSD generates a vector containing the range of differences.<snap_diff_data_t> The structure contains a `snap_diff_data_t` element that corresponds to a segment of difference data, including: a `rejoin` field: using pairs...<uint64_t,uint64_t> The format stores the "offset~len" of the difference data, which is the starting offset and data length of the difference data in the data object, specifying the exact location of the difference data; the bufferlistdata field temporarily stores the basic data corresponding to the difference data range (the actual difference data segment will be extracted based on this in subsequent step S14); OSD uses this structure as an output parameter and returns it to the client library (libclient) of the main cluster to complete the acquisition of the difference data range.

[0056] III. Example of Obtaining the Difference Data Range: Taking the scenario of creating snap3 (destination snapshot) after copying to snap1 (base snapshot) as an example, when querying the difference range of the obj1 data object, the OSD parses the clone_overlap metadata corresponding to snap1 (base snapshot) and snap3 (destination snapshot), and finds that after snap1 was generated, only two data segments of obj1, "256~256" and "2M~256", changed. The OSD generates a snap_diff_data_t structure containing these two ranges, and the rejoin field records "256~256" and "2097408~256" respectively (Note: 2M=2097152 bits, 2M~256 is the starting offset 2097152, length 256), and returns the result to libclient. Subsequent steps can then extract the actual difference data segments based on these two difference ranges.

[0057] Step S14: Determine the complete data of each data object based on the object identifier associated with the target copied file, extract the difference data segment from the complete data based on the difference data range, summarize the difference data segments of all data objects, and migrate the corresponding summary results to the slave cluster.

[0058] First, the data objects are located and their complete data is obtained based on the object identifiers associated with the target copy file. The metadata of the target copy file records unique object identifiers for all associated data objects (e.g., the unique IDs corresponding to data objects obj1 and obj2). libclient needs to determine the complete data of each data object through the following two steps: First, iterate through the list of data objects: libclient reads the metadata of the target copy file, extracts the object identifiers of all associated data objects, and generates a list of data objects to be processed in order, ensuring no data objects are missed. Second, call a dedicated interface to obtain the complete data: For each data object in the list, libclient calls the newly added dedicated data reading interface read_diff_trunc to initiate a data reading request to the storage node. This interface requires two core parameters: "data object identifier" and "baseline snapshot ID corresponding to the target copy file". After receiving the request, the OSD accurately locates the physical storage address of the data object based on the data object identifier, then reads the complete data of the data object (the data range covers all content from the generation of the base snapshot to the generation of the target snapshot), and returns the complete data to libclient in bufferlist format, providing a data foundation for the extraction of the difference segment.

[0059] Secondly, after obtaining the complete data of each data object, the difference data range (in vector) obtained in step S13 is further combined.<snap_diff_data_t> The data is stored in a structure, where each element contains a `rejoin` field. This field records the position of the difference data in the form of "offset ~ len (offset ~ length)". libclient performs a precise truncation operation on the complete data of each data object, which involves three steps: First, matching the difference range with the complete data: For the complete data of a single data object, libclient matches the vector one by one.<snap_diff_data_t> The process involves three steps: First, defining the offset~len range corresponding to the data object. By comparing the offset with the data length, the specific location of the difference data to be extracted from the complete data is determined. Second, accurately extracting the difference data segment: Based on the matched offset~len range, the corresponding data fragment is extracted from the complete data to generate the difference data segment corresponding to the data object. Taking data object obj1 as an example, if its complete data length is 4M, and the difference data range obtained in step S13 is "256~256" and "2M~256", then two data segments are extracted from the complete data of obj1: "starting offset 256, length 256" and "starting offset 2097152 (i.e., 2M), length 256". These two data segments together constitute the difference data segment of obj1. Third, marking the attributes of the difference data segment: To ensure accurate location of the storage position after receiving from the cluster, libclient adds metadata tags to each difference data segment. The tag content includes "data object identifier", "difference range offset~len", and "identifier of the target copy file".

[0060] After extracting the difference data segments from all data objects, libclient iterates through all data objects associated with the target copy file, collects all extracted difference data segments, and summarizes them according to the hierarchical structure of target copy file, data object, and difference data segments, ultimately generating a summary result. To ensure that the slave cluster can correctly receive and store the summary result, the summary result also needs to include additional location information such as "the storage path of the target copy file in the master cluster" and "the corresponding slave directory path in the slave cluster" to avoid data storage misalignment. After the summary is completed, libclient will migrate the difference data summary package to the slave cluster through the established remote replication link (Pair) between the master and slave clusters. During the migration process, it will strictly adhere to the replication rate limit preset by the Pair and only transmit the summary result of the difference data (instead of the full file in the traditional solution), significantly reducing the amount of data transmission. At the same time, the Pair will record the migration progress in real time, including information such as "the amount of data migrated," "the amount of data remaining," and "the migration rate." This progress data will be synchronized to the remote replication management interface for subsequent migration status monitoring and anomaly troubleshooting.

[0061] Furthermore, to achieve visualized management of the strategy, the remote replication management interface (management software interface) displays: 1. Remote replication strategies associated with remote replication links. The interface presents basic information for all remote replication links in a list format, including the Pair name, local home directory path, remote slave directory path, number of associated remote replication strategies (e.g., "Associated strategies: 3"), and replication status (e.g., "Synchronizing" or "Completed"). If a Pair is associated with a filtering strategy (i.e., a remote replication strategy), the list will be marked with "Filtering strategy configured" for easy user identification. Data source: Interface data is obtained by calling the pairs interface, which queries the configuration parameters of all Pairs, including information such as "whether a filtering strategy is associated" and "number of associated strategies." After the interface returns the data, the management software interface parses and renders the list. 2. Detailed display interface for a single remote replication link. Clicking on a Pair in the list enters the details interface, which is divided into "Basic Information" and "Filtering Strategy Details." Basic Information: Includes the pair's replication rate, recovery strategy (manual / automatic), and master-slave cluster connection information (peer address); Filtering Policy Details: Displays all remote replication policies associated with this pair in card format (maximum 16 policies per pair). Each policy card includes: policy name (required), replication directory specification information (manually set directory or default root directory), file attribute filter conditions (filename), file size limit (e.g., "greater than or equal to 1024 bits"), file operation time (e.g., "mtime less than 7 days"), user / user group identifier (e.g., "uid: 1001", "gid: 2001"), and policy matching logic (marked "all conditions for the same policy must be met simultaneously; for multiple policies, only one condition needs to be met"). The details interface also provides "Modify Policy" and "Delete Policy" buttons (subject to the constraint "pause the pair before modifying / deleting"). Clicking these buttons will redirect to the policy configuration interface.

[0062] Furthermore, when a failure occurs during the migration of the difference data aggregation package to the slave cluster (such as internal I / O failure during remote replication, link disconnection, or missing target directory in the slave cluster), the system automatically executes the following retry logic:

[0063] I. Methods for determining migration failure: (1) Link status monitoring: The remote replication link (Pair) monitors the connection with the slave cluster in real time. If the link is disconnected (e.g., network interruption) or the slave cluster node is offline, it is immediately marked as "migration failure"; (2) Data verification failure: After receiving the summary packet, the slave cluster verifies the integrity (e.g., checksum comparison). If the data is lost / damaged, it returns a "verification failure" response to the master cluster, and the master cluster marks "migration failure". II. New baseline snapshot retry process: (1) Determine the new baseline snapshot: After the migration fails, the system queries the "head data (head data, the latest synchronized state)" of the target file copied from the cluster, marks it as the "new baseline snapshot" and records the snapshot ID; (2) Re-acquire the difference range: Based on the new baseline snapshot ID and the destination snapshot ID of step S12, step S13 is triggered again: the client library calls the ICFS_OSD_OP_DIFF_SNAPS operation to the OSD, passing in the data object identifier and the new baseline / destination snapshot ID, and the OSD parses the clone_overlap metadata to obtain the new difference range; (3) Reprocess the migration: Reprocess according to the process of "complete data determination of data object, extraction of difference segment, and summarization", and then migrate to the slave cluster; if it still fails after 3 retries, a pop-up window will prompt "migration failed, please check the link or slave cluster directory configuration", and the failure log (including time and fault type) will be recorded. 3. Special Scenario Adaptation: If migration fails due to policy modification (e.g., the original policy restricts dir1 from being copied, and there is no dir1 on the slave end; synchronization of dir1 / a fails after deleting the policy), the slave cluster directory will be checked first before retrying: if the target directory (such as dir1) is missing, it will be automatically created according to the main cluster directory structure, and then the process of determining the new baseline, obtaining differences again, and migration will be executed to ensure successful retry.

[0064] Furthermore, the filtering strategy must adhere to the following constraints during configuration and execution: (1) Master-slave data consistency constraint: The filtering strategy achieves on-demand synchronization by filtering target replication files, which will result in incomplete data consistency between the master cluster (source end) and the slave cluster (destination end) (the slave end only stores the filtered target files). Therefore, the remote replication link containing the filtering strategy only supports backup scenarios and does not support master-slave switching (if a switch is attempted, the system returns error code -335 indicating "Master-slave switching is prohibited for Pairs containing the filtering strategy"); (2) Operation scope constraint: The filtering strategy only applies to file addition and modification operations, and does not filter file or directory deletion operations (i.e., if the file has been synchronized to the slave cluster after the master cluster deletes a file, the slave cluster will still perform the deletion operation), to prevent the exception that duplicate files cannot be processed due to differences between master and slave files; (3) Replication mode constraint: Pairs configured with the filtering strategy will be forced to switch to the polling comparison mode for data replication, instead of the default snapdiff mode (snap difference mode), to avoid the snapdiff mode being unable to handle (4) Policy modification constraints: Before modifying the filtering policy, you must first execute the "pausePair" operation through the remote replication management interface or command line (pause the remote replication link). If you modify it directly without pausing, the system will return an error message "you need to pause the Pair before modifying the filtering policy". Moreover, the modification of the filtering policy will temporarily occupy replication resources and affect the synchronization speed. Therefore, it is not recommended to modify it frequently (it is recommended that the modification interval be no less than 1 hour to avoid wasting resources). (5) Policy quantity constraints: A single Pair can support a maximum of 16 filtering policies. If the limit is exceeded, the system will return error code -332 to indicate "the number of filtering policies in a single Pair exceeds the limit (maximum 16)". Redundant policies need to be deleted before adding new ones. (6) Policy deletion and switching constraints: If all filtering policies in a Pair have been deleted, a complete synchronization task must be completed first (to ensure master-slave data consistency) before the master-slave switching operation can be performed. If you switch directly without completing the synchronization, the system will return error code -335 to indicate "you need to complete a synchronization before allowing master-slave switching".

[0065] It should be noted that when modifying the policy by selecting the target pair through the remote replication management interface, only the policy name cannot be modified. All other configuration items (replication directory, file name filtering, file size threshold, operation time range, user / user group identifier) ​​can be adjusted. After modification, the "resumePair" operation (to restore the remote replication link) must be executed again. The system will filter subsequently added / modified files based on the new policy, and historical files already synchronized to the slave cluster will not be affected.

[0066] It should be noted that in the lib library operations, after obtaining the difference data segments of each data object, in addition to summarizing the difference data segments, a difference data map is also generated. Its core function is to record the correspondence between "data object identifier - difference data segment - storage path from the cluster", as follows: The structure of the difference data map is stored in key-value pairs. The key is the data object identifier (such as "obj1"), and the value is a structure containing "offset~len of the difference data segment", "storage address of the data object in the cluster", and "path of the target copy file". Its core function is that after receiving the difference data summary package from the cluster, the difference data map can quickly locate the storage location corresponding to each difference data segment without traversing the entire directory, thus improving the efficiency of data reassembly. At the same time, if a difference data segment is lost during the migration process, the map can be used to locate the missing object identifier, and only the difference data segment of that object can be retransmitted, reducing the cost of retrying.

[0067] Furthermore, to address the differences in bandwidth usage of the primary cluster during different business periods and to prevent remote replication traffic from crowding out core business bandwidth during peak business periods, this application can also add a dynamic rate adjustment function to the remote replication link. The replication rate can be automatically adjusted based on the real-time business load of the primary cluster (such as CPU utilization), reducing the rate during peak periods and increasing the rate during off-peak periods, thus balancing business stability and replication efficiency.

[0068] As can be seen, this application proposes a distributed file remote replication method, including: determining a remote replication strategy that associates the remote replication links between the master cluster and the slave cluster; wherein, the remote replication strategy includes replication directory specification information and file attribute filtering conditions; determining the target replication file from the files stored in the master cluster according to the replication directory specification information and file attribute filtering conditions, and generating a file snapshot corresponding to the target replication file; obtaining the difference data range between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is a snapshot that has been replicated, and the destination snapshot is a snapshot to be replicated; determining the complete data of each data object according to the object identifier associated with the target replication file, extracting difference data segments from the complete data based on the difference data range, summarizing the difference data segments of all data objects, and migrating the corresponding summary results to the slave cluster. In summary, this application accurately determines the target file to be copied in the master cluster by specifying the copy directory information and filtering file attributes in the remote copy strategy. This avoids unnecessary files from occupying the slave cluster's storage resources, reduces the invalid storage overhead of the slave cluster, and improves the efficiency of storage resource utilization. Furthermore, this application obtains the range of difference data based on file snapshots, extracts the difference data segments from the complete data based on the difference data range, summarizes the difference data segments of all data objects, and migrates the corresponding summary results to the slave cluster, instead of copying the entire file. This significantly reduces the amount of data transmission during the remote copy process, especially for scenarios with large files and minor changes, which can significantly reduce data processing and transmission time and improve copy efficiency.

[0069] Accordingly, this application also discloses a distributed file remote copying device, see [link to relevant documentation]. Figure 2 As shown, the device includes:

[0070] The strategy determination module 11 is used to determine the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster; wherein, the remote replication strategy includes the replication directory specification information and file attribute filtering conditions;

[0071] The snapshot determination module 12 is used to determine the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions, and to generate a file snapshot corresponding to the target copy file;

[0072] The difference data determination module 13 is used to obtain the difference data range between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is the snapshot that has been copied, and the destination snapshot is the snapshot to be copied.

[0073] The differential data migration module 14 is used to determine the complete data of each data object based on the object identifier associated with the target copy file, extract differential data segments from the complete data based on the differential data range, summarize the differential data segments of all data objects, and migrate the corresponding summary results to the slave cluster.

[0074] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0075] Furthermore, embodiments of this application also provide an electronic device. Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0076] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the distributed file remote copying method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0077] In this embodiment, the power supply 26 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 24 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0078] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon may include computer programs 221, and the storage method may be temporary storage or permanent storage. The computer programs 221 may include, in addition to computer programs capable of performing the distributed file remote copying method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, computer programs capable of performing other specific tasks.

[0079] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed distributed remote file copying method.

[0080] For the specific steps of this method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0081] The various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For the same or similar parts between the various embodiments, refer to each other. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to in the method section.

[0082] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0084] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0085] The above provides a detailed description of a distributed file remote copying method, apparatus, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A distributed file remote copying method, characterized in that, include: A remote replication strategy is used to determine the remote replication link association between the master cluster and the slave cluster; wherein, the remote replication strategy includes information on the replication directory and file attribute filtering conditions; Based on the specified information of the replication directory and the file attribute filtering conditions, the target replication file is determined from the files stored in the main cluster, and a file snapshot corresponding to the target replication file is generated; Obtain the range of difference data between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is a snapshot that has been copied, and the destination snapshot is a snapshot to be copied; The complete data of each data object is determined based on the object identifier associated with the target copy file. Based on the difference data range, difference data segments are extracted from the complete data. The difference data segments of all the data objects are summarized, and the corresponding summary results are migrated to the slave cluster. The process of summarizing the difference data segments of all the data objects and migrating the corresponding summary results to the slave cluster also includes: If the migration fails, the header data corresponding to the target file copied from the cluster is used as a new baseline snapshot, and the step of obtaining the difference data range between the baseline snapshot and the target snapshot in the file snapshot is re-executed based on the new baseline snapshot and the target snapshot.

2. The distributed file remote copying method according to claim 1, characterized in that, The step of determining the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions includes: If the remote replication link is associated with only one remote replication policy, then when any file stored in the master cluster is located in the directory corresponding to the replication directory specified in the remote replication policy, and the file attributes of any file meet the file attribute filtering conditions in the remote replication policy, the file will be identified as the target replication file.

3. The distributed file remote copying method according to claim 1, characterized in that, The step of determining the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions includes: If the remote replication link is associated with multiple remote replication policies, then when any file stored in the master cluster is located in the directory corresponding to the replication directory specified in any of the remote replication policies, and the file attributes of any file meet the file attribute filtering conditions in any of the remote replication policies, the file will be identified as the target replication file.

4. The distributed file remote copying method according to claim 1, characterized in that, The specified information for the copied directory includes the directory name, and the file attribute filtering conditions include one or more of the following: file name filtering conditions, file size filtering conditions, file operation time filtering conditions, user identifier filtering conditions, or user group identifier filtering conditions.

5. The distributed file remote copying method according to claim 4, characterized in that, The remote replication strategy for determining the remote replication link association between the master cluster and the slave cluster includes: Set the directory name through the remote copy management interface, or use the directory name corresponding to the default directory; One or more target filenames can be set through the remote copy management interface, so as to determine the filename filtering conditions in the file attribute filtering conditions based on the one or more target filenames; By selecting the file size comparison type through the remote copy management interface and setting the corresponding file size threshold, the file size filtering condition in the file attribute filtering conditions can be determined. By selecting the file operation time type through the remote copy management interface and setting the corresponding time size threshold, the file operation time filtering condition in the file attribute filtering conditions can be determined. The target user or target user group is set through the remote copy management interface to determine the user identifier filter condition or the user group identifier filter condition in the file attribute filter conditions. The directory name, file name filter, file size filter, file operation time filter, user identifier filter, or user group identifier filter are integrated to obtain the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster.

6. The distributed file remote copying method according to claim 1, characterized in that, Also includes: The remote replication management interface displays the remote replication strategies associated with the remote replication links between the master cluster and the slave cluster; the corresponding display interface includes a list of remote replication links and a details display interface for each remote replication link.

7. A distributed file remote copying device, characterized in that, include: The strategy determination module is used to determine the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster; wherein, the remote replication strategy includes replication directory specification information and file attribute filtering conditions; The snapshot determination module is used to determine the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions, and generate a file snapshot corresponding to the target copy file; The difference data determination module is used to obtain the difference data range between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is a snapshot that has been copied, and the destination snapshot is a snapshot to be copied; The differential data migration module is used to determine the complete data of each data object based on the object identifier associated with the target copied file, extract differential data segments from the complete data based on the differential data range, summarize the differential data segments of all the data objects, and migrate the corresponding summary results to the slave cluster. The process of summarizing the difference data segments of all the data objects and migrating the corresponding summary results to the slave cluster also includes: If the migration fails, the header data corresponding to the target file copied from the cluster is used as a new baseline snapshot, and the step of obtaining the difference data range between the baseline snapshot and the target snapshot in the file snapshot is re-executed based on the new baseline snapshot and the target snapshot.

8. An electronic device, characterized in that, include: Memory is used to store computer programs; A processor for executing the computer program to implement the distributed file remote copying method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the distributed file remote copying method as described in any one of claims 1 to 6.

Images