Distributed file remote copying method and device, equipment and medium

Abstract

The invention discloses a distributed file remote copying method and device, equipment and a medium, and is applied to the field of computers. Determining a target copy file from the files stored in the main cluster according to copy directory specified information and file attribute screening conditions, and generating a file snapshot corresponding to the target copy file; obtaining a difference data range between a reference snapshot in the file snapshots and a target snapshot in the file snapshots; the reference snapshot is a copied snapshot, and the target snapshot is a to-be-copied snapshot; and determining complete data of each data object according to the object identifier associated with the target copy file, intercepting a difference data segment from the complete data based on the difference data range, summarizing the difference data segments of all the data objects, and migrating a corresponding summarizing result to the slave cluster. According to the method, the data transmission quantity in the remote copying process is reduced, the data processing and transmission time consumption can be remarkably reduced especially for a small-change scene of a large file, and the copying efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a distributed file remote replication method, device, equipment and medium. BACKGROUND

[0002] Distributed storage clusters have been applied more and more widely. With the continuous advancement of digitalization process, data has gradually become the core of the operation of enterprises and institutions, and users have increasingly high requirements for the stability of storage systems carrying data. In order to ensure the continuity, recoverability and high availability of data storage, remote disaster recovery solutions have emerged, and remote replication technology is one of the key technologies of the solutions: the core idea is to synchronize and replicate data to different places to avoid data loss caused by natural disasters or human destruction to the greatest extent.

[0003] In the current technical implementation, for files with data changes, full replication is required. Especially for large files, even if there is only a small amount of data change, it will significantly affect the replication efficiency and performance. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a distributed file remote replication method, device, equipment and medium, which can reduce the data transmission amount in the remote replication process, especially for the scenario of small changes in large files, can significantly reduce the data processing and transmission time, improve the replication efficiency, and the specific scheme is as follows:

[0005] In a first aspect, the present application discloses a distributed file remote replication method, comprising:

[0006] determining a remote replication strategy associated with a remote replication link between a master cluster and a slave cluster; wherein the remote replication strategy includes replication directory specification information and file attribute filtering conditions;

[0007] determining target replication files from the files stored in the master cluster according to the replication directory specification information and the file attribute filtering conditions, and generating file snapshots corresponding to the target replication files;

[0008] obtaining a difference data range between a reference snapshot in the file snapshots and a target snapshot in the file snapshots; the reference snapshot is a snapshot that has completed replication, and the target snapshot is a snapshot to be replicated;

[0009] determining complete data of each data object according to an object identifier associated with the target replication file, extracting difference data segments from the complete data based on the difference data range, aggregating the difference data segments of all data objects, and migrating the corresponding aggregation results to the slave cluster.

[0010] Optionally, the target replication file is determined from the files stored in the primary cluster according to the replication directory designation information and the file attribute filtering condition, comprising:

[0011] If the remote replication link is associated with one remote replication policy, when any file stored in the primary cluster is located in the directory corresponding to the replication directory designation information in the remote replication policy, and the file attribute of any file meets the file attribute filtering condition in the remote replication policy, the any file is determined as the target replication file.

[0012] Optionally, the target replication file is determined from the files stored in the primary cluster according to the replication directory designation information and the file attribute filtering condition, comprising:

[0013] If the remote replication link is associated with multiple remote replication policies, when any file stored in the primary cluster is located in the directory corresponding to the replication directory designation information in any remote replication policy, and the file attribute of any file meets the file attribute filtering condition in any remote replication policy, the any file is determined as the target replication file.

[0014] Optionally, the replication directory designation information comprises a directory name, and the file attribute filtering condition comprises one or more of a file name filtering condition, a file size filtering condition, a file operation time filtering condition, a user identifier filtering condition, or a user group identifier filtering condition.

[0015] Optionally, the remote replication policy associated with the remote replication link between the primary cluster and the secondary cluster is determined, comprising:

[0016] The directory name is set through the remote replication management interface, or the directory name corresponding to a default directory is used;

[0017] One or more target file names are set through the remote replication management interface, so as to determine the file name filtering condition in the file attribute filtering condition according to the one or more target file names;

[0018] A file size comparison type is selected through the remote replication management interface, and a corresponding file size threshold is set, so as to determine the file size filtering condition in the file attribute filtering condition;

[0019] A file operation time type is selected through the remote replication management interface, and a corresponding time size threshold is set, so as to determine the file operation time filtering condition in the file attribute filtering condition;

[0020] A target user or a target user group is set through the remote replication management interface, so as to determine the user identifier filtering condition or the user group identifier filtering condition in the file attribute filtering condition;

[0021] The directory name, the file name filtering condition, the file size filtering condition, the file operation time filtering condition, the user identifier filtering condition, or the user group identifier filtering condition are integrated to obtain a remote replication strategy associated with a remote replication link between the master cluster and the slave cluster.

[0022] Optionally, the distributed file remote replication method further includes:

[0023] The remote replication management interface is used to display the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster, and the corresponding display interface range includes a remote replication link list display interface and a single remote replication link detail display interface.

[0024] Optionally, in the process of aggregating the difference data segments of all data objects and migrating the corresponding aggregation result to the slave cluster, the method further includes:

[0025] If the migration fails, the head data corresponding to the target replication file in the slave cluster is taken as a new reference snapshot, and the step of obtaining the difference data range between the reference snapshot and the target snapshot in the file snapshot is re-executed based on the new reference snapshot and the target snapshot.

[0026] In a second aspect, the present application discloses a distributed file remote replication device, which includes:

[0027] The policy determination module is configured to determine a remote replication strategy associated with a remote replication link between the master cluster and the slave cluster, and the remote replication strategy includes replication directory specification information and file attribute filtering conditions.

[0028] The snapshot determination module is configured to determine a target replication file from the files stored in the master cluster according to the replication directory specification information and the file attribute filtering conditions, and generate a file snapshot corresponding to the target replication file.

[0029] The difference data determination module is configured to obtain a difference data range between a reference snapshot and a target snapshot in the file snapshot, the reference snapshot is a completed snapshot, and the target snapshot is a to-be-replicated snapshot.

[0030] The difference data migration module is configured to determine complete data of each data object according to an object identifier associated with the target replication file, extract difference data segments from the complete data based on the difference data range, aggregate the difference data segments of all data objects, and migrate the corresponding aggregation result to the slave cluster.

[0031] In a third aspect, the present application discloses an electronic device, which includes:

[0032] The memory is configured to save a computer program.

[0033] A processor is configured to execute a computer program to implement the distributed file remote replication method disclosed above.

[0034] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the distributed file remote replication method disclosed above.

[0035] It can be seen that the present application proposes a distributed file remote replication method, which comprises: determining a remote replication strategy associated with a remote replication link between a master cluster and a slave cluster; wherein the remote replication strategy comprises replication directory specification information and file attribute filtering conditions; determining target replication files from the files stored in the master cluster according to the replication directory specification information and the file attribute filtering conditions, and generating file snapshots corresponding to the target replication files; obtaining a difference data range between a reference snapshot in the file snapshots and a destination snapshot in the file snapshots; the reference snapshot is a completed replication snapshot, and the destination snapshot is a to-be-replicated snapshot; determining complete data of each data object according to an object identifier associated with the target replication file, intercepting a difference data segment from the complete data based on the difference data range, summarizing the difference data segments of all data objects, and migrating the corresponding summary results to the slave cluster. It can be seen from the above that, by using the replication directory specification information and the file attribute filtering conditions in the remote replication strategy, the target replication files in the master cluster can be accurately determined, so that the slave cluster storage resources can be avoided from being occupied by files that do not need to be synchronized, the invalid storage overhead of the slave cluster can be reduced, and the storage resource utilization efficiency can be improved; further, the difference data range is obtained based on the file snapshots, the difference data segment is intercepted from the complete data based on the difference data range, then the difference data segments of all data objects are summarized, and the corresponding summary results are migrated to the slave cluster, instead of replicating the entire file, so that the data transmission amount in the remote replication process can be greatly reduced, especially for the scenario of a large file with a small amount of changes, the data processing and transmission time can be significantly reduced, and the replication efficiency can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0037] Figure 1 A distributed file remote replication method disclosed by the present application is shown in a flowchart;

[0038] Figure 2 A distributed file remote replication device structure disclosed by the present application is shown in a schematic diagram;

[0039] Figure 3 An electronic device structure diagram is disclosed in the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0041] In the current technical implementation, for files with data changes, full-amount replication needs to be performed. Especially for large files, even if there is only a small amount of data change, the replication efficiency and performance will be significantly affected.

[0042] Therefore, the embodiments of the present application propose a distributed file remote replication scheme, which reduces the data transmission amount in the remote replication process, and especially for the scenario of small changes in large files, can significantly reduce the data processing and transmission time consumption, and improve the replication efficiency.

[0043] The embodiments of the present application disclose a distributed file remote replication method, as shown in Figure 1 The method comprises the following steps.

[0044] Step S11: determining a remote replication strategy associated with a remote replication link between a master cluster and a slave cluster; wherein the remote replication strategy comprises replication directory designation information and file attribute filtering conditions.

[0045] In the embodiments, the replication directory designation information comprises a directory name, and the file attribute filtering conditions comprise one or more of a file name filtering condition, a file size filtering condition, a file operation time filtering condition, a user identifier filtering condition, or a user group identifier filtering condition.

[0046] In the embodiment, the directory name is set through the remote copy management interface, or a default directory corresponding directory name is used; one or more target file names are set through the remote copy management interface, so as to determine the file name screening condition in the file attribute screening condition according to the one or more target file names; the file size comparison type is selected through the remote copy management interface, and the corresponding file size threshold is set, so as to determine the file size screening condition in the file attribute screening condition; the file operation time type is selected through the remote copy management interface, and the corresponding time size threshold is set, so as to determine the file operation time screening condition in the file attribute screening condition; the target user or the target user group is set through the remote copy management interface, so as to determine the user identifier screening condition or the user group identifier screening condition in the file attribute screening condition; and the directory name, the file name screening condition, the file size screening condition, the file operation time screening condition, the user identifier screening condition or the user group identifier screening condition are integrated, so as to obtain the remote copy strategy associated with the remote copy link between the master cluster and the slave cluster.

[0047] Exemplarily, 1. setting a unique policy name for the to-be-configured remote replication policy through the remote replication management interface (mandatory configuration, and the "policy name" is marked as a mandatory item in the interface); 2. setting a directory name through the remote replication management interface, or using a default directory corresponding to the directory name (the default directory is the root directory of the replication directory corresponding to the remote replication link, and only one directory entry can be configured for one remote replication policy); 3. setting one or more target file names (such as "data_" which can match all files starting with "data_") through the remote replication management interface to determine the file name filtering condition in the file attribute filtering condition according to the one or more target file names; 4. selecting a file size comparison type (including "greater than or equal to" and "less than" two types) through the remote replication management interface and setting a corresponding file size threshold (unit: bit) to determine the file size filtering condition in the file attribute filtering condition; 5. selecting a file operation time type through the remote replication management interface, including btime (creation time) and mtime (modification time) two types, and selecting a corresponding time comparison type ("greater than or equal to" or "less than") and setting a corresponding time size threshold (unit: day) to determine the file operation time filtering condition in the file attribute filtering condition; 6. setting a target user or a target user group through the remote replication management interface, and setting the "local" type when setting, and matching through the user identifier (uid, User Identifier) of the target user or the user group identifier (gid, Group Identifier) of the target user group to determine the user identifier filtering condition or the user group identifier filtering condition in the file attribute filtering condition; 7. integrating the directory name, the file name filtering condition, the file size filtering condition, the file operation time filtering condition, the user identifier filtering condition or the user group identifier filtering condition to obtain one remote replication policy associated with the remote replication link between the master cluster and the slave cluster; if multiple remote replication policies need to be configured for the remote replication link, repeat steps 1-6 above, and the total number of remote replication policies associated with a single remote replication link is not more than 16, otherwise the system returns error code -332 to prompt that the configuration is out of limit.

[0048] Step S12: determining the target replication file from the files stored in the master cluster according to the replication directory specification information and the file attribute filtering condition, and generating a file snapshot corresponding to the target replication file.

[0049] In this embodiment, if the remote replication link is associated with only one remote replication policy, any file stored in the primary cluster is determined as a target replication file when the file is located in a directory corresponding to the replication directory specified information in the remote replication policy and the file attribute of the file meets the file attribute filtering condition in the remote replication policy. For example, when the remote replication link is bound with only one remote replication policy, the "directory ownership check + full attribute matching check" needs to be performed on any file stored in the primary cluster, and the file is determined as a target replication file when both checks are met. The directory ownership check: it is judged whether the storage path of the file is completely located in the directory corresponding to the "replication directory specified information" in the remote replication policy (if the directory is not manually set in the policy, the root directory of the replication directory corresponding to the remote replication link is matched by default, and only it is checked whether the file is directly or indirectly stored under the root directory); the full attribute matching check: the attributes of the file are matched with all the file attribute filtering conditions configured in the remote replication policy one by one. If the policy is configured with "file name filtering condition", the file name needs to completely match one or more target file names set (such as "report_" matching all files starting with "report_"); if the policy is configured with "file size filtering condition", the actual size of the file needs to meet the preset file size comparison type ("greater than or equal to" or "less than") and the corresponding size threshold (unit: bit); if the policy is configured with "file operation time filtering condition", the operation time (btime creation time or mtime modification time) of the file needs to meet the preset time comparison type ("greater than or equal to" or "less than") and the corresponding time threshold (unit: day); if the policy is configured with "user identifier filtering condition" or "user group identifier filtering condition", the uid (user identifier) of the user to which the file belongs or the gid (user group identifier) of the user group to which the file belongs needs to be completely consistent with the target uid / gid set in the policy (only local user / group matching is supported, cross-cluster identifier is not supported); only when the "directory ownership check" is passed and "all file attribute filtering conditions configured in the policy are matched successfully", the file is determined as a target replication file.

[0050] In this embodiment, if the remote replication link is associated with multiple remote replication policies, when any file stored in the primary cluster is located in the directory corresponding to the replication directory specified information in any remote replication policy, and the file attribute of any file meets the file attribute filtering condition in any remote replication policy, the any file is determined as a target replication file. For example, when the remote replication link is bound to two or more remote replication policies, the "directory ownership verification + single policy attribute matching verification" is performed on any file stored in the primary cluster, and as long as the double conditions of any policy are met, the file is determined as a target replication file. Single policy double condition judgment: for each remote replication policy, whether the file meets the policy is determined according to "directory ownership verification (the rule is the same as the above directory verification) + full attribute matching verification (the rule is the same as the above attribute verification)"; multi-policy "or logic" matching: if the file meets the "directory ownership + full attribute matching" double conditions of any policy (i.e., at least one policy determines that the file is a matching file), there is no need to verify other policies, and the file is directly determined as a target replication file (the file meets the directory and attribute conditions of policy 1, even if it does not meet policies 2 and 3, it is still determined as a target replication file).

[0051] In this embodiment, after the selection of all target replication files is completed, the system will automatically generate a corresponding file snapshot for each target replication file, and the generation process and characteristics are as follows: (1) snapshot generation mechanism: the snapshot technology based on COW (Copy On Write) is adopted, before generating the snapshot, all data objects associated with the target replication file are preprocessed, if the data object may be modified subsequently, the data object in the current state is copied to generate an independent snapshot object, and then the storage address of the original data object is marked to ensure that the snapshot data and the original file data are isolated from each other; (2) snapshot metadata record: after the snapshot is generated, the system records the metadata information corresponding to the snapshot in the storage node (OSD, Object Storage Daemon, object storage daemon), including but not limited to: snapshot unique identifier (used to distinguish the baseline snapshot and the destination snapshot subsequently), snapshot generation time, path and object identifier of the associated target replication file, and clone_overlap information of the snapshot (in the form of "offset~len", the part of the original data object that has not been written after the snapshot clone data object is recorded, which provides a basis for subsequent acquisition of the difference data range); (3) snapshot use positioning: the file snapshot generated this time will be used as the basis for the subsequent remote replication process, wherein the snapshot that has completed replication to the slave cluster will be marked as "baseline snapshot", and the latest snapshot to be replicated will be marked as "destination snapshot", and the two are used together for the subsequent "acquisition of difference data range" operation.

[0052] Step S13: Obtain the difference data range between the reference snapshot in the file snapshot and the target snapshot in the file snapshot; the reference snapshot is the snapshot that has completed replication, and the target snapshot is the snapshot to be replicated.

[0053] In this embodiment, the difference data range between the reference snapshot in the file snapshot and the target snapshot in the file snapshot is obtained, and the specific process and technical details are as follows:

[0054] I. Conditions for obtaining the difference data range: Before performing the difference data range obtaining, two basic conditions need to be ensured: (1) Clear snapshot identification: The file snapshot corresponding to the target replication file has been generated through step S12, and a unique identifier (snapshot id) is assigned to each snapshot, and the "reference snapshot" and the "target snapshot" are clearly distinguished, wherein the snapshot id of the reference snapshot is associated with the state mark (maintained by the synchronization record module of the remote replication link) of "completed replication to the slave cluster", and the target snapshot is the latest generated snapshot to be replicated in step S12, and the snapshot id is bound to the current state of the target replication file; (2) Data object association is completed: The metadata of the target replication file is used to determine all the data objects (such as obj1, obj2, etc.) associated therewith, and the storage addresses and object identifiers of the data objects in the OSD of the master cluster are obtained, so as to ensure that the OSD can accurately locate the data object to be queried.

[0055] II. Specific acquisition process of difference data range: 1. Initiate difference snapshot query request: the client library (libclient) of the primary cluster initiates a difference data range query request to the storage node (OSD), calls the newly added ICFS OSD OP DIFF SNAPS operation in the OSD, and inputs three core parameters: (1) query object identifier: the unique identifier of each data object associated with the target replication file (such as the object ID of obj1), which ensures that the OSD can locate the specific data object to be compared; (2) reference snapshot id: the unique identifier of the reference snapshot that has completed replication, which is used by the OSD to locate the data object metadata in the reference state; (3) destination snapshot id: the unique identifier of the destination snapshot to be replicated, which is used by the OSD to locate the data object metadata in the latest state. 2. OSD parses clone overlap metadata to determine the difference range: after receiving the query request, the OSD determines the difference data range between the reference snapshot and the destination snapshot by parsing the clone overlap metadata of the data object: the clone overlap metadata is key information recorded by the OSD for each snapshot that has performed a clone operation. In the form of "offset~len", it records the data part of the original data object that has not been written after the clone generates the data object (i.e., the data range that has not been changed after the snapshot is generated). The OSD reversely deduces the data change range between the two snapshots by comparing the clone overlap metadata corresponding to the reference snapshot and the destination snapshot: if a certain segment of data exists in the clone overlap of the reference snapshot but is missing in the clone overlap of the destination snapshot, or the "offset~len" recorded by the two clone overlap records does not coincide, then this segment of data is the difference data range between the two snapshots. 3. Generate difference data range result and return: the OSD generates a vector<snap_diff_data_t> structure containing the difference data range based on the clone overlap metadata analysis result, where each snap_diff_data_t element corresponds to a segment of difference data and contains: rejoin field: stores the "offset~len" of the difference data in the form of pair<uint64_t, uint64_t>, which is the starting offset and data length of the difference data in the data object, clearly indicating the specific location of the difference data; bufferlist data field: temporarily stores the basic data corresponding to this difference data range (the actual difference data segment will be intercepted based on this in step S14); the OSD returns this structure as an output parameter to the client library (libclient) of the primary cluster, completing the acquisition of the difference data range.

[0056] III. Example of difference data range acquisition: Take the scenario of creating snap3 (destination snapshot) after snap1 (baseline snapshot) has been replicated as an example. When querying the difference range of obj1 data object, OSD parses the clone_overlap metadata corresponding to snap1 (baseline snapshot) and snap3 (destination snapshot) and finds that obj1 has only two changed data segments, "256~256" and "2M~256", after the generation of snap1. OSD generates a snap_diff_data_t structure containing these two ranges, records "256~256" and "2097408~256" (note: 2M = 2097152 bits, 2M~256 means the starting offset is 2097152 and the length is 256) in the rejoin field, and returns the result to libclient. The subsequent steps can be based on these two difference ranges to extract the actual difference data segment.

[0057] Step S14: Determine the complete data of each data object according to the object identifier associated with the target replication file, extract the difference data segment from the complete data based on the difference data range, aggregate the difference data segments of all data objects, and migrate the corresponding aggregation result to the slave cluster.

[0058] First, locate the data objects and obtain the complete data based on the object identifier associated with the target replication file. In the metadata of the target replication file, the unique object identifiers of all associated data objects (such as the exclusive IDs corresponding to data objects obj1 and obj2) have been recorded. libclient needs to determine the complete data of each data object through the following two steps: one is to traverse the data object list: libclient first reads the metadata of the target replication file, extracts the object identifiers of all associated data objects, and generates a list of data objects to be processed in order to ensure that no data object is missed; two is to call a special interface to obtain complete data: for each data object in the list, libclient will call the newly added read_diff_trunc special data reading interface to initiate a data reading request to the storage node. This interface needs to pass in two core parameters, namely "data object identifier" and "baseline snapshot id corresponding to the target replication file"; OSD receives the request and accurately locates the physical storage address of the data object based on the data object identifier. Then it reads the complete data of the data object (the data range covers all contents from the generation of the baseline snapshot to the generation of the destination snapshot), and returns the complete data to libclient in the form of bufferlist to provide data basis for difference segment extraction.

[0059] Secondly, after obtaining the complete data of each data object, further combine the difference data range (stored in the vector<snap_diff_data_t> structure, each element of the structure contains the rejoin field, which records the difference data position in the form of "offset~len") obtained in step S13, and perform accurate cutting operation on the complete data of each data object by libclient. Specifically, it includes three steps: first, match the difference range and the complete data: for the complete data of a single data object, libclient will match the offset~len range of the data object in vector<snap_diff_data_t> one by one, and through comparing the offset and the data length, it can be clear that the specific position of the difference data to be cut from the complete data; second, accurately cut the difference data segment: according to the matched offset~len range, cut the corresponding data segment from the complete data to generate the difference data segment corresponding to the data object. Taking data object obj1 as an example, if the length of its complete data is 4M, and the difference data range obtained in step S13 is "256~256" and "2M~256", then two pieces of data with "start offset 256, length 256" and "start offset 2097152 (i.e. 2M), length 256" are cut from the complete data of obj1 respectively, and these two pieces of data together constitute the difference data segment of obj1; third, mark the attribute of the difference data segment: in order to ensure accurate positioning of the storage location after receiving from the cluster, libclient will add metadata tags to each difference data segment, and the tag content includes "data object identifier" "difference range offset~len" "target replication file identifier".

[0060] After completing the difference data segment cutting of all data objects, libclient will traverse all data objects associated with the target replication file, collect all cut difference data segments, and summarize them according to the hierarchical structure of target replication file, data object, and difference data segment, and finally generate the summary result. In order to ensure correct receiving and storage from the cluster, the summary result also needs to contain additional positioning information such as "storage path of target replication file in master cluster" and "slave directory path corresponding to slave cluster" to avoid data storage misplacement. After summarizing, libclient will migrate the difference data summary package to the slave cluster through the remote replication link (Pair) established between the master cluster and the slave cluster. During the migration process, it will strictly follow the replication rate limit preset by Pair, and only transfer the summary result of the difference data (not the full file in the traditional scheme), which greatly reduces the data transmission amount; at the same time, Pair will record the migration progress in real time, including "migrated data amount" "remaining data amount" "migration rate" and other information, and these progress data will be synchronized to the remote replication management interface, which facilitates subsequent migration state monitoring and exception troubleshooting.

[0061] Further, to realize the visual management of the strategy, the remote replication management interface (management software interface) displays: 1, the remote replication strategy associated with the remote replication link, the display content: the interface presents the basic information of all remote replication links in the form of a list, including the Pair name, the local master directory path, the remote slave directory path, the number of associated remote replication strategies (such as "associated strategy: 3"), the replication state (such as "in synchronization" "completed"); if the Pair is associated with a filtering strategy (that is, a remote replication strategy), the list will be marked with a "filtering strategy configured" mark, which facilitates the user to quickly identify; the data source: the interface data is obtained by calling the pairls interface, which queries the configuration parameters of all Pairs, including "whether the filtering strategy is associated" "the number of associated strategies" and other information, and the interface returns the list rendered by the management software interface. 2, the detail display interface of a single remote replication link, the display content: click on a Pair in the list to enter the detail interface, which will display "basic information" "filtering strategy details" in two parts. Basic information: contains the replication rate of the Pair, the recovery strategy (manual / automatic), and the master-slave cluster connection information (Peer address); filtering strategy details: all remote replication strategies associated with the Pair are displayed in the form of a card (a single Pair has a maximum of 16), each strategy card contains: strategy name (mandatory configuration), replication directory specification information (manually set directory or default root directory), file attribute filtering condition (file name), file size limit (such as "greater than or equal to 1024bit"), file operation time (such as "mtime less than 7 days"), user / user group identifier (such as "uid: 1001" "gid: 2001"), and strategy matching logic (annotated "all conditions of the same strategy must be met, and multiple strategies can be met"). The detail interface also provides "strategy modification" "strategy deletion" buttons (which must meet the "pause Pair before modification / deletion" constraint), which can jump to the strategy configuration interface when clicked.

[0062] Further, when the difference data summary package migration to the slave cluster fails (such as remote replication internal I / O failure, link disconnection, slave cluster lacks target directory), the system automatically executes the following retry logic:

[0063] One, the determination method of migration failure: (1) Link state monitoring: The remote replication link (Pair) monitors the connection with the slave cluster in real time. If the link is disconnected (such as network interruption) or the slave cluster node is offline, it is immediately marked as "migration failure"; (2) Data verification failure: After receiving the summary package from the slave cluster, the integrity is verified (such as checksum comparison). If the data is lost / damaged, return "verification failure" response to the master cluster, and the master cluster marks "migration failure". Two, new baseline snapshot retry process: (1) Determine new baseline snapshot: After migration failure, the system queries the "head data" (the latest synchronized state) of the target replication file in the slave cluster, marks it as "new baseline snapshot" and records the snapshot id; (2) Reacquire the difference range: Based on the new baseline snapshot id and the destination snapshot id in step S12, retrigger step S13: the client library calls ICFS_OSD_OP_DIFF_SNAPS operation to OSD, inputs data object identifier, new baseline / destination snapshot id, OSD parses clone_overlap metadata to obtain new difference range; (3) Re-process migration: re-process according to the "data object complete data determination, difference segment cutting, summary" process, and then migrate to the slave cluster again; If it still fails after retrying 3 times, a pop-up window will be prompted: "migration failed, please check the link or slave cluster directory configuration", and failure logs (including time, fault type) will be recorded. Three, special scene adaptation: If the migration fails due to policy modification (such as the original policy restricts that dir1 is not replicated, and there is no dir1 on the slave end; after deleting the policy, synchronization of dir1 / a fails), check the slave cluster directory before retrying: if the target directory (such as dir1) is missing, automatically create according to the master cluster directory structure, then execute the process of new baseline determination, reacquire difference, and migration to ensure successful retry.

[0064] Further, the filtering strategy needs to follow the following constraints during configuration and execution: (1) master-slave data consistency constraint: filtering strategy achieves on-demand synchronization by screening target replication files, which will cause the data inconsistency between the master cluster (source end) and the slave cluster (destination end) (the slave end only stores the filtered target files), so the remote replication link containing the filtering strategy only supports the backup scenario, and does not support the master-slave switching function (if you try to switch, the system returns error code -335, prompting "Pair containing filtering strategy is prohibited from master-slave switching"); (2) operation range constraint: filtering strategy only takes effect on file addition and modification operations, and does not filter file or directory deletion operations (i.e. after the master cluster deletes a file, if the file has been synchronized to the slave cluster, the slave cluster will still perform the deletion operation), preventing abnormality caused by the difference between the master and slave files that cannot handle the renamed file; (3) replication mode constraint: the Pair configured with the filtering strategy will be forced to switch to the polling comparison mode for data replication, rather than the default snapdiff mode (snapshot difference mode), avoiding the synchronization omission caused by the snapdiff mode's inability to handle file rename (rename) operations; (4) strategy modification constraint: before modifying the filtering strategy, you must first execute the "pausePair" operation (pause the remote replication link) through the remote replication management interface or command line, and if you modify it directly without pausing, the system will return an error prompt "Filtering strategy needs to be modified after pausing Pair"; and filtering strategy modification will temporarily occupy replication resources, affecting synchronization speed, so frequent modification is not recommended (it is recommended to modify at intervals of no less than 1 hour to avoid resource waste); (5) strategy quantity constraint: a single Pair supports a maximum of 16 filtering strategies, and if the limit is exceeded, the system will return error code -332, prompting "Single Pair filtering strategy quantity exceeds limit (maximum 16)", and redundant strategies need to be deleted before being added; (6) strategy deletion switching constraint: if the Pair has deleted all filtering strategies, it needs to complete a complete synchronization task (to ensure master-slave data consistency) before performing the master-slave switching operation; if the synchronization is not completed directly, the system will return error code -335, prompting "Complete a synchronization before allowing master-slave switching".

[0065] It should be noted that when selecting the target Pair for strategy modification through the remote replication management interface, only the strategy name cannot be modified, and the remaining configuration items (replication directory, file name screening, file size threshold, operation time range, user / user group identifier) support adjustment; after modification, the "resumePair" operation (resume the remote replication link) needs to be executed again, and the system will filter the subsequently added / modified files based on the new strategy, and the historical files synchronized to the slave cluster will not be affected.

[0066] It should be noted that in the lib library operation, after obtaining the difference data segment of each data object, in addition to the summary of the difference data segment, a difference data map will also be generated, which records the corresponding relationship of "data object identifier-difference data segment-cluster storage path", and the specific implementation is as follows: the structure of the difference data map: stored in the form of key-value, the key is the data object identifier (such as "obj1"), and the value is a structure containing "offset~len of the difference data segment", "storage address of the data object in the cluster" and "target replication file path"; the core function: after receiving the difference data summary package from the cluster, the difference data map can quickly locate the storage location of each difference data segment, without traversing the entire slave directory, which improves the data reorganization efficiency; at the same time, if a difference data segment is lost during migration, the missing object identifier can be located through the map, and only the difference data segment of the object needs to be retransmitted, reducing the retry cost.

[0067] Further, in order to cope with the bandwidth occupation difference of the master cluster in different business periods, and avoid the remote replication traffic from occupying the core business bandwidth during the business peak, the application can also add a dynamic rate adjustment function in the remote replication link, automatically adjust the replication rate according to the real-time business load (such as CPU utilization) of the master cluster, reduce the rate during the peak period, and increase the rate during the low peak period, which takes into account the business stability and replication efficiency.

[0068] It can be seen that the application provides a distributed file remote replication method, comprising: determining a remote replication strategy associated with a remote replication link between a master cluster and a slave cluster; wherein the remote replication strategy comprises replication directory designation information and file attribute filtering conditions; determining target replication files from files stored in the master cluster according to the replication directory designation information and the file attribute filtering conditions, and generating file snapshots corresponding to the target replication files; obtaining a difference data range between a reference snapshot in the file snapshots and a target snapshot in the file snapshots; the reference snapshot is a completed replication snapshot, and the target snapshot is a to-be-replicated snapshot; determining complete data of each data object according to an object identifier associated with the target replication files, intercepting difference data segments from the complete data based on the difference data range, aggregating the difference data segments of all data objects, and migrating corresponding aggregation results to the slave cluster. As can be seen from the above, by using the replication directory designation information and the file attribute filtering conditions in the remote replication strategy, the target replication files in the master cluster can be accurately determined, so that the storage resources of the slave cluster can be avoided from being occupied by files that do not need to be synchronized, the invalid storage overhead of the slave cluster can be reduced, and the storage resource utilization efficiency can be improved; further, the difference data range is obtained based on the file snapshots, the difference data segments are intercepted from the complete data based on the difference data range, then the difference data segments of all data objects are aggregated, and the corresponding aggregation results are migrated to the slave cluster, instead of replicating the entire file, so that the data transmission amount in the remote replication process can be greatly reduced, especially for the scene of a large file with a small amount of changes, the data processing and transmission time can be significantly reduced, and the replication efficiency can be improved.

[0069] Correspondingly, the application also discloses a distributed file remote replication device, as shown in Figure 2 The device comprises:

[0070] A strategy determination module 11 is configured to determine a remote replication strategy associated with a remote replication link between a master cluster and a slave cluster; wherein the remote replication strategy comprises replication directory designation information and file attribute filtering conditions.

[0071] A snapshot determination module 12 is configured to determine target replication files from files stored in the master cluster according to the replication directory designation information and the file attribute filtering conditions, and generate file snapshots corresponding to the target replication files.

[0072] A difference data determination module 13 is configured to obtain a difference data range between a reference snapshot in the file snapshots and a target snapshot in the file snapshots; the reference snapshot is a completed replication snapshot, and the target snapshot is a to-be-replicated snapshot.

[0073] The difference data migration module 14 is configured to determine complete data of each data object according to the object identifier associated with the target copy file, extract a difference data segment from the complete data based on the difference data range, aggregate difference data segments of all data objects, and migrate the corresponding aggregation result to the slave cluster.

[0074] The working processes of the above modules are more specifically described in the foregoing embodiments, and thus are not described herein again.

[0075] Further, the embodiment of the present application further provides an electronic device. Figure 3 FIG. 2 is a structural diagram of an electronic device 20 according to an example embodiment, and the content in the figure should not be considered as any limitation on the use range of the present application.

[0076] Figure 3 FIG. 2 is a structural diagram of an electronic device 20 according to an example embodiment, and the content in the figure should not be considered as any limitation on the use range of the present application.

[0077] In the embodiment, the power supply 26 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 25 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 25 can be any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input and output interface 24 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited herein.

[0078] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, and the resources stored thereon can include a computer program 221, and the storage mode can be temporary storage or permanent storage. In addition to the computer program capable of completing the distributed file remote replication method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 221 can further include a computer program capable of completing other specific work.

[0079] Further, the embodiment of the present application further discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the distributed file remote replication method disclosed above.

[0080] The specific steps of the method can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0081] The embodiments in the present application are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0082] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0083] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0084] Finally, it should be noted that, in this document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0085] The above describes in detail the distributed file remote replication method, device, equipment and storage medium provided by the present application. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only applicable to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A distributed file remote copying method, characterized in that, include: A remote replication strategy is used to determine the remote replication link association between the master cluster and the slave cluster; wherein, the remote replication strategy includes information on the replication directory and file attribute filtering conditions; Based on the specified information of the replication directory and the file attribute filtering conditions, the target replication file is determined from the files stored in the main cluster, and a file snapshot corresponding to the target replication file is generated; Obtain the range of difference data between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is a snapshot that has been copied, and the destination snapshot is a snapshot to be copied; The complete data of each data object is determined based on the object identifier associated with the target copied file. Based on the difference data range, difference data segments are extracted from the complete data. The difference data segments of all the data objects are summarized, and the corresponding summary results are migrated to the slave cluster.

2. The distributed file remote copying method according to claim 1, characterized in that, The step of determining the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions includes: If the remote replication link is associated with only one remote replication policy, then when any file stored in the master cluster is located in the directory corresponding to the replication directory specified in the remote replication policy, and the file attributes of any file meet the file attribute filtering conditions in the remote replication policy, the file will be identified as the target replication file.

3. The distributed file remote copying method according to claim 1, characterized in that, The step of determining the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions includes: If the remote replication link is associated with multiple remote replication policies, then when any file stored in the master cluster is located in the directory corresponding to the replication directory specified in any of the remote replication policies, and the file attributes of any file meet the file attribute filtering conditions in any of the remote replication policies, the file will be identified as the target replication file.

4. The distributed file remote copying method according to claim 1, characterized in that, The specified information for the copied directory includes the directory name, and the file attribute filtering conditions include one or more of the following: file name filtering conditions, file size filtering conditions, file operation time filtering conditions, user identifier filtering conditions, or user group identifier filtering conditions.

5. The distributed file remote copying method according to claim 4, characterized in that, The remote replication strategy for determining the remote replication link association between the master cluster and the slave cluster includes: Set the directory name through the remote copy management interface, or use the directory name corresponding to the default directory; One or more target filenames can be set through the remote copy management interface, so as to determine the filename filtering conditions in the file attribute filtering conditions based on the one or more target filenames; By selecting the file size comparison type through the remote copy management interface and setting the corresponding file size threshold, the file size filtering condition in the file attribute filtering conditions can be determined. By selecting the file operation time type through the remote copy management interface and setting the corresponding time size threshold, the file operation time filtering condition in the file attribute filtering conditions can be determined. The target user or target user group is set through the remote copy management interface to determine the user identifier filter condition or the user group identifier filter condition in the file attribute filter conditions. The directory name, file name filter, file size filter, file operation time filter, user identifier filter, or user group identifier filter are integrated to obtain the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster.

6. The distributed file remote copying method according to claim 1, characterized in that, Also includes: The remote replication management interface displays the remote replication strategies associated with the remote replication links between the master cluster and the slave cluster; the corresponding display interface includes a list of remote replication links and a details display interface for each remote replication link.

7. The distributed file remote copying method according to any one of claims 1 to 6, characterized in that, The process of summarizing the difference data segments of all the data objects and migrating the corresponding summary results to the slave cluster also includes: If the migration fails, the header data corresponding to the target file copied from the cluster is used as a new baseline snapshot, and the step of obtaining the difference data range between the baseline snapshot and the target snapshot in the file snapshot is re-executed based on the new baseline snapshot and the target snapshot.

8. A distributed file remote copying device, characterized in that, include: The strategy determination module is used to determine the remote replication strategy associated with the remote replication link between the master cluster and the slave cluster; wherein, the remote replication strategy includes replication directory specification information and file attribute filtering conditions; The snapshot determination module is used to determine the target copy file from the files stored in the main cluster based on the specified information of the copy directory and the file attribute filtering conditions, and generate a file snapshot corresponding to the target copy file; The difference data determination module is used to obtain the difference data range between the baseline snapshot and the destination snapshot in the file snapshot; the baseline snapshot is a snapshot that has been copied, and the destination snapshot is a snapshot to be copied; The difference data migration module is used to determine the complete data of each data object based on the object identifier associated with the target copied file, extract difference data segments from the complete data based on the difference data range, summarize the difference data segments of all the data objects, and migrate the corresponding summary results to the slave cluster.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the distributed file remote copying method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the distributed file remote copying method as described in any one of claims 1 to 7.

Images