Incremental synchronization method and device for files in cluster, electronic equipment and storage medium

Abstract

The invention discloses a method and a device for incremental synchronization of files in a cluster, electronic equipment and a storage medium, and relates to the field of data migration, the method comprises the following steps: scanning source files to obtain first file field information of each source file and first metadata information of all the source files; calling all field information files and second metadata information corresponding to the first incremental synchronization task; comparing the first metadata information with the second metadata information to obtain a comparison result, and determining a field information file hit by the first metadata information; and matching the first file field information with the file field information of each source file in the hit field information file, determining an incremental file based on a matching result, and synchronizing the incremental file to the target cluster. The technical problem that in the related technology, when data migration is carried out, data in files of a source end and a target end needs to be compared one by one, so that incremental files are determined, incremental synchronization is carried out, and the synchronization efficiency is low is solved.

Description

Technical Field

[0001] The present invention relates to the field of data migration, and in particular to a method and device for synchronizing incremental files in a cluster, an electronic device and a storage medium. Background Art

[0002] With the rapid development of big data technology, data migration has become an important business in big data technology, not only because of the explosive growth of data volume, but also because of the continuous advancement of data storage and processing technology. Enterprises need to migrate data in order to improve data processing efficiency, reduce costs, enhance data security, and optimize business processes. Data migration involves transferring data from one storage system or location to another, which is not only a technical challenge but also a strategic decision. During the migration process, it is necessary to ensure the integrity, availability, and consistency of the data, while also considering the system compatibility and performance optimization after the migration. With the popularization of cloud computing and distributed computing, data migration also tends to be more flexible and automated to adapt to the ever-changing business needs and technical environment.

[0003] During the data migration process, in order to ensure data consistency or to achieve data disaster recovery during the data migration process, the newly generated incremental data of the source cluster is often synchronized to the new cluster regularly after the data migration. In related technologies, after the data migration is completed, it is necessary to read and compare the massive cluster files or data one by one to determine whether there are changes such as additions and modifications in the data in the source cluster, so as to synchronize the changed data to the new cluster. This incremental synchronization method is difficult to achieve efficient incremental synchronization and has the problem of low synchronization efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for incremental synchronization of files in a cluster, an electronic device and a storage medium, so as to at least solve the technical problem in the related art that when performing data migration, the data in the files on the source and target ends need to be compared one by one to determine the incremental files and perform incremental synchronization, and the synchronization efficiency is low.

[0006] According to one aspect of an embodiment of the present invention, a method for incremental synchronization of files in a cluster is provided, comprising: scanning source files in a source cluster at a target time point based on a preset scanning condition to obtain first file field information of each source file and first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is file field information of each source file obtained based on a scanning operation at the target time point, the first metadata information is metadata information of all source files obtained based on a scanning operation at the target time point, and M is a positive integer; calling all field information files and second metadata information corresponding to a first incremental synchronization task that has been executed, wherein the first incremental synchronization task represents an incremental synchronization task with the smallest time difference between a task execution time point and the target time point, The second metadata information is metadata information of all the source files obtained based on the first incremental synchronization task, and the field information file is used to store file field information of each of the source files during the execution of the first incremental synchronization task; the first metadata information and the second metadata information are compared to obtain a comparison result, and the field information file hit by the first metadata information is determined based on the comparison result; the first file field information is matched with the file field information of each of the source files in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file does not match any item of file field information of any source file in the field information file, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0007] Optionally, before scanning the source files in the source cluster at the target time point based on preset scanning conditions, it also includes: determining the source cluster and the target cluster for data migration; scanning the source files in the source cluster based on the preset scanning conditions to obtain the file field information of each source file; for each round of scanning operation, storing the file field information corresponding to the scanned M source files into the field information file to obtain N field information files, where N is a positive integer; counting the metadata information of all the source files in each field information file to obtain the metadata information corresponding to each field information file, and storing the metadata information corresponding to each field information file into the metadata information file; for each incremental synchronization task, updating the field information file and the metadata information file corresponding to the incremental synchronization task after the execution of the incremental synchronization task is completed.

[0008] Optionally, after comparing the first metadata information and the second metadata information to obtain a comparison result, the method further includes: when the comparison result indicates that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information, determining that the first metadata information hits the second metadata information; when the comparison result indicates that the metadata in the first metadata information is not within the range indicated by the metadata in the second metadata information, determining that the first metadata information does not hit the second metadata information.

[0009] Optionally, after determining that the first metadata information hits the second metadata information, it also includes: obtaining the metadata information of each field information file corresponding to the first incremental synchronization task; matching the first metadata information with the metadata information of each field information file, and when the first metadata information successfully matches the metadata information of the field information file, determining the field information file hit by the first metadata information based on the successfully matched metadata information.

[0010] Optionally, after determining that the first metadata information does not hit the second metadata information, it also includes: obtaining difference information between the first metadata information and the second metadata information; determining the incremental file based on the difference information, and synchronizing the incremental file to the target cluster.

[0011] Optionally, the file field information includes at least: a file identifier, an operation time point, a file byte count and migration result information, wherein the operation time point indicates a time point at which a modification operation is performed on the source file.

[0012] Optionally, the metadata information includes at least: the maximum value and the minimum value of the file identifiers of the M source files, the maximum value and the minimum value of the operation time points of the M source files, and the maximum value and the minimum value of the file byte counts of the M source files.

[0013] According to another aspect of an embodiment of the present invention, a device for incremental synchronization of files in a cluster is also provided, including: a scanning unit, used to scan the source files in the source cluster at a target time point based on a preset scanning condition, and obtain the first file field information of each source file and the first metadata information of all the source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is the file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is the metadata information of all the source files obtained based on the scanning operation at the target time point, and M is a positive integer; a calling unit, used to call all field information files and second metadata information corresponding to the executed first incremental synchronization task, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, The second metadata information is metadata information of all the source files obtained based on the first incremental synchronization task, and the field information file is used to store file field information of each of the source files during the execution of the first incremental synchronization task; a comparison unit is used to compare the first metadata information and the second metadata information to obtain a comparison result, and determine the field information file hit by the first metadata information based on the comparison result; a matching unit is used to match the first file field information with the file field information of each of the source files in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file does not match any item of file field information of any source file in the field information file, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0014] Optionally, the file incremental synchronization device in the cluster also includes: a first determination module, used to determine the source cluster and the target cluster for data migration; a first scanning module, used to scan the source files in the source cluster based on the preset scanning conditions, and obtain the file field information of each source file; a first storage module, used to store the file field information corresponding to the scanned M source files in the field information file for each round of scanning operation, and obtain N field information files, where N is a positive integer; a review statistics module, used to count the metadata information of all the source files in each field information file, obtain the metadata information corresponding to each field information file, and store the metadata information corresponding to each field information file in the metadata information file; a first update module, used to update the field information file and the metadata information file corresponding to the incremental synchronization task after the execution of the incremental synchronization task is completed for each incremental synchronization task.

[0015] Optionally, the incremental file synchronization device in the cluster also includes: a second determination module, used to determine that the first metadata information hits the second metadata information when the comparison result indicates that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information; and a third determination module, used to determine that the first metadata information does not hit the second metadata information when the comparison result indicates that the metadata in the first metadata information is not within the range indicated by the metadata in the second metadata information.

[0016] Optionally, the file incremental synchronization device in the cluster also includes: a first acquisition module, used to obtain metadata information of each field information file corresponding to the first incremental synchronization task; a first matching module, used to match the first metadata information with the metadata information of each field information file, and when the first metadata information successfully matches the metadata information of the field information file, determining the field information file hit by the first metadata information based on the successfully matched metadata information.

[0017] Optionally, the file incremental synchronization device in the cluster also includes: a second acquisition module, used to obtain difference information between the first metadata information and the second metadata information; a first synchronization module, used to determine the incremental file based on the difference information and synchronize the incremental file to the target cluster.

[0018] Optionally, the file field information includes at least: a file identifier, an operation time point, a file byte count and migration result information, wherein the operation time point indicates a time point at which a modification operation is performed on the source file.

[0019] Optionally, the metadata information includes at least: the maximum value and the minimum value of the file identifiers of the M source files, the maximum value and the minimum value of the operation time points of the M source files, and the maximum value and the minimum value of the file byte counts of the M source files.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned methods for incremental synchronization of files in a cluster.

[0021] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for incremental synchronization of files in a cluster.

[0022] In the present application, the following steps are performed: first, based on a preset scanning condition, the source files in the source cluster at the target time point are scanned to obtain the first file field information of each source file and the first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is the file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is the metadata information of all source files obtained based on the scanning operation at the target time point, M is a positive integer, and then all the field information files and the second metadata information corresponding to the executed first incremental synchronization task are called, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, and the second metadata information is the metadata information of all source files obtained based on the scanning operation at the target time point. The information is metadata information of all source files obtained based on the first incremental synchronization task. The field information file is used to store the file field information of each source file during the execution of the first incremental synchronization task, and compare the first metadata information and the second metadata information to obtain a comparison result, and determine the field information file hit by the first metadata information based on the comparison result. Finally, the first file field information is matched with the file field information of each source file in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file and any file field information of any source file in the field information file do not match successfully, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0023] In the present application, field information of multiple cluster files and metadata information of all scanned files are stored through field information files and metadata information files, so that when identifying incremental files, the metadata information of the currently scanned batch source files can be compared in segments with the metadata information obtained by the last incremental synchronization task, so as to identify incremental files based on the comparison results of the metadata information. At the same time, a secondary comparison is performed based on the file field information of each source file currently scanned and the file field information obtained by the last incremental synchronization task, so as to more accurately identify incremental files and synchronize the incremental files, without having to compare the files in the source cluster with the files in the target cluster one by one. Segmented reading and comparison can reduce memory resource overhead, and the incremental files can be quickly identified and synchronized through the metadata information and file field information retained by the incremental synchronization task, saving incremental synchronization time and accelerating the synchronization efficiency of incremental files, thereby solving the technical problem in the related technology that when performing data migration, the data in the files on the source and target ends need to be compared one by one to determine the incremental files and perform incremental synchronization, and the synchronization efficiency is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a flow chart of an optional method for incremental synchronization of files in a cluster according to an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of an optional process of establishing a field information file and a metadata information file according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an optional device for synchronizing incremental files in a cluster according to an embodiment of the present invention;

[0028] Figure 4 The present invention is a hardware structure block diagram of an electronic device (or mobile device) according to a method for incremental synchronization of files in a cluster according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the incremental file synchronization method and device in the cluster in the present application can be used in the field of data migration when incremental synchronization of files in the cluster is performed, and can also be used in any field except the field of data migration when incremental synchronization of files in the cluster is performed. The present application does not limit the application field of the incremental file synchronization method and device in the cluster.

[0032] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0033] It should be noted that in this application, when collecting and analyzing customer information, corresponding operation entrances are provided for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0034] The following embodiments of the present invention can be applied to various file incremental synchronization systems / applications / devices in clusters. The present invention updates and retains the file field information of the files in the source cluster and the metadata information of the batch files after each incremental synchronization task. When identifying incremental files, the current metadata information and file field information of the scanned batch files are compared with the metadata information and file field information of the previous batch files, and the scanning of the target cluster is eliminated, saving the resources of the target cluster. At the same time, the batch scanning and data comparison of batch files can greatly reduce the comparison time and speed up the recognition and synchronization efficiency of incremental files.

[0035] The present invention is described in detail below in conjunction with various embodiments.

[0036] Embodiment 1

[0037] According to an embodiment of the present invention, an embodiment of a method for incremental synchronization of files in a cluster is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 1 is a flow chart of an optional method for incremental synchronization of files in a cluster according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0039] Step S101, scanning source files in the source cluster at the target time point based on a preset scanning condition, obtaining first file field information of each source file and first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is metadata information of all source files obtained based on the scanning operation at the target time point, and M is a positive integer;

[0040] Step S102, calling all field information files and second metadata information corresponding to the executed first incremental synchronization task, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, the second metadata information is metadata information of all source files obtained based on the first incremental synchronization task, and the field information file is used to store the file field information of each source file during the execution of the first incremental synchronization task;

[0041] Step S103, comparing the first metadata information and the second metadata information to obtain a comparison result, and determining the field information file hit by the first metadata information based on the comparison result;

[0042] Step S104, matching the first file field information with the file field information of each source file in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file and any file field information of any source file in the field information file do not match successfully, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0043] Through the above steps, firstly, the source files in the source cluster at the target time point are scanned based on the preset scanning conditions to obtain the first file field information of each source file and the first metadata information of all source files, wherein the scanning conditions at least include: scanning M source files in each round, the first file field information is the file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is the metadata information of all source files obtained based on the scanning operation at the target time point, M is a positive integer, and then all the field information files and the second metadata information corresponding to the executed first incremental synchronization task are called, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, and the second metadata information is Based on the metadata information of all source files obtained by the first incremental synchronization task, the field information file is used to store the file field information of each source file during the execution of the first incremental synchronization task, and compare the first metadata information and the second metadata information to obtain a comparison result, and determine the field information file hit by the first metadata information based on the comparison result, and finally match the first file field information with the file field information of each source file in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file does not match any file field information of any source file in the field information file, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0044] In this embodiment, field information of multiple cluster files and metadata information of all scanned files are stored through field information files and metadata information files, so that when identifying incremental files, the metadata information of the currently scanned batch source files can be compared in segments with the metadata information obtained by the previous incremental synchronization task, so as to identify incremental files based on the comparison results of the metadata information. At the same time, a secondary comparison is performed based on the file field information of each source file currently scanned and the file field information obtained by the previous incremental synchronization task, so as to more accurately identify incremental files and synchronize incremental files, without having to compare files in the source cluster with files in the target cluster one by one. Segmented reading and comparison can reduce memory resource overhead, and incremental files can be quickly identified and synchronized through the metadata information and file field information retained by the incremental synchronization task, thereby saving incremental synchronization time and accelerating the synchronization efficiency of incremental files, thereby solving the technical problem in the related technology that when performing data migration, the data in the files on the source and target ends need to be compared one by one to determine incremental files and perform incremental synchronization, and the synchronization efficiency is low.

[0045] The embodiment of the present invention is described in detail below in combination with the above steps.

[0046] It should be noted that in the storage system, in order to achieve data disaster recovery in different storage systems, a full copy of the data in the source cluster is usually performed, and then incremental data synchronization is performed at fixed intervals or at a fixed time point to ensure the consistency of data in the source cluster and the target cluster, and to achieve data disaster recovery for the source cluster and the target cluster. In the event of an abnormality in the new target cluster, it can be rolled back to the source cluster without affecting the normal operation of the business.

[0047] The embodiment of the present invention establishes field information files and metadata information files for the source files in the source cluster based on full copy, and retains the file field information of each source file and the metadata information of the batch files obtained by one-time scanning. When identifying and synchronizing incremental files, the field information file and metadata information file corresponding to the most recent incremental synchronization task at the current time point can be called to compare the metadata information of the batch files with the file field information of each source file. By eliminating the scanning of the target cluster, the resources of the target cluster can be saved. At the same time, while ensuring the accurate identification of incremental files, the time for scanning and identifying incremental files can be saved, thereby improving the synchronization efficiency of incremental files.

[0048] Figure 2 This is a schematic diagram of an optional process of establishing a field information file and a metadata information file according to an embodiment of the present invention. Before scanning the source files in the source cluster at the target time point based on the preset scanning conditions, it also includes:

[0049] Step S201, determining a source cluster and a target cluster for data migration;

[0050] It should be noted that before real-time synchronization of incremental files, you first need to determine the source cluster and target cluster for data migration. The source cluster is the cluster for file migration out, and the target cluster is the cluster for file migration in. After the full copy, the initial field information file and metadata information file are established for the source file in the source cluster. After each subsequent incremental synchronization task, the initial field information file and metadata information file of the source file in the source cluster are updated to obtain the field information file and metadata information file corresponding to each incremental synchronization task.

[0051] Step S202: Scan the source files in the source cluster based on the preset scanning conditions to obtain file field information of each source file;

[0052] Optionally, the file field information includes at least: a file identifier, an operation time point, a file byte count, and migration result information, wherein the operation time point indicates a time point when a modification operation is performed on the source file.

[0053] It should be noted that in order to speed up the incremental synchronization efficiency, the embodiment of the present invention scans a certain number of batch files at the same time, analyzes and identifies the batch files. Specifically, the source files in the source cluster are scanned according to preset scanning conditions. The scanning conditions can limit the number of files to be scanned at one time and the file information to be scanned. For example, one hundred thousand files are scanned at one time to obtain the file field information of each source file. The file field information may include: file identification, operation time point of the file operation, file byte number of the file, and file migration result information, etc. The operation time point refers to the time point of modifying, adding, deleting and other operations on the source file.

[0054] Step S203, for each round of scanning operation, the file field information corresponding to the scanned M source files is stored in a field information file to obtain N field information files, where N is a positive integer;

[0055] It should be noted that for each round of scanning, the file field information corresponding to the scanned M source files is stored in the field information file, that is, each M source files corresponds to a field information file. This is to avoid the number of files being too large to be loaded into the memory. Therefore, the source files are split and scanned to obtain multiple field information files.

[0056] Step S204, counting the metadata information of all source files in each field information file, obtaining the metadata information corresponding to each field information file, and storing the metadata information corresponding to each field information file into the metadata information file;

[0057] Optionally, the metadata information includes at least: the maximum value and the minimum value of the file identifiers of the M source files, the maximum value and the minimum value of the operation time points of the M source files, and the maximum value and the minimum value of the file byte counts of the M source files.

[0058] It should be noted that after completing a scan, metadata information of each field information file is obtained based on the data stored in the field information file. The metadata information is data obtained by summarizing and analyzing the M source files in the field information file, and the metadata information obtained by the analysis is stored in the metadata information file. Specifically, the metadata information may include: the maximum value and minimum value of the file identifiers of the M source files in the field information file, the maximum value and minimum value of the operation time points of the M source files in the field information file, and the maximum value and minimum value of the file byte number of the M source files in the field information file.

[0059] Step S205: for each incremental synchronization task, after the incremental synchronization task is executed, the field information file and metadata information file corresponding to the incremental synchronization task are updated.

[0060] It should be noted that after each incremental synchronization task is completed, the field information file and metadata information file corresponding to the incremental synchronization task will be updated to ensure that the data in the field information file and metadata information file of the source cluster are the latest.

[0061] Step S101 : Scan source files in a source cluster at a target time point based on a preset scanning condition to obtain first file field information of each source file and first metadata information of all source files.

[0062] It should be noted that in order to ensure the consistency of files and data in the source cluster and the target cluster, it is necessary to regularly identify files in the source cluster, obtain incremental files, and synchronize incremental files in a timely manner. Incremental files can be files that are modified, added, or deleted in the source cluster. Through pre-set scanning conditions, multiple source files in the source cluster at the target time point are scanned at one time to obtain the first file field information of each source file and the first metadata information of all source files scanned this time.

[0063] It should be noted that the above scanning conditions at least include: scanning M source files in each round, the above first file field information is the file field information of each source file obtained based on the scanning operation at the target time point, and may include the file identification, operation time point, file byte number and migration result information of the source file scanned this time, and the first metadata information is the metadata information of all source files obtained based on the scanning operation at the target time point, and may include the maximum value and minimum value of the file identification corresponding to all the source files scanned this time, the maximum value and minimum value of the operation time point, and the maximum value and minimum value of the file byte number.

[0064] Step S102: Call all field information files and second metadata information corresponding to the executed first incremental synchronization task.

[0065] It should be noted that in order to quickly identify the incremental files in this scanning operation, all field information files and second metadata information corresponding to the first incremental synchronization task that has been executed can be called. The first incremental synchronization task refers to the incremental synchronization task that is closest to the target time point of this scan, that is, the incremental synchronization task with the smallest time difference between the task execution time point and the target time point. The above-mentioned second metadata information is the metadata information of all source files obtained based on the first incremental synchronization task, and the field information file is used to store the file field information of each source file during the execution of the first incremental synchronization task.

[0066] Step S103, comparing the first metadata information and the second metadata information to obtain a comparison result, and determining the field information file hit by the first metadata information based on the comparison result;

[0067] It should be noted that when identifying incremental files, the metadata information obtained from this scanning operation is compared with the metadata information obtained from the previous scanning operation, that is, the first metadata information is compared with the second metadata information. When comparing, all field information files of the previous scanning operation are loaded from the disk to the memory for comparison, thereby reducing the number of disk reads and writes. Based on the comparison results of the metadata information, it can be preliminarily determined whether there are incremental files. At the same time, based on the comparison results, the field information files hit by the first metadata information can also be determined.

[0068] Optionally, after comparing the first metadata information and the second metadata information to obtain a comparison result, the method further includes: when the comparison result indicates that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information, determining that the first metadata information hits the second metadata information; when the comparison result indicates that the metadata in the first metadata information is not within the range indicated by the metadata in the second metadata information, determining that the first metadata information does not hit the second metadata information.

[0069] Specifically, if the comparison result shows that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information, that is, the ranges of the file identifier, operation time point and file byte number in the first metadata information are all within the ranges of the file identifier, operation time point and file byte number in the second metadata information, it indicates that the first metadata is determined to hit the second metadata information; conversely, if any one of the ranges of the file identifier, operation time point and file byte number in the first metadata information is not within the ranges of the file identifier, operation time point and file byte number in the second metadata information, it indicates that the first metadata does not hit the second metadata information, that is, there is an incremental file during this scan, and the incremental file can be directly determined based on the metadata that exceeds the range.

[0070] Optionally, after determining that the first metadata information does not hit the second metadata information, it also includes: obtaining difference information between the first metadata information and the second metadata information; determining an incremental file based on the difference information, and synchronizing the incremental file to the target cluster.

[0071] It should be noted that, when the first metadata information does not hit the second metadata information, the difference information between the first metadata information and the second metadata information is calculated. The difference information may include the difference information of the file identifier, the difference information of the operation time point and the difference information of the number of file bytes. The incremental file of this scan is determined based on the difference information, so as to quickly synchronize the incremental file to the target cluster.

[0072] Optionally, after determining that the first metadata information hits the second metadata information, it also includes: obtaining the metadata information of each field information file corresponding to the first incremental synchronization task; matching the first metadata information with the metadata information of each field information file, and when the first metadata information matches the metadata information of the field information file successfully, determining the field information file hit by the first metadata information based on the successfully matched metadata information.

[0073] It should be noted that each field information file has corresponding metadata information. If the first metadata information hits the second metadata information, the metadata information of each field information file corresponding to the first incremental synchronization task is obtained, and the first metadata information is matched with the metadata information corresponding to each field information file. The field information file hit by the first metadata information is obtained based on whether the match is successful or not. If the first metadata information hits multiple field information files, the first metadata information is split, and the split first metadata information fragments are respectively matched with the metadata information corresponding to each field information file to determine the field information file hit by the first metadata information fragment.

[0074] Step S104, matching the first file field information with the file field information of each source file in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file and any file field information of any source file in the field information file do not match successfully, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0075] It should be noted that after obtaining the metadata information of the field information file that hits the first metadata information or the first metadata information fragment, the first field file information obtained from this scan is matched with the hit field information file to obtain a matching result. If the matching result indicates that the first file field information of the source file and all the file field information of any source file in the field information file are not matched successfully, that is, the file identification, operation time point, file byte number and migration result information corresponding to the source file scanned this time do not hit any of the file identification, operation time point, file byte number and migration result information corresponding to the source file in the last incremental synchronization task, it is determined that the source file is an incremental file and the incremental file is synchronized to the target cluster. On the contrary, if the first file field information of the source file and all the file field information of a source file in the field information file are matched successfully, it indicates that the source file has not changed and is not an incremental file, and can be skipped directly.

[0076] The following is a detailed description in conjunction with another embodiment.

[0077] Embodiment 2

[0078] The device for synchronizing incremental files in a cluster provided in this embodiment includes multiple implementation units, each of which corresponds to each implementation step in the above-mentioned embodiment 1. The specific implementation methods and beneficial effects thereof can be referred to the above-mentioned method embodiments, which will not be described in detail here.

[0079] Figure 3 is a schematic diagram of an optional device for synchronizing incremental files in a cluster according to an embodiment of the present invention. Figure 3 As shown, the file increment synchronization device in the cluster includes: a scanning unit 31, a calling unit 32, a comparison unit 33, and a matching unit 34, wherein:

[0080] The scanning unit 31 is used to scan the source files in the source cluster at the target time point based on a preset scanning condition to obtain the first file field information of each source file and the first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is the file field information of each source file obtained based on the scanning operation at the target time point, and the first metadata information is the metadata information of all source files obtained based on the scanning operation at the target time point, and M is a positive integer;

[0081] A calling unit 32 is used to call all field information files and second metadata information corresponding to the executed first incremental synchronization task, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, the second metadata information is metadata information of all source files obtained based on the first incremental synchronization task, and the field information file is used to store file field information of each source file during the execution of the first incremental synchronization task;

[0082] The comparison unit 33 is used to compare the first metadata information with the second metadata information to obtain a comparison result, and determine the field information file hit by the first metadata information based on the comparison result;

[0083] The matching unit 34 is used to match the first file field information with the file field information of each source file in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file and any file field information of any source file in the field information file are not successfully matched, the source file corresponding to the first file field information that is not successfully matched is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0084] The above-mentioned file incremental synchronization device in the cluster scans the source files in the source cluster at the target time point based on the preset scanning conditions through the scanning unit 31 to obtain the first file field information of each source file and the first metadata information of all source files, wherein the scanning conditions at least include: scanning M source files in each round, the first file field information is the file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is the metadata information of all source files obtained based on the scanning operation at the target time point, and M is a positive integer; calling all the field information files and the second metadata information corresponding to the executed first incremental synchronization task through the calling unit 32, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, and the second metadata information The information is metadata information of all source files obtained based on the first incremental synchronization task, and the field information file is used to store file field information of each source file during the execution of the first incremental synchronization task; the first metadata information and the second metadata information are compared by the comparison unit 33 to obtain a comparison result, and the field information file hit by the first metadata information is determined based on the comparison result; the first file field information is matched with the file field information of each source file in the hit field information file by the matching unit 34 to obtain a matching result. When the matching result indicates that the first file field information of the source file and any file field information of any source file in the field information file are not successfully matched, the source file corresponding to the first file field information that is not successfully matched is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0085] In this embodiment, field information of multiple cluster files and metadata information of all scanned files are stored through field information files and metadata information files, so that when identifying incremental files, the metadata information of the currently scanned batch source files can be compared in segments with the metadata information obtained by the previous incremental synchronization task, so as to identify incremental files based on the comparison results of the metadata information. At the same time, a secondary comparison is performed based on the file field information of each source file currently scanned and the file field information obtained by the previous incremental synchronization task, so as to more accurately identify incremental files and synchronize incremental files, without having to compare files in the source cluster with files in the target cluster one by one. Segmented reading and comparison can reduce memory resource overhead, and incremental files can be quickly identified and synchronized through the metadata information and file field information retained by the incremental synchronization task, thereby saving incremental synchronization time and accelerating the synchronization efficiency of incremental files, thereby solving the technical problem in the related technology that when performing data migration, the data in the files on the source and target ends need to be compared one by one to determine incremental files and perform incremental synchronization, and the synchronization efficiency is low.

[0086] Optionally, the file incremental synchronization device in the cluster also includes: a first determination module, used to determine the source cluster and the target cluster for data migration; a first scanning module, used to scan the source files in the source cluster based on preset scanning conditions to obtain the file field information of each source file; a first storage module, used to store the file field information corresponding to the scanned M source files into the field information file for each round of scanning operation, and obtain N field information files, where N is a positive integer; a review statistics module, used to count the metadata information of all source files in each field information file, obtain the metadata information corresponding to each field information file, and store the metadata information corresponding to each field information file into the metadata information file; a first update module, used to update the field information file and metadata information file corresponding to the incremental synchronization task after the incremental synchronization task is executed for each incremental synchronization task.

[0087] Optionally, the incremental file synchronization device in the cluster also includes: a second determination module, used to determine that the first metadata information hits the second metadata information when the comparison result indicates that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information; a third determination module, used to determine that the first metadata information does not hit the second metadata information when the comparison result indicates that the metadata in the first metadata information is not within the range indicated by the metadata in the second metadata information.

[0088] Optionally, the file incremental synchronization device in the cluster also includes: a first acquisition module, used to obtain metadata information of each field information file corresponding to the first incremental synchronization task; a first matching module, used to match the first metadata information with the metadata information of each field information file, and when the first metadata information matches the metadata information of the field information file successfully, determine the field information file hit by the first metadata information based on the successfully matched metadata information.

[0089] Optionally, the device for synchronizing incremental files in a cluster further includes: a second acquisition module for acquiring difference information between the first metadata information and the second metadata information; and a first synchronization module for determining incremental files based on the difference information and synchronizing the incremental files to the target cluster.

[0090] Optionally, the file field information includes at least: a file identifier, an operation time point, a file byte count, and migration result information, wherein the operation time point indicates a time point when a modification operation is performed on the source file.

[0091] Optionally, the metadata information includes at least: the maximum value and the minimum value of the file identifiers of all the source files scanned, the maximum value and the minimum value of the operation time points of all the source files scanned, and the maximum value and the minimum value of the file byte number of all the source files scanned.

[0092] The file incremental synchronization device in the above-mentioned cluster may also include a processor and a memory. The above-mentioned scanning unit 31, calling unit 32, comparison unit 33, matching unit 34, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0093] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the files in the cluster can be incrementally synchronized by adjusting kernel parameters.

[0094] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0095] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned methods for incremental synchronization of files in a cluster.

[0096] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned methods for incremental synchronization of files in a cluster.

[0097] According to another aspect of an embodiment of the present invention, a computer program product is further provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, any of the above-mentioned methods for incremental synchronization of files in a cluster is implemented.

[0098] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: scanning source files in a source cluster at a target time point based on a preset scanning condition to obtain first file field information of each source file and first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is metadata information of all source files obtained based on the scanning operation at the target time point, and M is a positive integer; calling all field information files and second metadata information corresponding to the executed first incremental synchronization task, wherein the first incremental synchronization task indicates the time between the task execution time point and the target time point. The incremental synchronization task with the smallest difference, the second metadata information is the metadata information of all source files obtained based on the first incremental synchronization task, and the field information file is used to store the file field information of each source file during the execution of the first incremental synchronization task; the first metadata information and the second metadata information are compared to obtain a comparison result, and the field information file hit by the first metadata information is determined based on the comparison result; the first file field information is matched with the file field information of each source file in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file does not match any file field information of any source file in the field information file, the source file corresponding to the first file field information that did not match successfully is used as an incremental file, and the incremental file is synchronized to the target cluster.

[0099] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that is initialized with the following method steps: before scanning the source files in the source cluster at the target time point based on preset scanning conditions, it also includes: determining the source cluster and the target cluster for data migration; scanning the source files in the source cluster based on the preset scanning conditions to obtain the file field information of each source file; for each round of scanning operation, storing the file field information corresponding to the scanned M source files into a field information file to obtain N field information files, where N is a positive integer; counting the metadata information of all source files in each field information file to obtain the metadata information corresponding to each field information file, and storing the metadata information corresponding to each field information file into the metadata information file; for each incremental synchronization task, updating the field information file and metadata information file corresponding to the incremental synchronization task after the incremental synchronization task is executed.

[0100] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that is initialized with the following method steps: after comparing the first metadata information and the second metadata information to obtain a comparison result, it also includes: when the comparison result indicates that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information, determining that the first metadata information hits the second metadata information; when the comparison result indicates that the metadata in the first metadata information is not within the range indicated by the metadata in the second metadata information, determining that the first metadata information does not hit the second metadata information.

[0101] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program initialized with the following method steps: after determining that the first metadata information hits the second metadata information, it also includes: obtaining the metadata information of each field information file corresponding to the first incremental synchronization task; matching the first metadata information with the metadata information of each field information file, and when the first metadata information successfully matches the metadata information of the field information file, determining the field information file hit by the first metadata information based on the successfully matched metadata information.

[0102] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: after determining that the first metadata information does not hit the second metadata information, it also includes: obtaining difference information between the first metadata information and the second metadata information; determining an incremental file based on the difference information, and synchronizing the incremental file to a target cluster.

[0103] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing an initialization program having the following method steps: the file field information includes at least: file identification, operation time point, file byte count and migration result information, wherein the operation time point indicates the time point at which the source file is modified.

[0104] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that is initialized with the following method steps: the metadata information at least includes: the maximum value and the minimum value of the file identifiers of all source files scanned, the maximum value and the minimum value of the operation time points of all source files scanned, and the maximum value and the minimum value of the file byte number of all source files scanned.

[0105] Figure 4 1 is a hardware structure block diagram of an electronic device (or mobile device) according to a method for synchronizing incremental files in a cluster according to an embodiment of the present invention. Figure 4As shown, the electronic device may include one or more ( Figure 4 402a, 402b, ..., 402n are used to illustrate) processor 402 (processor 402 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), memory 404 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0106] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0107] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0112] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for incremental synchronization of files in a cluster, characterized in that: include: Scanning source files in a source cluster at a target time point based on a preset scanning condition to obtain first file field information of each source file and first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is file field information of each source file obtained based on a scanning operation at the target time point, the first metadata information is metadata information of all source files obtained based on a scanning operation at the target time point, and M is a positive integer; Call all field information files and second metadata information corresponding to the executed first incremental synchronization task, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, and the second metadata information is metadata information of all the source files obtained based on the first incremental synchronization task, and the field information file is used to store the file field information of each of the source files during the execution of the first incremental synchronization task; Compare the first metadata information with the second metadata information to obtain a comparison result, and determine the field information file hit by the first metadata information based on the comparison result; The first file field information is matched with the file field information of each of the source files in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file does not successfully match any item of file field information of any source file in the field information file, the source file corresponding to the first file field information that did not successfully match is used as an incremental file, and the incremental file is synchronized to the target cluster.

2. The synchronization method according to claim 1, characterized in that: Before scanning the source files in the source cluster at the target time point based on the preset scanning conditions, the following steps are also included: Determining the source cluster and the target cluster for data migration; Scan the source files in the source cluster based on the preset scanning condition to obtain the file field information of each source file; For each round of scanning operation, the file field information corresponding to the scanned M source files is stored in the field information file to obtain N field information files, where N is a positive integer; Counting metadata information of all the source files in each of the field information files, obtaining metadata information corresponding to each of the field information files, and storing the metadata information corresponding to each of the field information files in the metadata information file; For each incremental synchronization task, the field information file and the metadata information file corresponding to the incremental synchronization task are updated after the incremental synchronization task is executed.

3. The synchronization method according to claim 1, characterized in that: After comparing the first metadata information and the second metadata information to obtain a comparison result, the method further includes: If the comparison result indicates that the metadata in the first metadata information is within the range indicated by the metadata in the second metadata information, determining that the first metadata information hits the second metadata information; When the comparison result indicates that the metadata in the first metadata information is not within the range indicated by the metadata in the second metadata information, it is determined that the first metadata information does not hit the second metadata information.

4. The synchronization method according to claim 3, characterized in that: After determining that the first metadata information hits the second metadata information, the method further includes: Obtain metadata information of each field information file corresponding to the first incremental synchronization task; The first metadata information is matched with the metadata information of each field information file. When the first metadata information successfully matches the metadata information of the field information file, the field information file hit by the first metadata information is determined based on the successfully matched metadata information.

5. The synchronization method according to claim 3, characterized in that: After determining that the first metadata information does not hit the second metadata information, the method further includes: Acquire difference information between the first metadata information and the second metadata information; The incremental file is determined based on the difference information, and the incremental file is synchronized to the target cluster.

6. The synchronization method according to claim 1, characterized in that: The file field information at least includes: a file identifier, an operation time point, a file byte count, and migration result information, wherein the operation time point indicates a time point at which a modification operation is performed on the source file.

7. The synchronization method according to claim 1, characterized in that: The metadata information includes at least: the maximum value and the minimum value of the file identifiers of the M source files, the maximum value and the minimum value of the operation time points of the M source files, and the maximum value and the minimum value of the file byte counts of the M source files.

8. A device for synchronizing files in a cluster, characterized in that: include: a scanning unit, configured to scan source files in a source cluster at a target time point based on a preset scanning condition, and obtain first file field information of each source file and first metadata information of all source files, wherein the scanning condition at least includes: scanning M source files in each round, the first file field information is file field information of each source file obtained based on the scanning operation at the target time point, the first metadata information is metadata information of all source files obtained based on the scanning operation at the target time point, and M is a positive integer; A calling unit, used to call all field information files and second metadata information corresponding to the executed first incremental synchronization task, wherein the first incremental synchronization task represents the incremental synchronization task with the smallest time difference between the task execution time point and the target time point, and the second metadata information is metadata information of all the source files obtained based on the first incremental synchronization task, and the field information file is used to store the file field information of each of the source files during the execution of the first incremental synchronization task; a comparison unit, configured to compare the first metadata information with the second metadata information to obtain a comparison result, and determine a field information file hit by the first metadata information based on the comparison result; A matching unit is used to match the first file field information with the file field information of each of the source files in the hit field information file to obtain a matching result. When the matching result indicates that the first file field information of the source file and any file field information of any source file in the field information file are not successfully matched, the source file corresponding to the first file field information that is not successfully matched is used as an incremental file, and the incremental file is synchronized to the target cluster.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for incremental synchronization of files in a cluster as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for incremental synchronization of files in a cluster as described in any one of claims 1 to 7.

Images