Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention discloses a data synchronization method and device, electronic equipment and a storage medium, and relates to the technical field of information, and the method comprises the steps: rapidly determining change data through a data synchronization condition; calculating a first directory hash value and a second directory hash value based on the source directory; whether a difference exists between the source directory and the target directory or not can be efficiently judged according to the first directory hash value and the second directory hash value; if a difference exists between the source directory and the target directory, each first file hash value is compared with each second file hash value, so that a difference file can be accurately determined; finally, the target data in the source file of the difference file pair can be quickly synchronized to the corresponding target file according to the data synchronization strategy of the thread, so that the problems that the comparison efficiency of the data in the file is low and the consistency change of the synchronous data is difficult to guarantee are solved; the technical effects of improving the comparison efficiency of the data in the file and ensuring the consistency change of the synchronous data are achieved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] With the development of information technology, there is a need to store user data, obtain data information through multiple networks, or back up data in many fields such as medical care, education, transportation, and finance. However, with the increase in user business needs, the amount of data has increased dramatically, and the need to store the rapidly growing data also requires more efficient, fast, and accurate data synchronization methods. Therefore, how to improve the efficiency of data synchronization is an urgent problem to be solved.

[0003] At present, the traditional data synchronization method is to directly compare the source file with the target file based on the file timestamp, file size and file content to determine the data difference between the target file and the source file, and then synchronize the data by full or incremental replication; these methods are only suitable for processing small batches of data with simple data formats. However, when faced with large batches of data or scenarios where data needs to change frequently, these methods often have low efficiency in comparing data in files and it is difficult to ensure the consistency of synchronized data changes, which also makes data synchronization slow. Summary of the invention

[0004] The present application provides a data synchronization method, device, electronic device and storage medium to at least solve the problem in the related art that the comparison efficiency of data in a file is low and the consistency change of synchronized data is difficult to ensure.

[0005] The present application provides a data synchronization method, including:

[0006] In response to determining that the data synchronization condition is met, determining at least one source storage path corresponding to the target data to be synchronized and a target storage path preconfigured for the target data; the source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized;

[0007] If it is determined that the target file is included in the target directory, a first directory hash value of the source directory storing the data where each target data is located is obtained based on the source directory name, and a second directory hash value of the data stored in each target directory is obtained based on the target directory name;

[0008] In response to the first directory hash value being different from the second directory hash value, obtaining a first file hash value of the source file data where each target data is located, and obtaining a second file hash value of the target file data corresponding to each target data;

[0009] Determine at least one difference file pair according to each first file hash value and each second file hash value, where the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value;

[0010] According to the data synchronization strategy of the thread, the target data in the source file of at least one difference file pair is synchronized to the corresponding target file.

[0011] The present application also provides a data synchronization device, including:

[0012] A determination module, configured to determine, in response to determining that the data synchronization condition is satisfied, a source storage path corresponding to at least one target data to be synchronized and a target storage path preconfigured for the target data; the source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized;

[0013] An acquisition module, for acquiring a first directory hash value of the source directory storing data where each target data is located based on the source directory name if it is determined that the target file is included in the target directory, and acquiring a second directory hash value of the data stored in each target directory based on the target directory name;

[0014] an acquisition module, configured to acquire, in response to the first directory hash value being different from the second directory hash value, a first file hash value of the source file data where each target data is located, and acquire a second file hash value of the target file data corresponding to each target data;

[0015] A determination module, configured to determine at least one difference file pair according to each first file hash value and each second file hash value, wherein the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value;

[0016] The synchronization module is used to synchronize the target data in the source file of at least one difference file pair to the corresponding target file according to the data synchronization strategy of the thread.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data synchronization methods when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data synchronization methods are implemented.

[0019] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data synchronization methods when executed by a processor.

[0020] Through the present application, the changed data can be quickly determined through the data synchronization condition, and when the target directory contains the target file, the first directory hash value of the source directory storing the data where each target data is located can be obtained based on the source directory name, and the second directory hash value of the target directory storing the data can be obtained based on the target directory name; and then it can also be quickly and efficiently determined whether there is a difference between the source directory and the target directory according to the first directory hash value and the second directory hash value, and even if there is a slight difference in the file, the difference between the source directory and the target directory can be determined by the first directory hash value and the second directory hash value; further, when there is a difference between the source directory and the target directory, each first file hash value can continue to be compared with each second file hash value, so that the difference file can be accurately determined; finally, based on the thread data synchronization strategy, the target data in the source file of this difference file pair can be quickly synchronized to the corresponding target file, so that the difference file can be quickly synchronized. Therefore, the problem of low efficiency in comparing the data in the file and difficulty in ensuring the consistency of the synchronized data can be solved, and the technical effect of improving the efficiency of comparing the data in the file and ensuring the consistency of the synchronized data can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 An application scenario diagram of a data synchronization method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0024] Figure 3 A flowchart of a method for obtaining a hash value provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a method for obtaining a hash value provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of a method for setting a symbolic link provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0028] Figure 7A schematic diagram of a data synchronization device provided in an embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0032] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.

[0033] At present, the traditional data synchronization method is to directly compare the source file with the target file based on the file's timestamp, file size and file content. Specifically, query the timestamp, file size and file content of all source and target files, and then compare the timestamp, file size and file content of each source file with the timestamp, file size and file content of the target file one by one to determine the file difference, and then synchronize the data by full or incremental replication; this method is only suitable for processing small batches of data with simple data formats. However, when faced with large batches of data or scenarios where data needs to change frequently, these methods are difficult to achieve fast data synchronization and difficult to ensure consistent changes in synchronized data, which also makes data synchronization slow.

[0034] Therefore, in order to improve the comparison efficiency of the data in the file and ensure the consistency of the synchronized data, the difference file can be determined based on the source file where the target data to be synchronized is located and the corresponding target file, and then only the difference file is synchronized. Moreover, since the hash value has the characteristic of uniqueness, that is, when the data in the file has a slight change, the hash value will also change synchronously, so whether the data has changed can be determined by obtaining the hash value of the source file and the hash value of the target file; and in order to further reduce the number of comparisons of the corresponding hash values ​​of the source file and the target file, it can be determined first whether there is a difference between the source directory and the target directory; that is, the first directory hash value of the source directory storing data where each target data is located and the second directory hash value of the target directory storing data calculated based on the target directory name are obtained, and the first directory hash value of the source directory storing data where each target data is located and the second directory hash value of the target directory storing data calculated based on the target directory name are compared to determine whether there is a difference between the target directory and the source directory; if there is a difference between the target directory and the source directory, in order to improve the speed of data synchronization and reduce the number of data synchronizations, only the difference files can be synchronized, therefore, the hash value corresponding to the source file and the corresponding hash value of the target file can be continuously compared, and the difference files can be quickly determined in turn; and the thread has the characteristics of fast response speed and support for parallel execution of tasks, so that according to the data synchronization strategy of the thread, the target data in the source file of at least one difference file pair can be quickly synchronized to the corresponding target file. Therefore, the efficiency of comparing the data in the file is improved and the consistency of the synchronized data can be guaranteed, thereby also improving the speed of data synchronization.

[0035] First, the terms involved in this application are explained:

[0036] Hash value: also known as hash value, usually represented by a short string of random letters and numbers. It is a binary value obtained by a hash algorithm for a set of input information of arbitrary length.

[0037] Hash algorithm: converts input data of any length through a hash function to obtain an output value of fixed length. This output value is usually called a hash value, which is used to identify the uniqueness and integrity of the data.

[0038] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0039] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data synchronization method depends, the specific application environment architecture or specific hardware architecture is described herein. Figure 1 , Figure 1The diagram is an application scenario of a data synchronization method, in which a server device 101 where a source file is located, a server device 102 where a target file is located, and a data synchronization device 103 may be included. The server device 101 where a source file is located is a server device for storing a source file of target data, and the server device 102 where a target file is located is a server device for storing a target file corresponding to the target data; the data synchronization device 103 is in communication connection with the server device 101 and the server device 102.

[0040] Optionally, the data synchronization device 103 may also be integrated in the server device 101 where the source file is located or in the server device 102 where the target file is located.

[0041] Specifically, the data synchronization device 103 monitors the data change of the server device 101 where the source file is located and determines that the data synchronization condition is met; the data synchronization device 103 determines a source storage path corresponding to at least one target data to be synchronized and a target storage path pre-configured for the target data. The source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized. If the data synchronization device 103 determines that the target file is included in the target directory, it sends a first directory hash value acquisition instruction carrying the source directory name information to the server device 101 where the source file is located, and sends a second directory hash value acquisition instruction carrying the target directory information to the server device 102 where the target file is located; after the server device 101 where the source file is located receives the first directory hash value acquisition instruction, it acquires the first directory hash value of the data stored in the source directory where the target data is located based on the source directory name and sends it to the data synchronization device 103; similarly, after the server device 102 where the target file is located receives the second directory hash value acquisition instruction, it acquires the second directory hash value of the data stored in the target directory based on the target directory name and sends it to the data synchronization device 103; the data synchronization device 103 responds to the first directory hash value being different from the second directory hash value, and continues to divide The server device 101 where the source file is located sends a command for obtaining a first file hash value and the server device 102 where the target file is located sends a command for obtaining a second file hash value; the server device 101 where the source file is located obtains the first file hash value of the source file data where each target data is located and sends it to the data synchronization device 103; the server device 102 where the target file is located obtains the second file hash value of the target file data corresponding to each target data and sends it to the data synchronization device 103; after receiving each first file hash value and each second file hash value, the data synchronization device determines at least one difference file pair; the data synchronization device 103 synchronizes the target data in the source file of at least one difference file pair in the server device 101 where the source file is located to the target file corresponding to the server device 102 where the target file is located through network transmission according to the data synchronization strategy of the thread. This can improve the comparison efficiency of the data in the file and ensure the consistency change of the synchronized data, thereby also realizing the rapid synchronization of the data.

[0042] Figure 2 Schematic diagram of the data synchronization method provided in the embodiment of the present application Figure 1 ,like Figure 2 As shown, an embodiment of the present application provides a data synchronization method, and the method is described in detail as follows:

[0043] S201: In response to determining that the data synchronization conditions are met, determine a source storage path corresponding to at least one target data to be synchronized and a target storage path pre-configured for the target data; the source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized.

[0044] Among them, the data synchronization condition refers to the condition used to determine whether data synchronization is required; optionally, the data synchronization condition can be set to determine whether the data change is caused by a change in business requirements. Among them, the business requirements can be pre-set according to the needs, including: data version upgrade business requirements, data rollback business requirements, data backup business requirements, or other requirements, which are not limited in this embodiment. Optionally, the data synchronization condition can also be to determine whether the changed data is monitored, etc., and the method for judging the data synchronization condition is not limited in this embodiment.

[0045] The target data refers to the data to be synchronized.

[0046] Among them, the source storage path refers to the address of the file corresponding to the target data before synchronizing the target data. Optionally, the address of the file corresponding to the target data can be composed of two slash symbols, a directory name and a file name. Exemplarily, the slash symbol can use " / ". The directory name can be defined according to the function and expressed in English. For example, the directory used to store important data can be defined as "home"; the file name can also be defined according to the data function and expressed in English. For example, for session data, the folder can be named "session", then " / home / session" represents a path; there can be multiple folder naming methods and path representation forms, which are not specifically limited in this embodiment. Other file addresses, directory naming methods, and folder naming methods in the method of the present application are characterized in accordance with the methods in this embodiment, and the following will not be repeated.

[0047] The target storage path refers to the file address corresponding to the target data after storage synchronization. To determine a target storage path, it is necessary to pre-set the storage directory corresponding to the target storage path. Specifically, a target directory name can be set, and then the target file name to be synchronized and the storage directory are combined according to the representation method of the file address. The result of combining the target file name and the storage directory according to the file address can be used as the target storage path. The target storage path can also be generated in other ways, which is not specifically limited in this embodiment.

[0048] The target directory name refers to the name of the folder where the target file is stored.

[0049] The target file refers to the folder where the data to be synchronized is stored.

[0050] The source file refers to the file storing the target data before synchronizing the target data.

[0051] The source directory name refers to the name of the folder that stores the source files.

[0052] The target file name to be synchronized is the name of the file where the target data to be synchronized is located.

[0053] Optionally, a file monitoring system and a monitoring thread may be configured in advance in a server corresponding to the storage source directory to determine whether a data synchronization condition is met and generate corresponding file name information.

[0054] Specifically, first, when data changes, the file monitoring system will record the source storage path corresponding to the changed data, the corresponding business requirements of the changed data, and the changed data type through its internal monitoring mechanism and store them in the log. Then, in the monitoring thread, by checking whether the access log is updated, it can be determined whether there is a change record and obtain the corresponding business requirements of the changed data from the log; and the business requirements corresponding to the changed data are compared with the business requirements set in the data synchronization conditions. When the corresponding business requirements of the changed data belong to the business requirements set in the data synchronization conditions, it is determined that the data synchronization conditions are met.

[0055] Specifically, when it is determined that the data synchronization conditions are met, the monitored change data is used as the target data, and the source storage path corresponding to the change data can be obtained from the log by continuing to access the log as the source storage path corresponding to the target data to be synchronized; and then the source storage path corresponding to the target data to be synchronized is parsed by the path parsing method to obtain the corresponding target file name to be synchronized. Furthermore, the corresponding target file name to be synchronized is combined with the pre-configured target directory according to the combination of file addresses, so as to obtain the pre-configured target storage path of the target data. Among them, the path parsing method can be a regular expression, a path parsing command in a preset operating system, or other parsing methods, which are not limited in this embodiment.

[0056] Optionally, you can continue to access the log to obtain the change data type and convert the data operation type into the form of the corresponding target data operation identifier. The change data type can be an addition type, a change type, and a deletion type of data; the change data identifier refers to a symbol used to identify the data operation type. For example, if it is a deletion type, it can be represented as "del"; if it is an addition type, it can be represented as "add"; if it is a modification type, it can be represented as "mod"; it can also be represented in other forms, which are not specifically limited in this embodiment.

[0057] After obtaining the above data, the target file name to be synchronized, the source storage path corresponding to the target data to be synchronized, the target storage path and the target data operation identifier are stored in the change information storage table in sequence. The change information storage table is a table for storing information related to the target data, which can store path information, file name information and target data operation identifier, which is not specifically limited in this embodiment.

[0058] S202: If it is determined that the target directory includes the target file, a first directory hash value of the source directory storing data where each target data is located is obtained based on the source directory name, and a second directory hash value of the target directory storing data is obtained based on the target directory name.

[0059] The first directory hash value refers to the hash value corresponding to the data content in the source directory.

[0060] The second directory hash value refers to the hash value corresponding to the data content in the target directory.

[0061] It can be understood that the hash value refers to a value calculated by a hash algorithm. The first directory hash value and the second directory hash value can both be calculated by a hash algorithm. The hash algorithm may include a message digest algorithm or a secure hash algorithm, etc., which is not specifically limited in this embodiment.

[0062] Specifically, to determine whether the target file is included in the target directory, the target file can be directly compared with each file name in the target directory for confirmation; or the target file can be compared with the first letter of each file name in the target directory for confirmation; or other methods can be used, which are not limited in this embodiment. Exemplarily, all file paths of the target directory can be obtained based on the operating system path reading command and the file path can be parsed by the path parsing method to parse out all file names in the target directory, and all file names parsed out of the target directory are stored in the file name storage table; then the fields of each file name corresponding to the target directory are compared with the file name field with the same first letter of the target file name to be synchronized obtained in step S201, and the comparison result is reported to the server. Among them, the comparison result can be preset to 1 or 0. If the comparison result is 1, it means that the target file is included in the target directory; if the comparison result is 0, it means that the target file is not included in the target directory. Among them, the comparison result can also be expressed in other forms, which are not specifically limited in this embodiment.

[0063] Specifically, when it is determined that the target file is included in the target directory, the server can use the conventional path combination form, and the combination of the preset directory path address is a combination of "slash" and the directory name. For example, assuming that the directory name is "tmp", the directory address is represented as " / tmp"; thus, the source file name and the target file name to be synchronized can be obtained from the change information storage table, and the source directory path and the target directory path corresponding to the source directory and the target directory can be matched according to the conventional path combination form. Then the source directory path and the target storage path are respectively input into the hash algorithm, and the hash algorithm can traverse all the file contents in the directory file in sequence, and can calculate the first directory hash value of the source directory storage data where the target data is located and the second directory hash value of the target directory storage data.

[0064] Optionally, the hash value of each file in the source directory and the target directory can be calculated separately, and then the hash values ​​of all files in the same directory can be concatenated to obtain the first directory hash value of the source directory storing the target data and the second directory hash value of the target directory storing the target data; the first directory hash value and the second directory hash value can also be calculated in other ways, which are not specifically limited in this embodiment.

[0065] S203: In response to the first directory hash value being different from the second directory hash value, obtaining the first file hash value of the source file data where each target data is located, and obtaining the second file hash value of the target file data corresponding to each target data.

[0066] The first file hash value refers to a hash value calculated by applying a hash algorithm to the source file data content.

[0067] The second file hash value refers to a hash value calculated by applying a hash algorithm to the source file data content.

[0068] Optionally, the first file hash value of the source file data where each target data is located and the second file hash value of the target file data corresponding to each target data may be pre-stored in a table.

[0069] Specifically, a hash algorithm can be used to sequentially calculate the first file hash value corresponding to the data content in each source file; and the first file hash value of each source file and its corresponding source storage path and the target file name to be synchronized are sequentially stored in a source file hash value storage table. The source file hash value storage table refers to a table for storing source file hash values ​​and data associated with the source file hash values.

[0070] Similarly, a hash algorithm can be used to sequentially calculate the second file hash value corresponding to the data content in each target file; and the second file hash value of each target file and its corresponding target storage path and the target file name to be synchronized are sequentially stored in the target file hash value storage table. The target file hash value storage table refers to a table for storing target file hash values ​​and data associated with source file hash values.

[0071] It is understandable that there are many ways to store and obtain the hash value. For example, the hash value can also be stored in a dictionary, which is not specifically limited in this example. The dictionary is a form of data storage.

[0072] Specifically, the first directory hash value obtained in step 203 is compared with the second directory hash value. When the first directory hash value is different from the second directory hash value, it indicates that there is a difference between the first directory and the second directory, and the first file hash value of the source file data where each target data is located is obtained respectively, and the second file hash value of the target file data corresponding to each target data is obtained. Specifically, the corresponding target file name to be synchronized can be obtained from the change information storage table in sequence, and the target file name in the source file hash value storage table can be matched to obtain the first file hash value and source storage path corresponding to the target file name to be synchronized; then the corresponding target file name to be synchronized is matched with the target file name to be synchronized in the target file hash value storage table to obtain the second file hash value and target storage path corresponding to the target file name to be synchronized, and the first file hash value, the second file hash value, the source storage path and the target storage path are stored in the hash value comparison table; until all the first file hash values ​​and second file hash values ​​corresponding to the target file names to be synchronized are matched; wherein the hash value comparison table is a table of hash values ​​to be stored and compared and their associated information.

[0073] S204: Determine at least one difference file pair according to each first file hash value and each second file hash value, where the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value.

[0074] Specifically, the first file hash value and the second file hash value corresponding to the same target file name in the hash value comparison table are compared in turn. If the compared first file hash value and the second file hash value are different, it indicates that there is a file difference between the source file corresponding to the first file hash value and the target file corresponding to the second file hash value, and the source file and the target file are used as a set of difference file pairs. The corresponding source storage path and target storage path can be queried from the hash value comparison table through the first file hash value or the second file hash value of the file pair, and the first file hash value or the second file hash value, the source storage path and the target storage path are stored as a difference file pair information in the file pair path storage table. Among them, the file pair path storage table is a table for storing difference file pair information.

[0075] S205: Synchronize target data in a source file of at least one difference file pair to a corresponding target file according to a data synchronization strategy of the thread.

[0076] The data synchronization strategy refers to a preset method for performing data synchronization. Exemplarily, the data synchronization strategy can be configured based on threads.

[0077] Optionally, the number of threads can be set according to the self-configured data synchronization time. Specifically, the number of bytes of target data that can be processed by a single thread can be tested in advance within a preset time, and the number of bytes of target data that can be processed by this single thread can be used as the threshold of the total number of bytes processed by the single thread; then, the source storage paths corresponding to the source files in the difference file pair are obtained in turn in the file pair path storage table, and the number of bytes occupied by the difference file pair is queried according to the source storage paths corresponding to the source files of the difference file pair and the file byte query command of the operating system, and the number of bytes occupied by all difference file pairs is added as the number of bytes occupied by the source files in the difference file pair; when the number of bytes occupied by the source file in a difference file pair is less than the threshold of the total number of bytes processed by a single thread, it indicates that the target data can be synchronized using one thread; when the number of bytes occupied by the source file in at least one difference file pair is greater than the threshold of the total number of bytes processed by a single thread, it indicates that the speed of synchronizing the target data using one thread within the preset time is slow, and multiple threads are needed to synchronize the target data to increase the speed of data synchronization. Therefore, the number of bytes occupied by the source file in at least one difference file pair divided by the total byte number threshold for single-thread processing can be used as the number of threads to be run, and the target data in the source file of at least one difference file pair is synchronized to the corresponding target file in a multi-threaded manner. The data synchronization method can also be a database operation or other methods, which are not specifically limited in this embodiment.

[0078] It can be understood that there may be multiple ways to determine data synchronization based on threads, and no specific limitation is made in this embodiment.

[0079] In this embodiment, in response to determining that the data synchronization condition is met, a source storage path corresponding to at least one target data to be synchronized and a target storage path pre-configured for the target data are determined; the source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized; if it is determined that the target file is included in the target directory, a first directory hash value of the source directory storage data where each target data is located is obtained based on the source directory name, and a second directory hash value of the target directory storage data is obtained based on the target directory name; in response to the first directory hash value being different from the second directory hash value, a first file hash value of the source file data where each target data is located is obtained, and a second file hash value of the target file data corresponding to each target data is obtained; at least one difference file pair is determined based on each first file hash value and each second file hash value. , the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value; according to the data synchronization strategy of the thread, the target data in the source file of at least one difference file pair is synchronized to the corresponding target file; therefore, by directly comparing the first directory hash value with the second directory hash value, it is possible to accurately determine whether there is a difference between the source directory and the target directory; when there is a difference between the source directory and the target directory, each first file hash value and each second file hash value are further obtained, and then the difference file pair can be quickly and accurately determined by comparing the first file hash value and each second file hash value; after completing the difference file pair confirmation, the target data in the source file of this difference file pair can also be quickly synchronized to the corresponding target file based on the data synchronization strategy of the thread, so as to achieve fast synchronization of the difference file. Therefore, it is possible to improve the comparison efficiency of the data in the file and ensure the consistency change of the synchronized data, thereby also improving the synchronization speed of the data.

[0080] As an optional implementation, obtaining a first directory hash value of the source directory storing data where each target data is located based on the source directory name, and obtaining a second directory hash value of the target directory storing data based on the target directory name, includes:

[0081] At least one source file is obtained based on the source directory name and the source storage path, and a first file hash value of each source file is calculated based on a preset hash algorithm; at least one target file is obtained based on the target directory name and the target storage path, and a second file hash value of each target file is calculated based on a preset hash algorithm; the first file hash values ​​are concatenated, and the result of the concatenation of the first file hash values ​​is input into the preset hash algorithm to obtain a first directory hash value; the second file hash values ​​are concatenated, and the result of the concatenation of the second file hash values ​​is input into the preset hash algorithm to obtain a second directory hash value.

[0082] The preset hash algorithm refers to an algorithm set for hash value calculation, which may be an information digest algorithm or other algorithms, and is not specifically limited in this embodiment.

[0083] Specifically, the information digest algorithm can locate the corresponding source file or target file through the target storage path or the source storage path and access the data in the source file or the target file, and group and process the data in the accessed source file or the target file according to specific rules. The data in the source file or the target file is grouped, and the grouped data is subjected to bit operations, addition, multiplication, and modular operations to generate multiple results. Each time a result is obtained, it is bitwise ANDed with the previous result until the bitwise ANDed calculation with the last result is completed, thereby generating the first file hash value of the source file or the second file hash value of the target file.

[0084] Specifically, the source storage paths and their corresponding target storage paths corresponding to the current source directory name can be matched from the change information storage table according to the current source directory name, and then the source storage paths can be input into the preset hash algorithm in turn. In the preset hash algorithm, the source file positions can be located by accessing the source storage paths, and the corresponding source file contents can be accessed. The source file contents are then input into the hash function to obtain the first file hash values ​​of the source files; further, the first file hash values ​​of the source files are concatenated to obtain a string related to the first file hash values ​​of the source files. Similarly, the target storage paths are input into the preset hash algorithm in turn. In the preset hash algorithm, the target file positions can be located by accessing the target storage paths, and the corresponding target file contents can be accessed. The target file contents are then input into the hash function to obtain the second file hash values ​​of the target files; further, the second file hash values ​​of the target files are concatenated to obtain a string related to the second file hash values ​​of the target files. Finally, the character string related to the first file hash value of each source file is input into the preset hash algorithm to obtain the first directory hash value of the source directory storage data where the target data is located; similarly, the character string related to the second file hash value of each target file is input into the preset hash algorithm to obtain the second directory hash value of the target directory storage data.

[0085] Optionally, the process of calculating the first directory hash value of the source directory storing the target data and the second directory hash value of the target directory storing the target data according to the above method can also be as follows: Figure 3As shown, specifically: the main synchronization process module is the main process module that completes the entire data synchronization. The main process module first calls the file hash value calculation module, which is used to obtain the path of each source file from the traversal path module. The traversal path module sends the corresponding source file path to the hash value calculation module, and then the file hash value calculation module locates the content of each source file according to each source file path, and calculates the hash value corresponding to each source file; the hash value calculation module then sends the hash value corresponding to each source file to the combined hash value module, the combined hash value module receives the hash value corresponding to each source file, and splices the hash value results corresponding to each source file; the combined hash value module then sends the hash value results corresponding to each source file to the hash value calculation module, and the hash value calculation module inputs the hash value results corresponding to each source file into the preset hash algorithm and outputs the first directory hash value of the source directory where the target data is located. In this way, repeating the above process can obtain the second directory hash value of the target directory storage data.

[0086] Optionally, when the first directory hash value and the second directory hash value or the first file hash value and the second file hash value need to be obtained from two devices, the first device and the second device can communicate through a standard data serialization and remote procedure call protocol (External Data Representation, referred to as XDR), such as Figure 4 As shown. The first device can send a command to obtain the second hash value to the second device and carry the target directory name and the target directory path. The second device calculates the second directory hash value according to the above method for obtaining the second directory hash value and sends it to the first device through data serialization and remote procedure call protocol. The first device is the device corresponding to the source file, which can be Figure 1 The server device 101 where the source file is located, and the second device is the device corresponding to the target file, which can be Figure 1 The server device 102 where the target file is located is shown. Therefore, data synchronization of multiple devices can also be achieved through standard data serialization and remote procedure call protocol transmission methods. Among them, standard data serialization and remote procedure call protocol refers to a network transmission protocol used to achieve cross-platform data exchange.

[0087] It can be understood that, since the dictionary has strong flexibility, is easy to manage and has a simple data structure, it can also reduce the storage space occupied and has the advantage of fast access speed. Therefore, in order to improve the subsequent reading speed of the hash value corresponding to the file, after calculating the first file hash value of the source file and the second file hash value of the target file, the dictionary storage hash value can also be used. Specifically: it is also possible to first match the source directory name corresponding to the current source directory name from the change information storage table according to the current source directory name, and obtain the source storage path corresponding to the current source directory and its corresponding target storage path; the source storage path corresponding to the current source directory and its associated matching target storage path form a file path pair; wherein the file path pair refers to the path pair formed by the source storage path corresponding to the current source directory and its associated matching target storage path. Then the file path pair formed by the source storage path corresponding to the current source directory and its associated matching target storage path is standardized, and the file path pair formed by the source storage path corresponding to the current source directory and its associated matching target storage path is sorted in ASCII alphabetical order, so as to ensure that the source storage path corresponding to the sorted source directory and its target storage path always present a matching relationship. Among them, the file path standardization processing can be to remove slashes in the path, remove redundancy in the path, etc., or other methods, which are not limited in this embodiment. Optionally, the paths can also be sorted in other ways, which are not specifically limited in this embodiment. Among them, ASCII is a standard code for information exchange, which uses a specified 7-bit or 8-bit binary number combination to represent 128 or 256 possible characters, which mainly include English letters, numbers, punctuation marks, and some special symbols.

[0088] Then, create a dictionary that stores hash values ​​corresponding to the source storage path and a dictionary that stores hash values ​​corresponding to the target storage path; wherein, the dictionary creation can be performed according to commonly used dictionary creation commands, such as the "dict" command, or other dictionary creation commands, which are not specifically limited in this embodiment. A dictionary usually consists of a primary key and multiple key pair values, wherein the primary key identifier is used to identify the unique identifier of the dictionary value, and the key pair value refers to the data associated with the primary key. Because each hash value is unique, the first file hash value of the source file can be used as the primary key, and its corresponding source storage path can be used as the key pair value, so that a dictionary that stores hash values ​​corresponding to the source storage path can be created. Similarly, the second file hash value of the target file can be used as the primary key, and its corresponding target storage path can be used as the key pair value, so that a dictionary that stores hash values ​​corresponding to the target storage path can be created.

[0089] Finally, the file path pairs formed by the source storage path corresponding to the current source directory and its matching target storage path are sequentially input into the preset hash algorithm to obtain the first file hash value of the source file and the second file hash value of the target file, and the first file hash value of the source file and its input source storage path are stored in the dictionary storing the hash values ​​corresponding to the source storage path; similarly, the second file hash value of the target file and its input target storage path are stored in the dictionary storing the hash values ​​corresponding to the source storage path until all file path pairs are traversed. Therefore, the hash value can be obtained by accessing the dictionary, which can be used to determine the difference file through the hash value to create a file link later.

[0090] Among them, Figure 5 As shown, the main process module first calls the path acquisition module to obtain each source storage path; the path acquisition module obtains the corresponding source storage paths from the source directory in turn and sends them to the main process module, the main process module receives each source storage path and sends each source storage path to the hash storage dictionary establishment module, the hash storage dictionary module establishes a hash dictionary corresponding to the source storage path according to each source storage path, and sends the hash dictionary corresponding to the source storage path to the main process module; the main process module then sends the hash dictionary corresponding to the source storage path to the hash value calculation module, the hash value calculation module calculates the hash value corresponding to each source file in turn and stores the calculated hash value in the hash dictionary corresponding to the source storage path; repeating the above process can also obtain the hash value corresponding to each target file and store the calculated hash value in the hash dictionary corresponding to the target storage path.

[0091] In this embodiment, at least one source file is obtained based on the source directory name and the source storage path, and the first file hash value of each source file is calculated based on a preset hash algorithm; at least one target file is obtained based on the target directory name and the target storage path, and the second file hash value of each target file is calculated based on the preset hash algorithm; the first file hash values ​​are spliced ​​and the result of the splicing of the first file hash values ​​is input into the preset hash algorithm to obtain a first directory hash value; the second file hash values ​​are spliced ​​and the result of the splicing of the second file hash values ​​is input into the preset hash algorithm to obtain a second directory hash value; the hash value is unique, and by calculating the hash value of the directory, it is possible to quickly verify whether there is a data difference between the source directory and the target directory; wherein, the file hash values ​​are spliced ​​and then the hash calculation is performed again, so that the hash value of the directory is represented as a shorter string, thereby saving the space for storing the hash value and improving the comparison speed when the target hash value is compared later.

[0092] As an optional implementation, determining at least one difference file pair according to each first file hash value and each second file hash value includes:

[0093] Compare each first file hash value with the corresponding second file hash value to obtain at least one target comparison result, where the target comparison result is a comparison result where the first file hash value is different from the second file hash value; determine the source file and the target file corresponding to each target comparison result as each difference file pair.

[0094] Specifically, a comparison result identification field is added to the hash value comparison table; the first file hash value corresponding to the same target file name in the hash value comparison table is compared with the second file hash value in turn, and then the comparison result identification is added to the position of the comparison result identification field corresponding to the same target file name in the hash value comparison table. The comparison result identification refers to a symbol used to identify the difference file. Exemplarily, the difference files can be identified as 0 and 1; then 0 identifies the difference file pair, and 1 represents the difference file pair. The 0 and 1 values ​​corresponding to the comparison result identification field in the hash value comparison table are the target comparison results. Therefore, the corresponding difference file pair can be quickly found by reading the difference file identification in the hash value comparison table.

[0095] In this embodiment, each first file hash value is compared with the corresponding second file hash value to obtain at least one target comparison result, the target comparison result is a comparison result where the first file hash value is different from the second file hash value; the source file and the target file corresponding to each target comparison result are determined as each difference file pair; by directly comparing each first file hash value with the corresponding second file hash value, it is possible to directly confirm whether there is a difference between the source file and the target file, and by identifying the target comparison result, the difference file can be quickly found in multiple files. Therefore, this method of comparing each first file hash value with the corresponding second file also improves the efficiency of file comparison.

[0096] As an optional implementation, according to the data synchronization strategy of the thread, synchronizing the target data in the source file of at least one difference file pair to the corresponding target file includes:

[0097] The target number of execution threads is determined according to the preset single-byte processing time, the total number of bytes occupied by the target data and the preset data synchronization time; each target data is divided into multiple target data sets based on the number of bytes processed by a single thread; the number of bytes processed by a single thread is determined according to the preset data synchronization time and the single-byte processing time; each target execution thread corresponding to the target number of threads is used to obtain the target data from the corresponding target data set, and a database operation statement is used to synchronize the target data from the target data in the source file of the difference file pair to the corresponding target file.

[0098] The single-byte processing time is the processing time of a single thread for a byte. The time for a single thread to process a byte of data can be tested and set as a preset single-byte processing time. A single-byte processing time can also be obtained by other methods, which are not specifically limited in this embodiment.

[0099] The total number of bytes occupied by the target data refers to the sum of the number of bytes occupied by all target data in the memory storage space.

[0100] Among them, the preset data synchronization time refers to the time required to synchronize the target data, which is a value set according to user needs. Exemplarily, it can be set to 10 seconds, or it can be set to other times. In this embodiment, no specific limitation is made. Optionally, the single-byte processing time and the preset data synchronization time can be stored in the thread configuration table. The thread configuration table includes a single-byte processing time field and its corresponding single-byte processing time; it also includes a preset data synchronization time field and its preset data synchronization time.

[0101] The target number of execution threads is the number of threads required to execute synchronization target data.

[0102] The number of bytes processed by a single thread is the number of bytes occupied by data processed by the single thread during a preset data synchronization time.

[0103] Specifically, the source storage path corresponding to the source file in the difference file pair can be obtained from the change information storage table, and based on the file byte query command of the operating system, the total number of bytes occupied by each source file corresponding to each source storage path is sequentially superimposed according to the total number of bytes occupied by each source file, so as to obtain the total number of bytes occupied by the target data. Then, the single-byte processing time field is matched from the thread configuration table and its corresponding single-byte processing time is obtained, and the preset data synchronization time is divided by the preset single-byte processing time as the number of bytes processed by a single thread; then, the total number of bytes occupied by the target data is divided by the number of bytes processed by a single thread as the target number of execution threads; then, the data of the files corresponding to all source storage paths are accessed and stored in the data table; and divided into multiple target data sets according to the number of bytes processed by a single thread; and synchronization threads are allocated to each target data set according to the target number of execution threads through the thread pool, and each target data is sequentially synchronized to the corresponding target file in the source file of the difference file pair using database operation statements in the thread.

[0104] Among them, the corresponding target data operation identifier can be identified from the change information storage table. If the identified change data operation identifier is "del", the target data in the source file of the difference file pair and the target data in the target file are deleted using a database delete statement. If the identified target data operation identifier is "add", the target data in the source file of the difference file pair is added to the corresponding target file using a database add statement. If the identified target data identifier is "mod", the target data in the source file of the difference file pair is changed to the corresponding target file using a database add statement.

[0105] In this embodiment, the target execution thread number is determined according to the preset single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time; each target data is divided into multiple target data sets based on the number of bytes processed by the single thread; the number of bytes processed by the single thread is determined according to the preset data synchronization time and the single-byte processing time; each target execution thread corresponding to the target thread number is used to obtain the target data from the corresponding target data set, and the target data is synchronized from the target data in the source file of the difference file pair to the corresponding target file using a database operation statement. The target execution thread number set can be quickly and effectively allocated to the target through the single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time, thereby saving the space for thread allocation and ensuring that each target data set has a matching thread; further, the target data set is synchronized based on the target execution thread, thereby improving the efficiency of data synchronization.

[0106] As an optional implementation manner, if it is determined that the target file is not included in the target directory, the method further includes:

[0107] Create a target file based on the target storage path; synchronize the target data in the source file to the corresponding target file according to the thread's data synchronization strategy.

[0108] Specifically, when it is determined that the target file is not included in the target directory, it indicates that there is no data backup of the target file in the target directory; a create command for the target file is generated based on the create file command in the operating system and the target file name; illustratively, a combination of the create file command "mkdir" in the operating system and the target file name can be used as the create command for the target file, or it can be created in other forms, which is not specifically limited in this embodiment.

[0109] After the target file is created, the corresponding processing thread is assigned to the target data in the source file according to the data synchronization strategy; after the corresponding processing thread is assigned to the target data, the target data in the source file can be synchronized to the target file using a database operation statement. The data synchronization strategy can be any specific scheme in this application, and will not be repeated in this embodiment.

[0110] In this embodiment, if it is determined that the target file is not included in the target directory, the target file is created based on the target storage path; according to the data synchronization strategy of the thread, the target data in the source file is synchronized to the corresponding target file; when the target file is not included in the target directory, the thread can also be used to synchronize the data, which can improve the efficiency of synchronization of the target data.

[0111] As an optional implementation, it also includes:

[0112] In response to the first file hash value being the same as the second file hash value, the target file is deleted based on the target storage path; a preset file operation command is used to generate a symbolic link between the target file and the source file based on the source storage path, and the symbolic link is used to access the data in the source file when the symbolic link is triggered.

[0113] The preset file operation command refers to a command for generating a symbolic link between a source storage path and a target file corresponding to a source file, which may be a combination of an operation command "ln -s" and a source storage path and a target storage path.

[0114] It can be understood that when the first file hash value is the same as the second file hash value, it indicates that the source file is consistent with the file in the target data; then the data only needs to be saved in one of the folders, thereby saving data storage space.

[0115] Specifically, the hash value comparison table in step 203 can be used to obtain the first file hash value and the second file hash value corresponding to the target file name to be synchronized in sequence and compare them; if the comparison results of the first file hash value and the second file hash value are the same, the corresponding hash value field is accessed in the hash value comparison table according to the same hash value, and the source storage path and the target storage path corresponding to the same hash value field are obtained; then the location of the text file is located based on the target storage path, and the target file is deleted using the delete command in the operating system; the source storage path and the target storage path are combined with the operation command to generate a preset file command and the preset file command is executed until all the first file hash values ​​and the second file hash values ​​are compared. In this way, a symbolic link corresponding to the source file of the target file with the same hash value can be generated, and the target file data can be accessed by accessing the symbolic link. The symbolic link corresponding to the source file of the target file can also be generated in other forms, which are not limited to this embodiment.

[0116] Optionally, when the first file hash value and the second file hash value are stored in a dictionary, Figure 5As shown, the main process module obtains the corresponding hash values ​​from the dictionary corresponding to the source storage path and the dictionary of hash values ​​corresponding to the target storage path and compares them in turn. If the comparison result is bad, the main process module sends a message to the symbol creation module to delete the target file and create a link. The symbol creation module generates a symbolic link between the target file with the same hash value and the source file.

[0117] In this embodiment, in response to the first file hash value being the same as the second file hash value, the target file is deleted based on the target storage path; a preset file operation command is used to generate a symbolic link between the target file and the source file based on the source storage path, and the symbolic link is used to access the data in the source file when the symbolic link is triggered. A symbolic link between the target file and the source file is generated through the source storage path instead of storing the target file, which can save storage space for the data.

[0118] As an optional implementation, it also includes:

[0119] In response to the first directory hash value being the same as the second directory hash value, synchronization of the source file corresponding to the target data to the corresponding target file is prohibited.

[0120] When the hash value of the first directory is the same as the hash value of the second directory, it indicates that there are no different files between the source directory and the target directory. Therefore, when the data of the source directory and the target directory differ, a synchronization prohibition mark is received inside the server. When the synchronization prohibition mark appears, the data synchronization process is automatically exited inside the server, thereby prohibiting the synchronization of the source file corresponding to the target data to the corresponding target file.

[0121] In this embodiment, in response to the first directory hash value being the same as the second directory hash value, synchronization of the source file corresponding to the target data to the corresponding target file is prohibited; by comparing the first directory hash value with the second directory hash value, it is possible to directly and quickly determine whether there is a difference between the target directory and the source directory; when the first directory hash value is the same as the second directory hash value, it indicates that there is no difference between the source directory and the target directory, and the data synchronization can be directly stopped. Therefore, it is possible to quickly determine whether to continue data synchronization, thereby reducing the continued calculation of the data synchronization process.

[0122] Optionally, when it is determined that the data synchronization conditions are met, before executing the data synchronization process, the server can also dynamically adjust the priority of the process according to the importance and occupancy of each process, and allocate processors and processes for execution to the data synchronization process, so as to reasonably allocate processes to ensure that the data synchronization process can be executed efficiently. Specifically, the system will set different priorities for the processes in advance, and the priorities can be divided into multiple levels, such as low level, lower than normal level, normal level, higher than normal level and high level. When the process is running, the system will monitor the running status of each process in real time, including CPU occupancy, memory usage, I / O operation frequency, etc.; when there is a new task that needs to run the process, the system will analyze the current resource occupancy. If the data synchronization process needs to quickly complete the transmission of a large amount of data, the system may increase the priority to a higher than normal level or a high level to ensure that it obtains sufficient processor resources.

[0123] Figure 6 The flowchart of the data synchronization method provided in one embodiment of the present application shows the process of data synchronization calculation. Specifically, in response to determining that the data synchronization condition is met, it is determined whether the target data exists. If not, it ends and exits the data synchronization process; if it exists, it determines at least one source storage path corresponding to the target data to be synchronized and the target storage path pre-configured for the target data; and allocates a processor and a process for the data synchronization task, and starts the calculation of the data synchronization process in the process.

[0124] Determine whether the target file is included in the target directory. If the target file exists in the target directory, create the target file based on the target storage path; synchronize the target data in the source file to the corresponding target file according to the data synchronization strategy of the thread. If the target file exists in the target directory, obtain the first directory hash value and the second directory hash value respectively. Determine whether the first directory hash value is the same as the second directory hash value. If they are the same, it is prohibited to synchronize the source file corresponding to the target data to the corresponding target file. If the first directory hash value is not the same as the second directory hash value, obtain the hash value of each first file and the hash value of the second file respectively. Determine whether the hash value of each first file is the same as the hash value of the second file. If it is determined that the hash value of each first file is not the same as the hash value of the second file, determine at least one difference file pair; and synchronize the target data in the source file of at least one difference file pair to the corresponding target file according to the data synchronization strategy of the thread. If it is determined that the hash value of each first file is the same as the hash value of the second file, use the preset file operation command and generate a symbolic link of the target file corresponding to the source file based on the source storage path.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0126] Figure 7 This is a schematic diagram of the structure of the data synchronization device provided in the embodiment of the present application. Figure 7 As shown, the embodiment of the present application further provides a data synchronization device, which is located in the data synchronization device. The data synchronization device 70 provided in this embodiment includes: a determination module 701, an acquisition calculation module 702, and a synchronization module 703.

[0127] Determination module 701 is used to determine, in response to determining that the data synchronization conditions are met, a source storage path corresponding to at least one target data to be synchronized and a target storage path pre-configured for the target data; the source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized.

[0128] The acquisition module 702 is used to acquire the first directory hash value of the source directory storing data where each target data is located based on the source directory name if it is determined that the target directory includes the target file, and acquire the second directory hash value of the target directory storing data based on the target directory name.

[0129] The acquisition module 702 is used to acquire the first file hash value of the source file data where each target data is located, and acquire the second file hash value of the target file data corresponding to each target data in response to the first directory hash value being different from the second directory hash value.

[0130] The determination module 701 is used to determine at least one difference file pair according to each first file hash value and each second file hash value, where the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value.

[0131] Synchronization module 703, used to synchronize the target data in the source file of at least one difference file pair to the corresponding target file according to the data synchronization strategy of the thread

[0132] Optionally, the acquisition module 702, when acquiring the first directory hash value of the source directory storing data where each target data is located based on the source directory name, and acquiring the second directory hash value of the target directory storing data based on the target directory name, is specifically used to:

[0133] At least one source file is obtained based on the source directory name and the source storage path, and a first file hash value of each source file is calculated based on a preset hash algorithm; at least one target file is obtained based on the target directory name and the target storage path, and a second file hash value of each target file is calculated based on a preset hash algorithm; the first file hash values ​​are concatenated, and the result of the concatenation of the first file hash values ​​is input into a preset hash algorithm to obtain a first directory hash value; the second file hash values ​​are concatenated, and the result of the concatenation of the second file hash values ​​is input into a preset hash algorithm to obtain a second directory hash value.

[0134] Optionally, when determining at least one difference file pair according to each first file hash value and each second file hash value, the determination module 702 is specifically configured to:

[0135] Compare each first file hash value with the corresponding second file hash value to obtain at least one target comparison result, where the target comparison result is a comparison result where the first file hash value is different from the second file hash value; determine the source file and the target file corresponding to each target comparison result as each difference file pair.

[0136] Optionally, the synchronization module 703, when synchronizing the target data in the source file of at least one difference file pair to the corresponding target file according to the data synchronization strategy of the thread, is specifically configured to:

[0137] The target number of execution threads is determined according to the preset single-byte processing time, the total number of bytes occupied by the target data and the preset data synchronization time; each target data is divided into multiple target data sets based on the number of bytes processed by a single thread; the number of bytes processed by a single thread is determined according to the preset data synchronization time and the single-byte processing time; each target execution thread corresponding to the target number of threads is used to obtain the target data from the corresponding target data set, and a database operation statement is used to synchronize the target data from the target data in the source file of the difference file pair to the corresponding target file.

[0138] Optionally, if it is determined that the target file is not included in the target directory, the synchronization module 703 is specifically configured to:

[0139] Create a target file based on the target storage path; synchronize the target data in the source file to the corresponding target file according to the thread's data synchronization strategy.

[0140] Optionally, the data synchronization device provided in this embodiment further includes: a generating module and a prohibiting module.

[0141] Accordingly, the generation module is specifically used to: in response to the first file hash value being the same as the second file hash value, delete the target file based on the target storage path; use a preset file operation command and generate a symbolic link between the target file and the source file based on the source storage path, the symbolic link is used to access the data in the source file when the symbolic link is triggered.

[0142] Correspondingly, the prohibition module is specifically used to: in response to the first directory hash value being the same as the second directory hash value, prohibit the execution of synchronization of the source file corresponding to the target data into the corresponding target file.

[0143] For the description of the features in the embodiment corresponding to the data synchronization device, reference can be made to the relevant description of the embodiment corresponding to the data synchronization method, which will not be repeated here.

[0144] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0145] In the specific implementation process, at least one processor 801 executes the computer execution instructions stored in the memory 802, so that at least one processor 801 executes the above-mentioned data synchronization method embodiment.

[0146] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0147] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0148] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0149] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0150] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data synchronization method embodiments when running.

[0151] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0152] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data synchronization method embodiments are implemented.

[0153] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data synchronization method embodiments are implemented.

[0154] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] The above is a detailed introduction to a data synchronization method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data synchronization method, characterized in that: include: In response to determining that the data synchronization condition is met, determining a source storage path corresponding to at least one target data to be synchronized and a target storage path preconfigured for the target data; The source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized; If it is determined that the target file is included in the target directory, then obtaining a first directory hash value of the source directory storing data where each of the target data is located based on the source directory name, and obtaining a second directory hash value of the target directory storing data based on the target directory name; In response to the first directory hash value being different from the second directory hash value, obtaining a first file hash value of the source file data where each target data is located, and obtaining a second file hash value of the target file data corresponding to each target data; Determine at least one difference file pair according to each of the first file hash values ​​and each of the second file hash values, where the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value; According to the data synchronization strategy of the thread, the target data in the source file of at least one of the difference file pairs is synchronized to the corresponding target file.

2. The method according to claim 1, characterized in that The step of obtaining a first directory hash value of the source directory storing data where each target data is located based on the source directory name, and obtaining a second directory hash value of the target directory storing data based on the target directory name, includes: Acquire at least one of the source files based on the source directory name and the source storage path, and calculate a first file hash value of each of the source files based on a preset hash algorithm; Acquire at least one target file based on the target directory name and the target storage path, and calculate a second file hash value of each target file based on the preset hash algorithm; Concatenate the hash values ​​of the first files, and input the concatenated result of the hash values ​​of the first files into a preset hash algorithm to obtain a first directory hash value; The hash values ​​of the second files are concatenated, and the concatenated result is input into a preset hash algorithm to obtain a second directory hash value.

3. The method according to claim 1, characterized in that The determining at least one difference file pair according to each of the first file hash values ​​and each of the second file hash values ​​includes: Comparing each of the first file hash values ​​with the corresponding second file hash values ​​to obtain at least one target comparison result, where the target comparison result is a comparison result where the first file hash value is different from the second file hash value; The source files and the target files corresponding to the target comparison results are determined as difference file pairs.

4. The method according to claim 1, characterized in that: The step of synchronizing target data in a source file of at least one of the difference files to a corresponding target file according to the data synchronization strategy of the thread includes: Determine the target number of execution threads according to the preset single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time; Dividing each of the target data into a plurality of target data sets based on the number of bytes processed by a single thread; the number of bytes processed by the single thread is determined according to the preset data synchronization time and the single byte processing time; The target execution threads corresponding to the target number of threads are used to obtain target data from the corresponding target data set, and the target data is synchronized from the target data in the source file of the difference file pair to the corresponding target file using a database operation statement.

5. The method according to claim 1, characterized in that If it is determined that the target file is not included in the target directory, the method further includes: Creating a target file based on the target storage path; According to the data synchronization strategy of the thread, the target data in the source file is synchronized to the corresponding target file.

6. The method according to claim 1, characterized in that Also includes: In response to the first file Hash value being the same as the second file Hash value, deleting the target file based on the target storage path; A preset file operation command is used to generate a symbolic link between the target file and the source file based on the source storage path, and the symbolic link is used to access the data in the source file when the symbolic link is triggered.

7. The method according to claim 1, characterized in that Also includes: In response to the first directory hash value being the same as the second directory hash value, synchronization of the source file corresponding to the target data into the corresponding target file is prohibited.

8. A data synchronization device, characterized in that: include: A determination module, configured to determine, in response to determining that the data synchronization condition is satisfied, a source storage path corresponding to at least one target data to be synchronized and a target storage path preconfigured for the target data; the source storage path includes a source directory name, the source directory includes a source file where the target data is located, and the target storage path includes a target directory name and a target file name to be synchronized; an acquisition module, configured to acquire, if it is determined that the target directory includes the target file, a first directory hash value of the source directory storing the data where each of the target data is located based on the source directory name, and acquire a second directory hash value of the target directory storing the data based on the target directory name; The acquisition module is configured to, in response to the first directory hash value being different from the second directory hash value, acquire the first file hash value of the source file data where each target data is located, and acquire the second file hash value of the target file data corresponding to each target data; The determination module is used to determine at least one difference file pair according to each of the first file hash values ​​and each of the second file hash values, wherein the difference file pair is a file pair formed by a source file and a target file whose first file hash value is different from the corresponding second file hash value; The synchronization module is used to synchronize the target data in the source file of at least one of the difference file pairs to the corresponding target file according to the data synchronization strategy of the thread.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data synchronization method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 7.

Images