Data synchronization methods, devices, electronic equipment and storage media

Abstract

This application discloses a data synchronization method, apparatus, electronic device, and storage medium, relating to the field of information technology. The method includes: quickly identifying changed data based on data synchronization conditions; calculating a first directory hash value and a second directory hash value based on the source directory; efficiently determining whether there is a difference between the source directory and the target directory based on the first and second directory hash values; if there is a difference between the source and target directories, comparing the hash values ​​of each first file with the hash values ​​of each second file to accurately identify the differing files; and finally, quickly synchronizing the target data in the source file of this differing file pair to the corresponding target file according to the thread's data synchronization strategy. This solves the problems of low efficiency in comparing data in files and difficulty in guaranteeing the consistency of synchronized data changes, achieving the technical effect of improving the efficiency of comparing data in files and ensuring the consistency of synchronized data changes.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a data synchronization method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of information technology, there is a demand for user data storage, multi-location data retrieval, and data backup in many fields such as healthcare, education, transportation, and finance. However, the increasing business demands of users have led to a rapid increase in data volume, requiring more efficient, faster, and more accurate data synchronization methods for storage. Therefore, improving data synchronization efficiency is an urgent problem to be solved.

[0003] Currently, traditional data synchronization methods involve directly comparing source and target files based on their timestamps, sizes, and contents to identify differences, and then synchronizing the data using full or incremental copying. These methods are only suitable for processing small batches of data with simple formats. However, when dealing with large volumes of data or scenarios requiring frequent data changes, these methods often suffer from low efficiency in comparing data within files and difficulty in ensuring consistency of synchronized data, resulting in slow data synchronization speeds. Summary of the Invention

[0004] This application provides a data synchronization method, apparatus, electronic device, and storage medium to at least solve the problems of low efficiency in comparing data in files and difficulty in guaranteeing the consistency of synchronized data in related technologies.

[0005] This application provides a data synchronization method, including:

[0006] In response to determining that the data synchronization conditions are met, at least one source storage path corresponding to the target data to be synchronized and a target storage path pre-configured for the target data are determined; the source storage path includes the source directory name, the source directory includes the source file where the target data is located, and the target storage path includes the target directory name and the target file name to be synchronized.

[0007] If it is determined that the target file is included in the target directory, then the first directory hash value of the data stored in the source directory where each target data is located is obtained based on the source directory name, and the second directory hash value of the data stored in each target directory is obtained based on the target directory name.

[0008] In response to the difference between the hash values ​​of the first directory and the second directory, the first file hash value of the source file data where each target data is located is obtained, and the second file hash value of the target file data corresponding to each target data is obtained;

[0009] Based on the hash values ​​of each first file and each second file, at least one pair of differing files is determined. A pair of differing files is a file pair formed by a source file and a target file whose first file hash values ​​are different from their corresponding second file hash values.

[0010] Based on the thread's data synchronization strategy, the target data in the source file of at least one difference file pair is synchronized to the corresponding target file.

[0011] This application also provides a data synchronization device, including:

[0012] The determination module is used to determine, in response to determining that the data synchronization conditions are met, at least one source storage path corresponding to the target data to be synchronized and a target storage path pre-configured for the target data; the source storage path includes the source directory name, the source directory includes the source file where the target data is located, and the target storage path includes the target directory name and the target file name to be synchronized;

[0013] The acquisition module is used to, if it is determined that the target file is included in the target directory, obtain the first directory hash value of the data stored in the source directory where each target data is located based on the source directory name, and obtain the second directory hash value of the data stored in each target directory based on the target directory name;

[0014] The acquisition module is used to obtain the first file hash value of the source file data where each target data is located and the second file hash value of the target file data corresponding to each target data in response to the difference between the hash values ​​of the first directory and the hash values ​​of the second directory.

[0015] The determination module is used to determine at least one difference file pair based on the hash values ​​of each first file and each second file. The difference file pair is a file pair formed by a source file and a target file whose hash values ​​of the first file and the corresponding hash values ​​of the second file are different.

[0016] The synchronization module is used to synchronize the target data in the source file of at least one difference file pair to the corresponding target file according to the thread's data synchronization strategy.

[0017] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described data synchronization methods when executing the computer program.

[0018] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data synchronization methods.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data synchronization methods.

[0020] This application, by leveraging data synchronization conditions, can quickly identify changed data. Furthermore, when the target directory contains target files, it can obtain the first directory hash value of the data stored in the source directory based on the source directory name, and the second directory hash value of the data stored in the target directory based on the target directory name. This allows for efficient and rapid determination of differences between the source and target directories based on the first and second directory hash values. Even with minor file variations, the differences between the source and target directories can be identified using these hash values. Moreover, when differences exist between the source and target directories, the hash values ​​of each first and second file can be compared to accurately identify the differing files. Finally, based on a thread-based data synchronization strategy, the target data in the source file of this differing file pair can be quickly synchronized to the corresponding target file, thus achieving rapid synchronization of differing files. Therefore, it solves the problems of low efficiency in comparing data in files and difficulty in guaranteeing the consistency of synchronized data changes, achieving the technical effects of improving the efficiency of data comparison in files and ensuring the consistency of synchronized data changes. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an application scenario diagram of a data synchronization method provided in the embodiments of this application;

[0023] Figure 2 This is a flowchart of a data synchronization method provided in the embodiments of this application;

[0024] Figure 3 This is a flowchart of a method for obtaining a hash value provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a method for obtaining hash values ​​provided in an embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating a method for setting symbolic links provided in an embodiment of this application;

[0027] Figure 6 This is a flowchart illustrating a data synchronization method provided in an embodiment of this application.

[0028] Figure 7This is a schematic diagram of a data synchronization device provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0031] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0033] Currently, traditional data synchronization methods directly compare source and target files based on their timestamps, sizes, and contents. Specifically, this involves querying the timestamps, sizes, and contents of all source and target files, then comparing each source file's timestamp, size, and content with the target file's timestamp, size, and content to identify differences. Only then can full or incremental copying be used to synchronize the data. This method is only suitable for processing small batches of data with simple formats. However, when dealing with large volumes of data or scenarios requiring frequent data changes, these methods struggle to achieve rapid data synchronization and ensure consistent data changes, resulting in slow data synchronization speeds.

[0034] Therefore, in order to improve the efficiency of comparing data in files and ensure the consistency of synchronized data changes, the difference file can be determined based on the source file and the corresponding target file of the target data to be synchronized, and then synchronization can be performed only on the difference file. Furthermore, due to the uniqueness of hash values—meaning that even slight changes in the data within a file will synchronize the hash value—we can determine if data has changed by comparing the hash values ​​of the source and target files. To further reduce the number of hash value comparisons between the source and target files, we can first determine if there are differences between the source and target directories. This involves obtaining the hash value of the first directory containing the target data in the source directory and calculating the hash value of the second directory based on the target directory name. If there are differences, to improve data synchronization speed and reduce the amount of data to be synchronized, we can synchronize only the differing files. Therefore, we can continue comparing the hash values ​​of the source and target files to quickly identify the differing files. Threads, with their fast response time and support for parallel task execution, can quickly synchronize the target data from at least one differing file pair from the source file to the corresponding target file, based on the thread's data synchronization strategy. Therefore, improving the efficiency of comparing data in files and ensuring consistent changes in synchronized data also improves the data synchronization speed.

[0035] First, let me explain the terms used in this application:

[0036] Hash value: also known as hash value, is usually represented by a short string of random letters and numbers. It is a binary value obtained by hashing a set of input information of arbitrary length.

[0037] Hash algorithms transform input data of arbitrary length using a hash function, resulting in a fixed-length output value. This output value is typically called a hash value, and it is used to identify the uniqueness and integrity of the data.

[0038] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] This section describes the specific application environment architecture or hardware architecture upon which the data synchronization method depends. (References) Figure 1 , Figure 1This diagram illustrates an application scenario for a data synchronization method. The scenario includes a server device 101 containing the source file, a server device 102 containing the target file, and a data synchronization device 103. The server device 101 stores the source file (the source file), and the server device 102 stores the target file corresponding to the target data. The data synchronization device 103 is communicatively connected to both server devices 101 and 102.

[0040] Alternatively, the data synchronization device 103 can also be integrated into the server device 101 where the source file is located or the server device 102 where the target file is located.

[0041] Specifically, the data synchronization device 103 monitors for data changes on the server device 101 where the source file is located and determines that the data synchronization conditions are met; the data synchronization device 103 determines at least one source storage path corresponding to the target data to be synchronized and a target storage path pre-configured for the target data. The source storage path includes the source directory name, the source directory includes the source file where the target data is located, and the target storage path includes the target directory name and the target file name to be synchronized. If the data synchronization device 103 determines that the target file is included in the target directory, it sends a command to the server device 101 where the source file is located, carrying the source directory name information, to obtain a first directory hash value instruction, and sends a second directory hash instruction to the server device 102 where the target file is located, carrying the target directory information. Upon receiving the command to obtain the first directory hash value, the server device 101 where the source file is located obtains the first directory hash value of the data stored in the source directory based on the source directory name and sends it to the data synchronization device 103. Similarly, upon receiving the command to obtain the second directory hash value, the server device 102 where the target file is located obtains the second directory hash value of the data stored in the target directory based on the target directory name and sends it to the data synchronization device 103. The data synchronization device 103 responds to the difference between the first directory hash value and the second directory hash value and continues to process... The system sends a command to the source file server 101 to obtain the first file hash value and a command to the target file server 102 to obtain the second file hash value. The source file server 101 obtains the first file hash value of the source file data for each target data and sends it to the data synchronization device 103. The target file server 102 obtains the second file hash value of the target file data corresponding to each target data and sends it to the data synchronization device 103. After receiving the first and second file hash values, the data synchronization device identifies at least one pair of differing files. Based on the thread's data synchronization strategy, the data synchronization device 103 synchronizes the target data from the source files of at least one pair of differing files in the source file server 101 to the corresponding target file in the target file server 102 via network transmission. This improves the efficiency of comparing data in files and ensures consistent changes in synchronized data, thereby enabling rapid data synchronization.

[0042] Figure 2 Flowchart of the data synchronization method provided in the embodiments of this application Figure 1 ,like Figure 2 As shown, embodiments of this application provide a data synchronization method, which is described in detail below:

[0043] S201: In response to determining that the data synchronization conditions are met, determine at least one source storage path corresponding to the target data to be synchronized and a target storage path pre-configured for the target data; the source storage path includes the source directory name, the source directory includes the source file where the target data is located, and the target storage path includes the target directory name and the target file name to be synchronized.

[0044] The data synchronization condition refers to the condition used to determine whether data synchronization is required. Optionally, the data synchronization condition can be set to whether the data change is caused by a change in business requirements. The business requirements can be pre-defined according to needs, including: data version upgrade business requirements, data rollback business requirements, data backup business requirements, or other requirements; this embodiment does not limit the specific requirements. Optionally, the data synchronization condition can also be to determine whether changed data has been detected; this embodiment does not limit the method for determining the data synchronization condition.

[0045] The target data refers to the data to be synchronized.

[0046] The source storage path refers to the address of the file corresponding to the target data before synchronization. Optionally, the address of the file corresponding to the target data can consist of two forward slashes, a directory name, and a file name. For example, the forward slashes can be " / ". The directory name can be defined according to its function and expressed in English. For example, a directory used to store important data can be defined as "home". The file name can also be defined according to the data function and expressed in English. For example, for session data, the folder can be named "session", then " / home / session" represents a path. There can be multiple folder naming methods and path representations. In this embodiment, no specific limitation is made. Other file addresses, directory naming methods, and folder naming methods in the method of this application are represented as in this embodiment, and will not be repeated below.

[0047] The target storage path refers to the file address corresponding to the target data after synchronization. Determining a target storage path requires pre-setting the corresponding storage directory. Specifically, a target directory name can be set, and then the target file name to be synchronized and the storage directory can be combined using a file address representation method. The result of combining the target file name and storage directory using file addresses can be used as the target storage path. Alternatively, the target storage path can be generated in other ways; this embodiment does not impose specific limitations.

[0048] The target directory name refers to the name of the folder where the target file is stored.

[0049] The target file refers to the folder where the data to be synchronized is stored.

[0050] The source file refers to the file that stores the target data before synchronizing the target data.

[0051] The source directory name refers to the name of the folder where the source files are stored.

[0052] The target file name to be synchronized is the name of the file containing the target data to be synchronized.

[0053] Optionally, a configuration monitoring system and monitoring threads can be pre-configured on the server corresponding to the storage source directory to determine whether the data synchronization conditions are met and generate the corresponding file name information.

[0054] Specifically, when data changes, the file monitoring system first records the source storage path of the changed data, the corresponding business requirements, and the data type of the changed data in its internal monitoring mechanism and stores this information in a log. Then, the monitoring thread checks if the log is updated to determine if a change record exists and retrieves the corresponding business requirements from the log. These requirements are then compared with the business requirements set in the data synchronization conditions. If the business requirements match the data synchronization conditions, the data synchronization conditions are considered met.

[0055] Specifically, when the data synchronization conditions are met, the monitored changed data is used as the target data. This can be achieved by accessing the logs and obtaining the source storage path corresponding to the changed data from the logs, which is then used as the source storage path for the target data to be synchronized. The source storage path corresponding to the target data is then parsed using a path resolution method to obtain the corresponding target filename to be synchronized. Further, the obtained target filename to be synchronized is combined with a pre-configured target directory according to file address combinations to obtain the pre-configured target storage path for the target data. The path resolution method can be a regular expression, a path resolution command in the preset operating system, or other parsing methods; this embodiment does not impose any limitations.

[0056] Optionally, the system can further access the logs to obtain the changed data type and convert the data operation type into the form of the corresponding target data operation identifier. The changed data type can be an add, modify, or delete type of data; the changed data identifier is a symbol used to identify the data operation type. For example, a delete type can be represented as "del"; an add type as "add"; and a modify type as "mod"; other forms are also possible, and this embodiment does not impose specific limitations.

[0057] After obtaining the above data, the target file name to be synchronized, the source storage path corresponding to the target data to be synchronized, the target storage path, and the target data operation identifier are stored sequentially in the change information storage table. The change information storage table is used to store tables related to the target data, and can store path information, file name information, and target data operation identifier. In this embodiment, no specific limitation is made.

[0058] S202: If it is determined that the target file is included in the target directory, then obtain the first directory hash value of the data stored in the source directory where each target data is located based on the source directory name, and obtain the second directory hash value of the data stored in each target directory based on the target directory name.

[0059] The hash value of the first directory refers to the hash value corresponding to the data content in the source directory.

[0060] The second directory hash value refers to the hash value corresponding to the data content in the target directory.

[0061] Understandably, a hash value refers to a numerical value calculated using a hash algorithm. Both the hash values ​​of the first directory and the hash values ​​of the second directory can be calculated using a hash algorithm. The hash algorithm can include message digest algorithms or secure hash algorithms, etc., and is not specifically limited in this embodiment.

[0062] Specifically, determining whether the target file is included in the target directory can be done by directly comparing the target file with the filenames in the target directory; or by comparing the first letter of the target file with the filenames in the target directory; or by other methods, which are not limited in this embodiment. For example, all file paths in the target directory can be obtained based on the operating system path reading command, and the file paths can be parsed using a path resolution method to extract all filenames in the target directory, and all the extracted filenames in the target directory can be stored in a filename storage table; then, the fields of each filename corresponding to the target directory are compared with the filename fields of the target filenames to be synchronized obtained in step S201 that have the same first letter, and the comparison result is reported to the server. The comparison result can be preset to 1 or 0; if the comparison result is 1, it indicates that the target file is included in the target directory; if the comparison result is 0, it indicates that the target file is not included in the target directory. The comparison result can also be represented in other forms, which are not specifically limited in this embodiment.

[0063] Specifically, when it is determined that the target file is included in the target directory, the server can use a conventional path combination format. The default directory path address combination method is a combination of a forward slash and the directory name. For example, assuming the directory name is "tmp", the directory address is represented as " / tmp". Therefore, the source file name and the target file name to be synchronized can be obtained from the change information storage table, and the source directory path and target directory path corresponding to the source directory and target directory can be matched according to the conventional path combination format. Then, the source directory path and target storage path are respectively input into the hash algorithm. The hash algorithm can sequentially traverse the contents of all files in the directory files and calculate the first directory hash value of the data stored in the source directory where the target data is located and the second directory hash value of the data stored in the target directory.

[0064] Optionally, the hash value of each file in the source directory and the target directory can be calculated separately, and then the hash values ​​of all files in the same directory can be concatenated to obtain the first directory hash value of the source directory where the target data is stored and the second directory hash value of the target directory where the data is stored; the first directory hash value and the second directory hash value can also be calculated in other ways, and no specific limitation is made in this embodiment.

[0065] S203: In response to the difference between the hash values ​​of the first directory and the second directory, obtain the first file hash value of the source file data where each target data is located, and obtain the second file hash value of the target file data corresponding to each target data.

[0066] The first file hash value refers to the hash value calculated by applying a hash algorithm to the data content of the source file.

[0067] The second file hash value refers to the hash value calculated by applying a hash algorithm to the data content of the source file.

[0068] Optionally, the first file hash value of the source file data where each target data is located and the second file hash value of the target file data corresponding to each target data can be pre-stored in a table.

[0069] Specifically, a hash algorithm can be used to calculate the first file hash value corresponding to the data content in each source file sequentially; and the first file hash value of each source file, along with its corresponding source storage path and the target file name to be synchronized, can be stored sequentially in the source file hash value storage table. The source file hash value storage table is a table used to store the source file hash values ​​and the data associated with them.

[0070] Similarly, a hash algorithm can be used to calculate the second file hash value corresponding to the data content in each target file in turn; and the second file hash value of each target file, along with its corresponding target storage path and the target file name to be synchronized, can be stored in the target file hash value storage table in turn. The target file hash value storage table is a table used to store the target file hash values ​​and the data associated with the source file hash values.

[0071] Understandably, there are multiple ways to store and retrieve hash values. For example, they can also be stored in a dictionary, but this example does not specify a particular method. A dictionary is one form of data storage.

[0072] Specifically, the hash values ​​of the first directory and the second directory obtained in step 203 are compared. When the hash values ​​of the first directory and the second directory are different, it indicates that there is a difference between the first directory and the second directory. The first file hash value of the source file data where each target data is located is then obtained, and the second file hash value of the target file data corresponding to each target data is also obtained. Specifically, the corresponding target file name to be synchronized can be obtained sequentially from the change information storage table, and matched with the target file name in its source file hash value storage table to obtain the first file hash value and source storage path corresponding to the target file name to be synchronized; then, the corresponding target file name to be synchronized is matched with the target file name to be synchronized in the target file hash value storage table to obtain the second file hash value and target storage path corresponding to the target file name to be synchronized, and the first file hash value, second file hash value, source storage path, and target storage path are stored in the hash value comparison table; this process continues until all the first file hash values ​​and second file hash values ​​corresponding to the target file names to be synchronized are matched; the hash value comparison table is a table that stores the hash values ​​to be compared and their associated information.

[0073] S204: Based on the hash values ​​of each first file and each second file, determine at least one difference file pair. The difference file pair is a file pair formed by a source file and a target file whose hash values ​​of the first file and the corresponding hash values ​​of the second file are different.

[0074] Specifically, the hash values ​​of the first and second files corresponding to the same target filename in the hash value comparison table are compared sequentially. If the hash values ​​are different, it indicates that there is a file difference between the source file corresponding to the first file hash value and the target file corresponding to the second file hash value. This source file and target file are treated as a pair of differing files. The corresponding source and target storage paths can be retrieved from the hash value comparison table using either the first or second file hash value of this pair. This first or second file hash value, the source storage path, and the target storage path are then stored as a single differing file pair in the file pair path storage table. The file pair path storage table is used to store information about differing file pairs.

[0075] S205: According to the thread's data synchronization strategy, synchronize the target data in the source file of at least one difference file pair to the corresponding target file.

[0076] The data synchronization strategy refers to a preset method for performing data synchronization. For example, the data synchronization strategy can be configured based on threads.

[0077] Optionally, the number of threads can be set according to the configured data synchronization time. Specifically, the number of bytes of target data that a single thread can process can be tested in advance within a preset time, and this number of bytes can be used as the threshold for the total number of bytes processed by a single thread. Then, the source storage paths corresponding to the source files in the difference file pairs are retrieved sequentially from the file pair path storage table. Based on the source storage paths corresponding to the source files in the difference file pairs and the file byte query command of the operating system, the number of bytes occupied by the difference file pairs is queried, and the number of bytes occupied by all difference file pairs is added together to obtain the number of bytes occupied by the source file in the difference file pair. When the number of bytes occupied by the source file in a difference file pair is less than the threshold for the total number of bytes processed by a single thread, it indicates that one thread can be used to synchronize the target data. When the number of bytes occupied by the source file in at least one difference file pair is greater than the threshold for the total number of bytes processed by a single thread, it indicates that using one thread to synchronize the target data within the preset time is too slow, and multiple threads are needed to synchronize the target data to improve the data synchronization speed. Therefore, the number of bytes occupied by the source file in at least one difference file pair can be divided by the threshold of the total number of bytes processed by a single thread to determine the number of threads required to run. The target data in the source file of at least one difference file pair can then be synchronized to the corresponding target file using a multi-threaded approach. The data synchronization method can also be a database operation or other methods; this embodiment does not specify a particular method.

[0078] It is understandable that there are multiple ways to determine data synchronization based on threads, and this embodiment does not impose any specific limitations.

[0079] In this embodiment, in response to determining that the data synchronization conditions are met, a source storage path corresponding to at least one target data to be synchronized and a target storage path pre-configured for the target data are determined. The source storage path includes a source directory name, the source directory includes the source file where the target data is located, and the target storage path includes a target directory name and the target file name to be synchronized. If it is determined that a target file is included in the target directory, a first directory hash value of the data stored in the source directory where each target data is located is obtained based on the source directory name, and a second directory hash value of the data stored in the target directory is obtained based on the target directory name. In response to the first directory hash value and the second directory hash value being different, a first file hash value of the data in the source file where each target data is located is obtained, and a second file hash value of the data in the target file corresponding to each target data is obtained. Based on each first file hash value and each second file hash value, at least one difference file pair is determined. A difference file pair is a pair of source and target files whose first file hash value differs from their corresponding second file hash value. According to the thread's data synchronization strategy, the target data in the source file of at least one difference file pair is synchronized to the corresponding target file. Therefore, by directly comparing the hash values ​​of the first and second directories, it is possible to accurately determine whether there is a difference between the source and target directories. When a difference exists, the hash values ​​of each first and second file are further obtained, and by comparing these hash values, the difference file pair can be quickly and accurately identified. After confirming the difference file pair, the target data in the source file of this difference file pair can be quickly synchronized to the corresponding target file based on the thread's data synchronization strategy, thus achieving rapid synchronization of the difference files. Therefore, this improves the efficiency of comparing data in files and ensures the consistency of synchronized data changes, thereby increasing the data synchronization speed.

[0080] As an optional implementation, the method involves obtaining the first directory hash value of the data stored in the source directory of each target data location based on the source directory name, and obtaining the second directory hash value of the data stored in each target directory based on the target directory name, including:

[0081] At least one source file is obtained based on the source directory name and source storage path, and a first file hash value is calculated for each source file based on a preset hash algorithm. At least one target file is obtained based on the target directory name and target storage path, and a second file hash value is calculated for each target file based on a preset hash algorithm. The first file hash values ​​are concatenated, and the result of the concatenation is input into a preset hash algorithm to obtain a first directory hash value. The second file hash values ​​are concatenated, and the result of the concatenation is input into a preset hash algorithm to obtain a second directory hash value.

[0082] The preset hash algorithm refers to the algorithm set for calculating the hash value. It can be a message digest algorithm or other algorithms. This embodiment does not make any specific limitations.

[0083] Specifically, the message digest algorithm can locate the corresponding source or target file through the target or source storage path and access the data in the source or target file. It then groups and processes the accessed data according to specific rules. For each group of data in the source or target file, bitwise operations, addition, multiplication, and modulo operations are performed to generate multiple results. Each result is then ANDed with the previous result until the last result is reached. This process generates the first file hash value of the source file or the second file hash value of the target file.

[0084] Specifically, based on the current source directory name, the change information storage table can be used to match the corresponding source storage paths and their corresponding target storage paths. Then, each source storage path is sequentially input into a preset hash algorithm. This algorithm locates the source file location and accesses its content by accessing each source storage path. The source file content is then input into a hash function to obtain the first file hash value for each source file. Further, the first file hash values ​​are concatenated to obtain a string related to the first file hash value of each source file. Similarly, each target storage path is sequentially input into the preset hash algorithm. This algorithm locates the target file location and accesses its content by accessing each target storage path. The target file content is then input into a hash function to obtain the second file hash value for each target file. Further, the second file hash values ​​are concatenated to obtain a string related to the second file hash value of each target file. Finally, the string related to the first file hash value of each source file is input into the preset hash algorithm to obtain the first directory hash value of the data stored in the source directory where the target data is located; similarly, the string related to the second file hash value of each target file is input into the preset hash algorithm to obtain the second directory hash value of the data stored in the target directory.

[0085] Optionally, the process of calculating the hash value of the first directory where the source directory stores the target data and the hash value of the second directory where the target data is stored, according to the above method, can also be as follows: Figure 3As shown, specifically: the main synchronization process module is the main process module that completes the entire data synchronization. The main process module first calls the file hash value calculation module. This module obtains the paths of each source file from the path traversal module. The path traversal module sends the corresponding source file paths to the hash value calculation module. Then, the hash value calculation module locates the content of each source file based on its path and calculates the hash value corresponding to each source file. The hash value calculation module then sends the hash values ​​of each source file to the hash value combination module. The hash value combination module receives the hash values ​​of each source file and concatenates the hash value results. The hash value combination module then sends the hash value results of each source file back to the hash value calculation module. The hash value calculation module inputs the hash value results of each source file into a preset hash algorithm and outputs the first directory hash value of the data stored in the source directory where the target data is located. Repeating this process yields the second directory hash value of the data stored in the target directory.

[0086] Optionally, when the hash values ​​of the first directory and the second directory, or the hash values ​​of the first file and the second file, need to be obtained from two devices, the first device and the second device can communicate using standard data serialization and remote procedure call protocols (XDR), such as... Figure 4 As shown in the diagram. The first device can send a command to the second device to obtain the second hash value, carrying the target directory name and path. The second device calculates the second directory hash value according to the method described above and sends it to the first device via data serialization and a remote procedure call protocol. The first device is the device corresponding to the source file; for example, it could be... Figure 1 The source file is located on server device 101, and the second device is the device corresponding to the target file. (For example, it could be...) Figure 1 The target file shown is located on server device 102. Therefore, data synchronization across multiple devices can also be achieved through standard data serialization and remote procedure call (RPC) protocols. Here, standard data serialization and RPC protocols refer to network transmission protocols used to achieve cross-platform data exchange.

[0087] Understandably, dictionaries offer advantages such as high flexibility, ease of management, simple data structure, reduced storage space, and fast access speed. Therefore, to improve the subsequent reading speed of the hash values ​​corresponding to files, after calculating the first file hash value of the source file and the second file hash value of the target file, a dictionary-based hash value storage method can be used. Specifically: First, the source directory name can be matched from the change information storage table based on the current source directory name, and the source storage path and its corresponding target storage path can be obtained. The source storage path and its associated target storage path form a file path pair; a file path pair refers to the path pair formed by the source storage path and the target storage path. Then, the file path pair formed by the source storage path and its associated target storage path is standardized, and the file path pair is sorted according to ASCII alphabetical order. This ensures that the sorted source storage path and its associated target storage path always show a matching relationship. The file path standardization process can involve removing forward slashes or redundancy from the path, or other methods; this embodiment does not impose any limitations. Optionally, the path can also be sorted in other ways; this embodiment does not impose any specific limitations. ASCII is a standard code for information exchange that uses specified 7-bit or 8-bit binary combinations to represent 128 or 256 possible characters, mainly including English letters, numbers, punctuation marks, and some special symbols.

[0088] Then, a dictionary is created to store the hash values ​​corresponding to the source storage paths, and a dictionary is created to store the hash values ​​corresponding to the target storage paths. The dictionary can be created using common dictionary creation commands, such as the "dict" command, but other dictionary creation commands can also be used; this embodiment does not impose specific limitations. A dictionary typically consists of a primary key and multiple key-value pairs. The primary key is a unique identifier used to identify dictionary values, and the key-value pairs are the data associated with the primary key. Because each hash value is unique, the hash value of the first file in the source file can be used as the primary key, and its corresponding source storage path as the key-value pair, thus creating a dictionary that stores the hash values ​​corresponding to the source storage paths. Similarly, the hash value of the second file in the target file can be used as the primary key, and its corresponding target storage path as the key-value pair, thus creating a dictionary that stores the hash values ​​corresponding to the target storage paths.

[0089] Finally, the file path pairs formed by the source storage path corresponding to the current source directory and its matching target storage path are sequentially input into a preset hash algorithm to obtain the first file hash value of the source file and the second file hash value of the target file. The first file hash value of the source file and its input source storage path are then stored in a dictionary storing hash values ​​corresponding to the source storage path. Similarly, the second file hash value of the target file and its input target storage path are stored in the dictionary storing hash values ​​corresponding to the source storage path, until all file path pairs have been traversed. Therefore, hash values ​​can be obtained by accessing the dictionary, which can then be used to determine the difference files and create file links.

[0090] Among them, such as Figure 5 As shown, the main process module first calls the path acquisition module to obtain the source storage paths. The path acquisition module retrieves the corresponding source storage paths from the source directories in sequence and sends them to the main process module. The main process module receives the source storage paths and sends them to the hash storage dictionary creation module. The hash storage dictionary creation module creates a hash dictionary corresponding to each source storage path and sends the hash dictionary to the main process module. The main process module then sends the hash dictionary corresponding to the source storage path to the hash value calculation module. The hash value calculation module calculates the hash value corresponding to each source file in sequence and stores the calculated hash value in the hash dictionary corresponding to the source storage path. Repeating the above process can also obtain the hash value corresponding to each target file and store the calculated hash value in the hash dictionary corresponding to the target storage path.

[0091] In this embodiment, at least one source file is obtained based on the source directory name and source storage path, and a first file hash value for each source file is calculated based on a preset hash algorithm. At least one target file is obtained based on the target directory name and target storage path, and a second file hash value for each target file is calculated based on a preset hash algorithm. The first file hash values ​​are concatenated, and the result is input into a preset hash algorithm to obtain a first directory hash value. The second file hash values ​​are concatenated, and the result is input into a preset hash algorithm to obtain a second directory hash value. Hash values ​​are unique, and by calculating the hash value of the directory, it is possible to quickly verify whether there are data differences between the source directory and the target directory. In particular, the file hash values ​​are concatenated and then hashed again, so that the hash value of the directory is represented as a shorter string, thereby saving hash value storage space and improving the comparison speed when comparing target hash values ​​later.

[0092] As an optional implementation, at least one pair of differing files is determined based on the hash values ​​of each first file and each second file, including:

[0093] The hash values ​​of each first file are compared with the corresponding hash values ​​of the second file to obtain at least one target comparison result. The target comparison result is the comparison result where the hash values ​​of the first file and the hash values ​​of the second file are different. The source file and the target file corresponding to each target comparison result are determined as each difference file pair.

[0094] Specifically, a comparison result identifier field is added to the hash value comparison table. The hash values ​​of the first and second files corresponding to the same target filename in the hash value comparison table are compared sequentially. Then, a comparison result identifier is added to the corresponding comparison result identifier field in the hash value comparison table. The comparison result identifier is a symbol used to identify the differing files. For example, differing files can be identified as 0 and 1; 0 indicates a pair of differing files, and 1 indicates a pair of differing files. The 0 and 1 values ​​corresponding to the comparison result identifier field in the hash value comparison table are the target comparison results. Therefore, the corresponding difference file pairs can be quickly found by reading the difference file identifier from the hash value comparison table.

[0095] In this embodiment, the hash values ​​of each first file are compared with the corresponding hash values ​​of the second files to obtain at least one target comparison result. A target comparison result is a comparison result where the hash values ​​of the first and second files differ. The source and target files corresponding to each target comparison result are identified as pairs of differing files. By directly comparing the hash values ​​of each first file with the corresponding hash values ​​of the second files, the existence of differences between the source and target files can be directly confirmed. Furthermore, by identifying the target comparison results, differing files can be quickly located among multiple files. Therefore, this method of comparing the hash values ​​of each first file with the corresponding second files also improves file comparison efficiency.

[0096] As an optional implementation, according to the thread's data synchronization strategy, the target data in the source file of at least one difference file pair is synchronized to the corresponding target file, including:

[0097] The target execution thread number is determined based on the preset single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time. Each target data is divided into multiple target datasets based on the number of bytes processed by a single thread. The number of bytes processed by a single thread is determined based on the preset data synchronization time and the single-byte processing time. Each target execution thread corresponding to the target thread number retrieves the target data from the corresponding target dataset and uses database operation statements to synchronize the target data from the source file of the difference file pair to the corresponding target file.

[0098] The single-byte processing time refers to the processing time of a single thread on one byte. The time it takes for a single thread to process one byte of data can be tested and used as a preset single-byte processing time. Alternatively, a single-byte processing time can be obtained through other methods; this embodiment does not impose specific limitations.

[0099] The total number of bytes occupied by the target data refers to the sum of the number of bytes occupied by all target data in the memory storage space.

[0100] The preset data synchronization time refers to the time required to synchronize the target data. This time is a value set according to user needs; for example, it can be set to 10 seconds, or other times. In this embodiment, no specific limitation is made. Optionally, both the single-byte processing time and the preset data synchronization time can be stored in a thread configuration table. The thread configuration table includes a single-byte processing time field and its corresponding single-byte processing time; it also includes a preset data synchronization time field and its preset data synchronization time.

[0101] The target number of execution threads is the number of threads required to execute the target data synchronization.

[0102] The number of bytes processed by a single thread refers to the number of bytes of data processed by a single thread during the preset data synchronization time.

[0103] Specifically, the source storage paths corresponding to the source files in the difference file pairs can be obtained from the change information storage table. Then, based on the operating system's file byte query command, the total number of bytes occupied by each source file corresponding to each source storage path is calculated, and the byte counts of each source file are summed to obtain the total number of bytes occupied by the target data. Next, the single-byte processing time field is matched from the thread configuration table, and its corresponding single-byte processing time is obtained. The preset data synchronization time is divided by the preset single-byte processing time to obtain the number of bytes processed by a single thread. Then, the total number of bytes occupied by the target data is divided by the number of bytes processed by a single thread to obtain the target execution thread count. The data accessed from all source storage paths and corresponding files is stored in a data table. The dataset is then divided into multiple target datasets according to the number of bytes processed by a single thread. A synchronization thread is allocated to each target dataset using a thread pool according to the target execution thread count. Within each thread, database operation statements are used to synchronize the target data from the source files of the difference file pairs to the corresponding target files.

[0104] Specifically, the target data operation identifier can be identified from the change information storage table. If the identified change data operation identifier is "del", a database delete statement is used to delete the target data from the source file of the difference file pair and then delete the target data from the target file. If the identified target data operation identifier is "add", a database add statement is used to add the target data from the source file of the difference file pair to the corresponding target file. If the identified target data identifier is "mod", a database add statement is used to modify the target data from the source file of the difference file pair to the corresponding target file.

[0105] In this embodiment, the number of target execution threads is determined based on a preset single-byte processing time, the total number of bytes occupied by the target data, and a preset data synchronization time. Each target data set is divided into multiple target datasets based on the number of bytes processed per thread. The number of bytes processed per thread is determined based on the preset data synchronization time and the single-byte processing time. Each target execution thread corresponding to the target thread number retrieves the target data from its corresponding target dataset, and database operation statements are used to synchronize the target data from the source files of the difference file pair to the corresponding target files. By using the single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time, the set number of target execution threads can be allocated quickly and effectively, thereby saving thread allocation space and ensuring that each target dataset has a matching thread. Furthermore, synchronizing the target datasets based on the target execution threads improves the efficiency of data synchronization.

[0106] As an optional implementation, if it is determined that the target file is not included in the target directory, the method further includes:

[0107] Create a target file based on the target storage path; synchronize the target data in the source file to the corresponding target file according to the thread's data synchronization strategy.

[0108] Specifically, when it is determined that the target file is not included in the target directory, it indicates that there is no data backup of the target file in the target directory; a creation command for the target file is generated based on the file creation command in the operating system and the target file name; for example, the combination of the file creation command "mkdir" in the operating system and the target file name can be used as the creation command for the target file, or it can be created in other forms, which are not specifically limited in this embodiment.

[0109] After the target file is created, a corresponding processing thread is allocated to the target data in the source file according to the data synchronization strategy. After allocating the corresponding processing thread to the target data, database operation statements can be used to synchronize the target data in the source file to the target file. The data synchronization strategy can be any specific scheme described in this application, and will not be repeated in this embodiment.

[0110] In this embodiment, if it is determined that the target file is not included in the target directory, the target file is created based on the target storage path; according to the thread's data synchronization strategy, the target data in the source file is synchronized to the corresponding target file; even when the target file is not included in the target directory, the thread can still be used to synchronize the data, which can improve the efficiency of target data synchronization.

[0111] As an optional implementation, it also includes:

[0112] In response to the first file hash value being the same as the second file hash value, the target file is deleted based on the target storage path; a symbolic link to the source file corresponding to the target file is generated using a preset file operation command and based on the source storage path. The symbolic link is used to access the data in the source file when the symbolic link is triggered.

[0113] Among them, the preset file operation command refers to the command that generates symbolic links to source files corresponding to the source files of the target files from the source storage path. It can be a combination of the operation command "ln -s" and the source and target storage paths.

[0114] Understandably, when the hash value of the first file is the same as that of the second file, it indicates that the source file and the file in the target data are consistent; therefore, the data can be stored in only one of the folders, thus saving data storage space.

[0115] Specifically, from the hash value comparison table in step 203, the hash values ​​of the first and second files corresponding to the target filename to be synchronized can be obtained sequentially and compared. If the comparison results of the first and second file hash values ​​are the same, the corresponding hash value field is accessed in the hash value comparison table based on the same hash value, and the source storage path and target storage path corresponding to the same hash value field are obtained. Then, the location of the text file is located based on the target storage path, and the target file is deleted using the operating system's delete command. The source storage path and target storage path are combined with the operation command to generate a preset file command and execute the preset file command until all comparisons of the first and second file hash values ​​are completed. In this way, a symbolic link to the source file corresponding to the target file with the same hash value can be generated, and the target file data can be accessed by accessing the symbolic link. The generation of the symbolic link to the source file corresponding to the target file can also be done in other ways, and is not limited in this embodiment.

[0116] Optionally, when the hash values ​​of the first and second files are stored in a dictionary, they can also be... Figure 5As shown, the main process module sequentially retrieves the corresponding hash values ​​from the dictionary corresponding to the source storage path and the dictionary corresponding to the hash values ​​of the target storage path, and compares them. If the comparison result is the same, the main process module sends a message to the symbol creation module to delete the target file and create a link. The symbol creation module then generates symbolic links to the source files corresponding to the target files with the same hash values.

[0117] In this embodiment, in response to the first file hash value being the same as the second file hash value, the target file is deleted based on the target storage path; a preset file operation command is used to generate a symbolic link to the source file corresponding to the target file based on the source storage path. The symbolic link is used to access the data in the source file when the symbolic link is triggered. By generating a symbolic link to the source file corresponding to the target file through the source storage path instead of storing the target file, storage space for the data can be saved.

[0118] As an optional implementation, it also includes:

[0119] If the hash values ​​of the first directory and the second directory are the same, then the synchronization of the source file corresponding to the target data to the corresponding target file is prohibited.

[0120] When the hash values ​​of the first and second directories match, it indicates that there are no differing files between the source and target directories. Therefore, when there are data differences between the source and target directories, a synchronization prohibition flag is received internally by the server. When this flag appears, the server automatically exits the data synchronization process, thus preventing the synchronization of the source files corresponding to the target data to the corresponding target files.

[0121] In this embodiment, in response to the first directory hash value being the same as the second directory hash value, the synchronization of the source file corresponding to the target data to the corresponding target file is prohibited. By comparing the first directory hash value and the second directory hash value, it can be quickly and directly determined whether there is a difference between the target directory and the source directory. When the first directory hash value is the same as the second directory hash value, it indicates that there is no difference between the source directory and the target directory, and the data synchronization can be stopped directly. Therefore, it is possible to quickly determine whether to continue data synchronization, thereby reducing the calculation of the data synchronization process.

[0122] Optionally, when the data synchronization conditions are met, before executing the data synchronization process, the server can dynamically adjust the process priorities based on the importance and resource usage of each process, allocating processors and processes for execution to ensure efficient data synchronization. Specifically, the system pre-sets different priorities for processes, with multiple priority levels such as low, below normal, normal, above normal, and high. While processes are running, the system monitors their running status in real time, including CPU usage, memory usage, and I / O operation frequency. When a new task requires a process to run, the system analyzes the current resource usage. If the data synchronization process needs to quickly transfer a large amount of data, the system may raise its priority to above normal or high to ensure it receives sufficient processor resources.

[0123] Figure 6 The flowchart of a data synchronization method provided in an embodiment of this application illustrates the data synchronization calculation process. Specifically, in response to determining that the data synchronization conditions are met, it is determined whether target data exists. If not, the process ends and exits the data synchronization process. If it exists, at least one source storage path corresponding to the target data to be synchronized and a target storage path pre-configured for the target data are determined. A processor and process are allocated for the data synchronization task, and the calculation of the data synchronization process is started in the process.

[0124] The process involves several steps: First, determine if the target file is contained in the target directory. If it exists in the target directory, create the target file based on the target storage path. Then, according to the thread's data synchronization strategy, synchronize the target data from the source file to the corresponding target file. If the target file exists in the target directory, obtain the hash values ​​of the first and second directories. Check if the hash values ​​of the first and second directories are the same. If they are the same, prevent the synchronization of the source file corresponding to the target data to the corresponding target file. If the hash values ​​of the first and second directories are different, obtain the hash values ​​of each first and second file. Check if the hash values ​​of each first and second file are the same. If they are different, identify at least one pair of differing files and synchronize the target data from the source files of at least one pair of differing files to the corresponding target file according to the thread's data synchronization strategy. If the hash values ​​of each first and second file are the same, use a preset file operation command and generate a symbolic link to the source file corresponding to the target file based on the source storage path.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0126] Figure 7 This is a schematic diagram of the data synchronization device provided in an embodiment of this application. Figure 7 As shown, an embodiment of this application also provides a data synchronization device, which is located in a data synchronization equipment. The data synchronization device 70 provided in this embodiment includes: a determination module 701, an acquisition calculation module 702, and a synchronization module 703.

[0127] The determination module 701 is used to determine, in response to determining that the data synchronization conditions are met, at least one source storage path corresponding to the target data to be synchronized and a target storage path pre-configured for the target data; the source storage path includes the source directory name, the source directory includes the source file where the target data is located, and the target storage path includes the target directory name and the target file name to be synchronized.

[0128] The acquisition module 702 is used to, if it is determined that the target file is included in the target directory, obtain the first directory hash value of the data stored in the source directory where each target data is located based on the source directory name, and obtain the second directory hash value of the data stored in each target directory based on the target directory name.

[0129] The acquisition module 702 is used to obtain the first file hash value of the source file data where each target data is located and the second file hash value of the target file data corresponding to each target data in response to the difference between the hash value of the first directory and the hash value of the second directory.

[0130] The determination module 701 is used to determine at least one difference file pair based on the hash values ​​of each first file and each second file. The difference file pair is a file pair formed by a source file and a target file whose hash values ​​of the first file and the corresponding hash values ​​of the second file are different.

[0131] Synchronization module 703 is used to synchronize target data from the source files of at least one difference file pair to the corresponding target files according to the thread's data synchronization strategy.

[0132] Optionally, the acquisition module 702, when acquiring the first directory hash value of the data stored in the source directory of each target data based on the source directory name, and acquiring the second directory hash value of the data stored in each target directory based on the target directory name, is specifically used for:

[0133] At least one source file is obtained based on the source directory name and source storage path, and a first file hash value is calculated for each source file based on a preset hash algorithm. At least one target file is obtained based on the target directory name and target storage path, and a second file hash value is calculated for each target file based on a preset hash algorithm. The first file hash values ​​are concatenated, and the result of the concatenation is input into a preset hash algorithm to obtain a first directory hash value. The second file hash values ​​are concatenated, and the result of the concatenation is input into a preset hash algorithm to obtain a second directory hash value.

[0134] Optionally, the determining module 702, when determining at least one pair of differing files based on the hash values ​​of each first file and each second file, is specifically used for:

[0135] The hash values ​​of each first file are compared with the corresponding hash values ​​of the second file to obtain at least one target comparison result. The target comparison result is the comparison result where the hash values ​​of the first file and the hash values ​​of the second file are different. The source file and the target file corresponding to each target comparison result are determined as each difference file pair.

[0136] Optionally, the synchronization module 703, when synchronizing target data from the source file in at least one difference file pair to the corresponding target file according to the thread's data synchronization strategy, is specifically used for:

[0137] The target execution thread number is determined based on the preset single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time. Each target data is divided into multiple target datasets based on the number of bytes processed by a single thread. The number of bytes processed by a single thread is determined based on the preset data synchronization time and the single-byte processing time. Each target execution thread corresponding to the target thread number retrieves the target data from the corresponding target dataset and uses database operation statements to synchronize the target data from the source file of the difference file pair to the corresponding target file.

[0138] Optionally, the synchronization module 703, if it is determined that the target file is not included in the target directory, is specifically used for:

[0139] Create a target file based on the target storage path; synchronize the target data in the source file to the corresponding target file according to the thread's data synchronization strategy.

[0140] Optionally, the data synchronization device provided in this embodiment further includes a generation module and a prohibition module.

[0141] Accordingly, the generation module is specifically used to: delete the target file based on the target storage path in response to the first file hash value being the same as the second file hash value; and generate a symbolic link to the source file corresponding to the target file using a preset file operation command and based on the source storage path. The symbolic link is used to access the data in the source file when the symbolic link is triggered.

[0142] Accordingly, the prohibition module is specifically used to: in response to the first directory hash value being the same as the second directory hash value, prohibit the synchronization of the source file corresponding to the target data to the corresponding target file.

[0143] For a description of the features in the embodiment corresponding to the data synchronization device, please refer to the relevant description in the embodiment corresponding to the data synchronization method, which will not be repeated here.

[0144] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.

[0145] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to execute the above-described data synchronization method embodiment.

[0146] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0147] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0148] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0149] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0150] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data synchronization method embodiments at runtime.

[0151] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0152] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data synchronization method embodiments.

[0153] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data synchronization method embodiments.

[0154] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] The data synchronization method, apparatus, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method of data synchronization, the method comprising: The method comprises the following steps: In response to determining that the data synchronization condition is met, determining the source storage path corresponding to at least one target data to be synchronized and the target storage path pre-configured for the target data; The source storage path includes a source directory name, and the source directory includes a source file where the target data is located. The target storage path includes a target directory name and a target file name to be synchronized to; If it is determined that the target file is included in the target directory, a first directory hash value of the source directory storage data where each target data is located is obtained based on the source directory name, and a second directory hash value of the target directory storage data is obtained based on the target directory name; In response to the first directory hash value being different from the second directory hash value, a first file hash value of the source file data where each target data is located is obtained, and a second file hash value of the target file data corresponding to each target data is obtained; Each target comparison result is compared with the corresponding second file hash value to obtain at least one target comparison result. The target comparison result is the comparison result of the first file hash value being different from the second file hash value; Each target comparison result is compared with the corresponding second file hash value to obtain at least one target comparison result. The target comparison result is the comparison result of the first file hash value being different from the second file hash value; According to the data synchronization strategy of the thread, the target data in at least one source file of the difference file pair is synchronized to the corresponding target file.

2. The method of claim 1, wherein, The first directory hash value of the source directory storage data where each target data is located is obtained based on the source directory name, and the second directory hash value of the target directory storage data is obtained based on the target directory name, which comprises: At least one source file is obtained based on the source directory name and the source storage path, and a first file hash value of each source file is calculated based on a preset hash algorithm; At least one target file is obtained based on the target directory name and the target storage path, and a second file hash value of each target file is calculated based on the preset hash algorithm; Each first file hash value is spliced, and the result after splicing each first file hash value is input into a preset hash algorithm to obtain a first directory hash value; Each second file hash value is spliced, and the result after splicing each second file hash value is input into a preset hash algorithm to obtain a second directory hash value.

3. The method of claim 1, wherein, According to the data synchronization strategy of the thread, the target data in at least one source file of the difference file pair is synchronized to the corresponding target file, which comprises: The target execution thread number is determined according to the preset single-byte processing time, the total number of bytes occupied by the target data, and the preset data synchronization time; Each target data is divided into a plurality of target data sets based on the number of bytes processed by a single thread. The number of bytes processed by a single thread is determined according to the preset data synchronization time and the single-byte processing time; Each target data is obtained from the corresponding target data set by using each target execution thread corresponding to the target thread number, and the target data is synchronized from the target data in the source file of the difference file pair to the corresponding target file by using a database operation statement.

4. The method of claim 1, wherein, If it is determined that the target file is not included in the target directory, the method further comprises: creating a target file based on the target storage path; synchronizing target data in the source file into the corresponding target file according to the data synchronization policy of the thread.

5. The method of claim 1, wherein, Further comprising: in response to the first file hash value being the same as the second file hash value, deleting the target file based on the target storage path; generating a symbolic link of the target file corresponding to the source file based on the source storage path using a preset file operation command, the symbolic link being used to access data in the source file when the symbolic link is triggered.

6. The method of claim 1, wherein, Further comprising: in response to the first directory hash value being the same as the second directory hash value, prohibiting the synchronization of the source file corresponding to the target data into the corresponding target file.

7. A data synchronization apparatus, characterized by comprising: Comprising: a determination module configured to determine a source storage path corresponding to at least one target data to be synchronized and a target storage path pre-configured for the target data in response to determining that a data synchronization condition is met; the source storage path including a source directory name, and the source directory including a source file in which the target data is located, and the target storage path including a target directory name and a target file name to be synchronized to; an acquisition module configured to, if it is determined that the target file is included in the target directory, acquire a first directory hash value of the source directory storing data of each target data based on the source directory name, and acquire a second directory hash value of the target directory storing data of each target data based on the target directory name; the acquisition module is configured to, in response to the first directory hash value being different from the second directory hash value, acquire a first file hash value of the source file storing data of each target data, and acquire a second file hash value of the target file storing data of each target data; the determination module is configured to compare each first file hash value with the corresponding second file hash value to obtain at least one target comparison result, the target comparison result being a comparison result of the first file hash value being different from the second file hash value, and determine the source file and the target file corresponding to each target comparison result as each difference file pair, the difference file pair being a file pair formed by the source file and the target file whose first file hash value is different from the corresponding second file hash value; a synchronization module configured to synchronize target data in the source file of at least one difference file pair into the corresponding target file according to the data synchronization policy of the thread.

8. An electronic device, comprising: Comprising: a memory configured to store a computer program; a processor configured to implement the steps of the data synchronization method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the data synchronization method according to any one of claims 1 to 6.

Images