A method, device, computer equipment and storage medium for batch data synchronization

Abstract

The present invention discloses a data batch synchronization method, device, computer equipment and storage medium, including: obtaining a source storage directory of data to be synchronized, generating a corresponding list of directories to be synchronized; sequentially obtaining at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assigning each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and executing the operation of synchronizing the current level data in each directory to be synchronized to a target directory matching each directory to be synchronized through each synchronization thread; when it is determined that the batch synchronization condition is met, again executing the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, and assigning each directory to be synchronized in the new synchronization processing batch to a corresponding synchronization thread. The technical solution of the embodiment of the present invention can realize batch parallel processing of multiple data synchronization tasks, thereby improving the efficiency of data synchronization.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of operation and maintenance technology, and in particular to a method, device, computer equipment and storage medium for batch data synchronization. Background Art

[0002] Supercomputing platforms have been widely used in various industries. Users of supercomputing platforms sometimes ask operation and maintenance personnel to migrate servers. In order to respond to this demand, operation and maintenance personnel need to synchronize data from the source server to the target server.

[0003] The synchronization operation of existing data is realized through the execution instruction "rsync -a source server directory target server directory" in the remote synchronization tool (Remote sync, Rsync). This synchronization method synchronizes the files in the source server one by one, that is, the synchronization operation of the next file is started only after the synchronization of the previous file is completed.

[0004] However, the execution efficiency of the existing data synchronization method is low. When the number of files to be synchronized is large, the execution time of the existing synchronization method is long, which easily results in a high time cost. Summary of the invention

[0005] The embodiments of the present invention provide a method, apparatus, computer device and storage medium for batch synchronization of data, which can realize batch parallel processing of multiple data synchronization tasks and improve the efficiency of data synchronization.

[0006] In a first aspect, an embodiment of the present invention provides a method for batch synchronization of data, the method comprising:

[0007] Obtain a source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each of the directories;

[0008] Sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute, through each synchronization thread, an operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory matching each directory to be synchronized;

[0009] When it is determined that the batch synchronization conditions are met, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to the corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch.

[0010] In a second aspect, an embodiment of the present invention further provides a data batch synchronization device, the device comprising:

[0011] An acquisition module, used to acquire a source storage directory of data to be synchronized, and traverse directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each directory;

[0012] A synchronization module is used to sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute, through each synchronization thread, an operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory matching each directory to be synchronized;

[0013] The batch synchronization module is used to, when it is determined that the batch synchronization conditions are met, again execute the operation of obtaining at least two remaining to-be-synchronized directories in the to-be-synchronized directory list as a new synchronization processing batch, and assign each to-be-synchronized directory in the new synchronization processing batch to the corresponding synchronization thread, so as to perform batch synchronization processing on the data in each to-be-synchronized directory in each synchronization processing batch.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0015] one or more processors;

[0016] A storage device for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement a data batch synchronization method provided by any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, a method for batch synchronization of data provided by any embodiment of the present invention is implemented.

[0019] The technical solution of the embodiment of the present invention obtains the source storage directory of the data to be synchronized, and traverses the directories of each level included in the source storage directory, generates a list of directories to be synchronized corresponding to each directory, obtains at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assigns each directory to be synchronized in the current synchronization processing batch to the corresponding synchronization thread, and executes the operation of synchronizing the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized through each synchronization thread, and when it is determined that the batch synchronization condition is met, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, assigning each directory to be synchronized in the new synchronization processing batch to the corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch. The technical solution of the embodiment of the present invention can realize batch parallel processing of multiple data synchronization tasks, can reduce the time consumed by data in the synchronization process, and improve the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a method for batch synchronization of data in Embodiment 1 of the present invention;

[0021] Figure 2 is a flow chart of a method for batch synchronization of data in Embodiment 2 of the present invention;

[0022] Figure 3 It is a structural diagram of a data batch synchronization device in Embodiment 3 of the present invention;

[0023] Figure 4 It is a structural diagram of a computer device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0025] Embodiment 1

[0026] Figure 1 This is a flow chart of a data batch synchronization method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of synchronizing data from a source server to a target server in batches. The method can be performed by a data batch synchronization device, which can be implemented by software and / or hardware and can generally be integrated in a computer device. Specifically, the method includes the following steps:

[0027] Step 110: Obtain a source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each of the directories.

[0028] In this step, the data to be synchronized is the data that needs to be synchronized from the source server to the target server, and the source storage directory is the root directory corresponding to the data to be synchronized on the source server.

[0029] In one implementation of an embodiment of the present invention, optionally, traversing the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each directory, including: taking the source storage directory as the traversal starting point, traversing the directories of each level included in the source storage directory layer by layer, and generating a directory to be synchronized for each directory; combining the generated directories to be synchronized to obtain the list of directories to be synchronized.

[0030] Among them, the source storage directory is used as the traversal starting point, and the source storage directory and the subdirectories of each level included in the source storage directory are traversed layer by layer. The source storage directory and the subdirectories of each level are all directories to be synchronized, and the directories to be synchronized are combined together to obtain a list of directories to be synchronized.

[0031] For example, assuming that the source storage directory is src, three directories to be synchronized are generated according to the source storage directory, namely src, src1 and src2, where: src = / mnt / lustre / asrdata / home / xch27 / , src1 = / mnt / lustre / asrdata / home / xch27 / carno2_wav, src2 = / mnt / lustre / asrdata / home / xch27 / Videos, then src, src1 and src2 are combined together to obtain a list of directories to be synchronized.

[0032] Step 120, sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and through each synchronization thread, execute the operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory that matches each directory to be synchronized.

[0033] In this embodiment, a thread pool for configuring multithreading is pre-established, and a plurality of synchronization threads are pre-configured in the thread pool. A synchronization thread refers to a program fragment that can independently run data synchronization operations in a program, and a target directory refers to a storage directory on a target server for storing the current level data in the directory to be synchronized. Each directory to be synchronized in the list of directories to be synchronized corresponds to a synchronization thread, and each synchronization thread is used to synchronize the current level data in the corresponding directory to be synchronized to the matching target directory.

[0034] In one implementation of an embodiment of the present invention, optionally, an operation of synchronizing the current level data in the directory to be synchronized to the target directory that matches the directory to be synchronized is performed through a synchronization thread, including: constructing a target directory that matches the directory to be synchronized through a synchronization thread; and executing an operation of synchronizing the current level data in the directory to be synchronized to the target directory through a synchronization thread.

[0035] In this step, the current level data is the data in the directory to be synchronized, and does not include the subdirectories and data in each subdirectory included in the directory to be synchronized. After each synchronization thread builds a target directory that matches each directory to be synchronized, it can start synchronizing the current level data in each directory to be synchronized to the matching target directory in the background without waiting for other synchronization threads to complete the synchronization task, thereby realizing parallel synchronization of the data in each directory to be synchronized in the current synchronization batch.

[0036] Among them, a target directory matching the directory to be synchronized is constructed through a synchronization thread, including: extracting the last-level directory name and the directory path before the last-level directory name in the source storage directory; identifying the directory path in the directory to be synchronized, and replacing the directory path in the directory to be synchronized with a preset target storage directory to obtain the target directory.

[0037] In this embodiment, the target storage directory is a root directory corresponding to the data to be synchronized on the target server. Specifically, assuming that the source storage directory is src, the directory to be synchronized is src1, src = / mnt / lustre / asrdata / home / xch27 / , src1 = / mnt / lustre / asrdata / home / xch27 / carno2_wav, the last directory name in src is "xch27", and the directory path before the last directory name is " / mnt / lustre / asrdata / home / ", assuming that the pre-set target storage directory is des, des = / mnt / lustre02 / jiangsu / aispeech / home / yj912 / asrdata_yj912 / , then replace the directory path in the directory to be synchronized src1 with des, and obtain the target directory des_path, des_path = / mnt / lustre02 / jiangsu / aispeech / home / yj912 / asrdata_yj912 / xch27 / carno2_wav.

[0038] Step 130: When it is determined that the batch synchronization conditions are met, the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to a corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch.

[0039] In this embodiment, optionally, when it is detected that the number of directories to be synchronized in the current synchronization processing batch is equal to a preset threshold, it is determined that the batch synchronization condition is met, and after waiting for a preset time, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to a corresponding synchronization thread.

[0040] Therefore, when it is detected that the number of directories to be synchronized in the previous synchronization processing batch is equal to the preset threshold, the data synchronization task in the next synchronization processing batch can be started without waiting for the data synchronization task in the previous synchronization processing batch to be completed, thereby realizing batch processing of multiple data synchronization tasks.

[0041] In this embodiment, while each synchronization thread synchronizes the data in each to-be-synchronized directory in parallel, it also includes: storing all the data in each hierarchical directory in the source storage directory item by item in the target storage directory through the synchronization instructions in the remote synchronization tool Rsync for storage verification.

[0042] Among them, while each synchronization thread synchronizes the data in each directory to be synchronized in parallel, it also stores all the data in each level directory in the source storage directory item by item in a specific directory in the target storage directory through the synchronization instructions in the remote synchronization tool Rsync. The purpose is to verify the data that the synchronization thread has synchronized, avoid data errors and losses during the synchronization process, and thus ensure the accuracy of the data during the synchronization process; secondly, when the synchronization thread has not yet started to synchronize the data in a directory to be synchronized, the data in the directory to be synchronized can be synchronized through the synchronization instructions in the remote synchronization tool Rsync, which can shorten the time consumed by the data in the synchronization process and improve the efficiency of data synchronization.

[0043] The directory name of the specific directory in the target storage directory is equal to the last-level directory name in the source storage directory.

[0044] The technical solution of the embodiment of the present invention obtains the source storage directory of the data to be synchronized, and traverses the directories of each level included in the source storage directory, generates a list of directories to be synchronized corresponding to each directory, obtains at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assigns each directory to be synchronized in the current synchronization processing batch to the corresponding synchronization thread, and executes the operation of synchronizing the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized through each synchronization thread, and when it is determined that the batch synchronization condition is met, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, assigning each directory to be synchronized in the new synchronization processing batch to the corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch. The technical solution of the embodiment of the present invention can realize batch parallel processing of multiple data synchronization tasks, can reduce the time consumed by data in the synchronization process, and improve the efficiency of data synchronization.

[0045] Embodiment 2

[0046] This embodiment is a further refinement of the above embodiment. The explanation of terms that are the same or corresponding to those in the above embodiment will not be repeated in this embodiment. Figure 2 This is a flow chart of a data batch synchronization method provided in Embodiment 2 of the present invention. In this embodiment, the technical solution of this embodiment can be combined with one or more methods in the solutions of the above embodiments. In this embodiment, Figure 2 As shown, the method provided by the embodiment of the present invention may also include:

[0047] Step 210: Obtain a source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each of the directories.

[0048] In this embodiment, the to-be-synchronized directory list includes multiple to-be-synchronized directories, where the to-be-synchronized directories are the source storage directory and sub-directories of various levels included in the source storage directory.

[0049] Step 220: sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch.

[0050] In this step, optionally, at least two sub-directories included in the source storage directory can be used as the current synchronization processing batch.

[0051] Step 230: Allocate a directory to be synchronized in the current synchronization processing batch to a synchronization thread, and update the number of synchronization tasks matching the current synchronization processing batch.

[0052] In this step, the directories to be synchronized in the current synchronization processing batch may be subdirectories included in the source storage directory. When each directory to be synchronized in the current synchronization processing batch is assigned to a synchronization thread, the number of synchronization tasks matching the current synchronization processing batch is increased by one.

[0053] Step 240 , determine whether the number of synchronization tasks in the current synchronization processing batch is less than the preset number threshold. If so, return to step 230 until the number of synchronization tasks in the current synchronization processing batch is equal to the preset number threshold; if not, execute step 250 .

[0054] In this step, the preset quantity threshold is the maximum value of the number of synchronization tasks in each synchronization processing batch.

[0055] Step 250: Through each synchronization thread, an operation of synchronizing the current level data in each to-be-synchronized directory to a target directory matching each to-be-synchronized directory is performed.

[0056] In this step, the directories to be synchronized in the current synchronization processing batch can be the subdirectories included in the source storage directory, and the current hierarchical data in each subdirectory is synchronized to the target directory matching the subdirectory through each synchronization thread. While each synchronization thread synchronizes the data in each subdirectory, all the data in each hierarchical directory in the source storage directory is stored item by item in the target storage directory through the synchronization instructions in the remote synchronization tool Rsync.

[0057] The total data includes the data in the source storage directory, and each subdirectory included in the source storage directory and the data in each subdirectory.

[0058] In this embodiment, after executing the operation of synchronizing the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized through each synchronization thread, it also includes: obtaining the number of currently running synchronization threads and the target number of central processing unit (CPU); wherein the CPU target number of cores is the total number of CPU cores used to execute data synchronization tasks; determining whether the number of currently running synchronization threads is greater than the CPU target number of cores, and if so, determining the sleep time of the first thread, and controlling the currently running synchronization thread to be in a sleep state within the sleep time of the first thread; after waiting for the sleep time of the first thread, the currently running synchronization thread again executes the operation of synchronizing the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized.

[0059] In this step, the number of synchronization threads currently running is the number of synchronization threads executing the data synchronization task. Before obtaining the target number of CPU cores, the total number of CPU cores and the preset proportional coefficient are obtained in advance, and the total number of CPU cores is multiplied by the preset proportional coefficient to obtain the target number of CPU cores. Among them, the preset proportional coefficient is used to reflect the proportion of the number of CPU cores that can be used to execute the synchronization task to the total number of CPU cores.

[0060] If the number of currently running synchronous threads is greater than the target number of CPU cores, it means that the CPU load is too high. By determining the sleep time of the first thread and controlling each synchronous thread to be in a sleep state, the CPU load pressure can be reduced and the CPU service life can be extended.

[0061] In this embodiment, the preset proportional coefficient can be set to 0.6, and the sleep time of the first thread can be set to 10s. The specific values ​​are preset according to actual conditions, and this embodiment does not limit this.

[0062] Step 260: When it is determined that the batch synchronization conditions are met, the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to a corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch.

[0063] In one implementation of an embodiment of the present invention, optionally, when it is determined that the number of synchronization tasks in the current synchronization processing batch is equal to a preset number threshold, after waiting for the second thread to sleep for a period of time, the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to a corresponding synchronization thread.

[0064] Among them, the sleep time of the second thread can be set to a shorter time. When it is determined that the number of synchronization tasks in the current synchronization processing batch is equal to the preset number threshold, by determining a shorter thread sleep time, it is possible to avoid each synchronization thread occupying CPU resources all the time. That is, when each synchronization thread is in a sleep state, the CPU will call resources to other threads, thereby reducing the time each synchronization thread occupies the CPU and improving CPU utilization.

[0065] Therefore, when it is detected that the number of synchronization tasks in the previous synchronization processing batch is equal to the preset number threshold, the data synchronization tasks in the next synchronization processing batch can be started without waiting for the completion of the data synchronization tasks in the previous synchronization processing batch, thereby realizing batch processing of multiple data synchronization tasks.

[0066] In this embodiment, the sleep time of the second thread can be set to 5s. The specific value is preset according to the actual situation, and this embodiment does not limit this.

[0067] The technical solution of the embodiment of the present invention obtains the source storage directory of the data to be synchronized, and traverses the directories of each level included in the source storage directory, generates a list of directories to be synchronized corresponding to each directory, obtains at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch in turn, assigns each directory to be synchronized in the current synchronization processing batch to the corresponding synchronization thread, and executes the operation of synchronizing the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized through each synchronization thread, and when it is determined that the number of synchronization tasks of the current synchronization processing batch is equal to the preset number threshold, after waiting for the second thread to sleep for a time, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch and assigning each directory to be synchronized in the new synchronization processing batch to the corresponding synchronization thread is executed again. The technical solution of the embodiment of the present invention can realize batch parallel processing of multiple data synchronization tasks, can reduce the time consumed by data in the synchronization process, and improve the efficiency of data synchronization.

[0068] In order to better introduce the technical solution provided by the embodiment of the present invention, the embodiment of the present invention can refer to the following implementation methods:

[0069] Step 1: Set up the source storage directory on the source server and the target storage directory on the target server.

[0070] The source storage directory is the root directory of the data to be synchronized on the source server; the target storage directory is the root directory of the data to be synchronized on the target server. Assume that the source storage directory is set to src, where: src = / mnt / lustre / asrdata / home / xch27 / ; the target storage directory is set to des, where: des = / mnt / lustre02 / jiangsu / aispeech / home / yj912 / asrdata_yj912 / .

[0071] Step 2: Determine the thread sleep time and the preset number threshold, where the preset number threshold is the maximum number of synchronization tasks in each synchronization processing batch. The thread sleep time includes the first time when the synchronization thread sleeps when the system CPU load is too high, and the interval time between the creation of two adjacent batches of synchronization tasks (i.e., the second time). In this step, the first time can be set to 10s and the second time can be set to 5s.

[0072] Step 3: Set a ratio coefficient ratio, which reflects the ratio of the number of CPU cores that can be used to execute synchronization tasks to the total number of CPU cores. The ratio coefficient can be set to 0.6.

[0073] Step 4: Count the number of all CPU cores in the system, recorded as cpucores.

[0074] Step 5: Query the source storage directory and all sub-directories in it on the source server. The source storage directory and all sub-directories in it constitute a list of directories to be synchronized.

[0075] In this step, the query obtains the list of directories to be synchronized, which consists of the following directories:

[0076] src= / mnt / lustre / asrdata / home / xch27 / ;

[0077] src1= / mnt / lustre / asrdata / home / xch27 / carno2_wav;

[0078] src2= / mnt / lustre / asrdata / home / xch27 / Videos.

[0079] Step 6: Calculate the number of directories to be synchronized in the list of directories to be synchronized obtained in step 5.

[0080] Step 7: traverse each to-be-synchronized directory in the to-be-synchronized directory list, obtain a to-be-synchronized directory as the current to-be-synchronized directory, and assign the current to-be-synchronized directory to the synchronization thread.

[0081] Step 8: Build the target directory corresponding to the current directory to be synchronized.

[0082] In this step, the source storage directory is intercepted to extract the last level directory name (recorded as last_src_path) and the directory path other than the last level directory name (recorded as before_src_path). This can be achieved by the following code:

[0083] last_src_path=$(echo$src|awk-F / '{print$(NF-1)}');

[0084] before_src_path=$(echo${src%$last_src_path*});

[0085] From the above steps, we can see that the value of last_src_path is "xch27". The value of before_src_path is " / mnt / lustre / asrdata / home / ".

[0086] Then replace the content containing before_src_path from the directory to be synchronized with the target storage directory to obtain the target directory (denoted as des_path) corresponding to the directory to be synchronized. This can be achieved through the following code:

[0087] des_path=$(echo$orgin_path|sed"s#${before_src_path}#${des}#");

[0088] Taking the above example, if the current directory to be synchronized (remembered as orgin_path) is:

[0089] / mnt / lustre / asrdata / home / xch27 / carno2_wav;

[0090] The target directory (des_path) corresponding to the current directory to be synchronized is:

[0091] / mnt / lustre02 / jiangsu / aispeech / home / yj912 / asrdata_yj912 / xch27 / carno2_wav.

[0092] Step 9: Execute the load control function.

[0093] The load control function flow is as follows:

[0094] Count the number of currently running synchronous threads, recorded as sysload;

[0095] Calculate the total number of CPU cores used to execute data synchronization tasks, recorded as usage; where:

[0096] usage = cpucores*ratio;

[0097] Determine whether the number of currently running synchronization threads (sysload) is greater than the total number of CPU cores (usage) used to execute data synchronization tasks;

[0098] If yes, it means that the CPU load is too high, and each synchronization thread is controlled to be in a dormant state, and the dormant time is equal to the first time, and then return to step 7;

[0099] If not, determine whether the number of synchronization tasks in the current synchronization processing batch is less than the preset number threshold; if so, return to step 7; if not, control each synchronization thread to be in a dormant state, the dormant time is equal to the second time, and then return to step 7.

[0100] Step 10: Determine whether the current directory to be synchronized is the same as the source storage directory; if so, execute step 11; if not, execute step 12.

[0101] Step 11. Through the synchronization instructions in the remote synchronization tool Rsync, all data in each level directory in the source storage directory (including the data in the source storage directory, and each subdirectory included in the source storage directory and the data in each subdirectory) are stored item by item in a specific directory in the target storage directory for storage verification.

[0102] The directory name of the specific directory in the target storage directory is equal to the last directory name in the source storage directory. This can be achieved by the following code:

[0103] rsync-avP$src$des$last_src_path / >>$src_logs&;

[0104] In this embodiment, the execution process of step 11 does not need to wait for completion before executing step 7.

[0105] Step 12: Synchronize the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized through each synchronization thread.

[0106] The method provided by the embodiment of the present invention can realize batch parallel processing of multiple data synchronization tasks, can reduce the time consumed in the data synchronization process, and improve the efficiency of data synchronization.

[0107] Embodiment 3

[0108] Figure 3This is a structural diagram of a data batch synchronization device provided in Embodiment 3 of the present invention. The device includes: an acquisition module 310 , a synchronization module 320 and a batch synchronization module 330 .

[0109] Among them, the acquisition module 310 is used to obtain the source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each directory; the synchronization module 320 is used to sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute the operation of synchronizing the current level data in each directory to be synchronized to a target directory matching each directory to be synchronized through each synchronization thread; the batch synchronization module 330 is used to, when it is determined that the batch synchronization conditions are met, again execute the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, assign each directory to be synchronized in the new synchronization processing batch to a corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch.

[0110] The technical solution of the embodiment of the present invention obtains the source storage directory of the data to be synchronized, and traverses the directories of each level included in the source storage directory, generates a list of directories to be synchronized corresponding to each directory, obtains at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assigns each directory to be synchronized in the current synchronization processing batch to the corresponding synchronization thread, and executes the operation of synchronizing the current level data in each directory to be synchronized to the target directory matching each directory to be synchronized through each synchronization thread, and when it is determined that the batch synchronization condition is met, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, and assigning each directory to be synchronized in the new synchronization processing batch to the corresponding synchronization thread is executed again. The technical solution of the embodiment of the present invention can realize batch parallel processing of multiple data synchronization tasks, can reduce the time consumed by data in the synchronization process, and improve the efficiency of data synchronization.

[0111] Based on the above embodiments, the acquisition module 310 may include:

[0112] A to-be-synchronized directory generating unit, configured to use the source storage directory as a traversal starting point, traverse the directories of each level included in the source storage directory layer by layer, and generate a to-be-synchronized directory for each directory;

[0113] The to-be-synchronized directory list generating unit is used to combine the generated to-be-synchronized directories to obtain the to-be-synchronized directory list.

[0114] The synchronization module 320 may include:

[0115] A target directory construction unit, configured to construct a target directory matching the directory to be synchronized through the synchronization thread;

[0116] A current-level data synchronization unit, configured to execute an operation of synchronizing the current-level data in the to-be-synchronized directory to the target directory through the synchronization thread;

[0117] An extraction unit, used to extract the last level directory name and the directory path before the last level directory name in the source storage directory;

[0118] a replacement unit, configured to identify the directory path in the directory to be synchronized, and replace the directory path in the directory to be synchronized with a preset target storage directory to obtain the target directory;

[0119] A task updating unit, used to assign a to-be-synchronized directory in the current synchronization processing batch to a synchronization thread, and to update the number of synchronization tasks matching the current synchronization processing batch;

[0120] A judging unit, used to judge whether the number of synchronization tasks in the current synchronization processing batch is less than a preset number threshold;

[0121] The return execution unit is used to return to execute the operation of assigning a to-be-synchronized directory in the current synchronization processing batch to a synchronization thread when the number of synchronization tasks in the current synchronization processing batch is less than a preset number threshold, until the number of synchronization tasks in the current synchronization processing batch is equal to the preset number threshold;

[0122] A synchronization thread number acquisition unit is used to acquire the number of synchronization threads currently running and the target number of CPU cores of the central processing unit; wherein the target number of CPU cores is the total number of CPU cores used to execute the data synchronization task;

[0123] A synchronization thread determination unit, used to determine whether the number of currently running synchronization threads is greater than the target number of CPU cores;

[0124] a first thread sleep time determining unit, configured to determine the first thread sleep time when the number of currently running synchronization threads is greater than the target number of CPU cores, and control the currently running synchronization threads to be in a sleep state within the first thread sleep time;

[0125] A first waiting unit is used for, after waiting for the first thread to sleep for a period of time, for the currently running synchronization thread to again execute the operation of synchronizing the current level data in each of the directories to be synchronized to a target directory matching each of the directories to be synchronized;

[0126] The storage verification unit is used to store all the data in each hierarchical directory in the source storage directory in the target storage directory item by item through the synchronization instructions in the remote synchronization tool Rsync while each synchronization thread executes the synchronization of the current hierarchical data in each directory to be synchronized to the target directory matching each directory to be synchronized, so as to perform storage verification.

[0127] The batch synchronization module 330 includes:

[0128] The second waiting unit is used to, when it is determined that the number of synchronization tasks in the current synchronization processing batch is equal to the preset number threshold, wait for the second thread to sleep for a period of time, and then again execute the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, and assigning each directory to be synchronized in the new synchronization processing batch to a corresponding synchronization thread.

[0129] The data batch synchronization device provided in the embodiment of the present invention can execute the data batch synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0130] Embodiment 4

[0131] Figure 4 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention is shown in FIG. Figure 4 As shown, the computer device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of processors 410 in the computer device can be one or more. Figure 4 A processor 410 is taken as an example; the processor 410, memory 420, input device 430 and output device 440 in the computer device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0132] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to a data batch synchronization method in an embodiment of the present invention (for example, an acquisition module 310, a synchronization module 320 and a batch synchronization module 330 in a data batch synchronization device). The processor 410 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 420, that is, implements the above-mentioned data batch synchronization method. That is, when the program is executed by the processor, it implements:

[0133] Obtain a source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each of the directories;

[0134] Sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute, through each synchronization thread, an operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory matching each directory to be synchronized;

[0135] When it is determined that the batch synchronization conditions are met, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to the corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch.

[0136] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The input device 430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device, and may include a keyboard and a mouse, etc. The output device 440 may include a display device such as a display screen.

[0138] Embodiment 5

[0139] Embodiment 5 of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment of the present invention is implemented. Of course, a computer-readable storage medium provided by an embodiment of the present invention can perform related operations in a data batch synchronization method provided by any embodiment of the present invention. That is, when the program is executed by a processor, the following is implemented:

[0140] Obtain a source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each of the directories;

[0141] Sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute, through each synchronization thread, an operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory matching each directory to be synchronized;

[0142] When it is determined that the batch synchronization conditions are met, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to the corresponding synchronization thread, so as to perform batch synchronization processing on the data in each directory to be synchronized in each synchronization processing batch.

[0143] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0144] It is worth noting that in the above-mentioned embodiment of a data batch synchronization device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0145] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for batch synchronization of data, characterized in that: include: Obtain a source storage directory of the data to be synchronized, and traverse the directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each of the directories; Sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute, through each synchronization thread, an operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory matching each directory to be synchronized; When it is determined that the batch synchronization condition is met, the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to a corresponding synchronization thread, so as to perform batch synchronization processing on the data in each of the directories to be synchronized in each synchronization processing batch; The operation of synchronizing the current level data in each of the directories to be synchronized to the target directory matching each of the directories to be synchronized by each of the synchronization threads includes: Building a target directory matching the directory to be synchronized through the synchronization thread; By means of the synchronization thread, an operation of synchronizing the current level data in the directory to be synchronized to the target directory is performed; The step of constructing a target directory matching the directory to be synchronized through the synchronization thread includes: Extracting the last level directory name and the directory path before the last level directory name in the source storage directory; The directory path is identified in the directory to be synchronized, and the directory path in the directory to be synchronized is replaced with a preset target storage directory to obtain the target directory; wherein the target storage directory is a root directory corresponding to the data to be synchronized on the target server.

2. The method according to claim 1, characterized in that: Traversing the directories of each level included in the source storage directory, generating a list of directories to be synchronized corresponding to each of the directories, including: Taking the source storage directory as the traversal starting point, traverse the directories of each level included in the source storage directory layer by layer, and generate a directory to be synchronized for each directory; The generated directories to be synchronized are combined to obtain the list of directories to be synchronized.

3. The method according to claim 1, characterized in that After executing the operation of synchronizing the current level data in each of the to-be-synchronized directories to the target directory matching each of the to-be-synchronized directories through each of the synchronization threads, the method further includes: Get the number of currently running synchronization threads and the target number of CPU cores of the central processing unit (CPU); wherein the target number of CPU cores is the total number of CPU cores used to execute data synchronization tasks; Determine whether the number of the currently running synchronization threads is greater than the target number of CPU cores, and if so, determine the sleep time of the first thread, and control the currently running synchronization thread to be in a sleep state during the sleep time of the first thread; After waiting for the first thread to sleep for a certain period of time, the currently running synchronization thread again executes the operation of synchronizing the current level data in each of the directories to be synchronized to the target directory matching each of the directories to be synchronized.

4. The method according to claim 1, characterized in that: The step of allocating each to-be-synchronized directory in the current synchronization processing batch to a corresponding synchronization thread includes: Allocate a to-be-synchronized directory in the current synchronization processing batch to a synchronization thread, and update the number of synchronization tasks matching the current synchronization processing batch; Determine whether the number of synchronization tasks in the current synchronization processing batch is less than a preset number threshold; If so, the process returns to executing the operation of assigning a directory to be synchronized in the current synchronization processing batch to a synchronization thread until the number of synchronization tasks in the current synchronization processing batch is equal to the preset number threshold.

5. The method according to claim 4, characterized in that When it is determined that the batch synchronization condition is met, the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch and assigning each directory to be synchronized in the new synchronization processing batch to a corresponding synchronization thread includes: When it is determined that the number of synchronization tasks in the current synchronization processing batch is equal to the preset number threshold, after waiting for the second thread to sleep for a period of time, the operation of obtaining the remaining at least two directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch is executed again, and each directory to be synchronized in the new synchronization processing batch is assigned to the corresponding synchronization thread.

6. The method according to any one of claims 1 to 5, characterized in that: While each synchronization thread executes synchronization of the current level data in each to-be-synchronized directory to the target directory matching each to-be-synchronized directory, it also includes: All data in each level directory in the source storage directory are stored item by item in the target storage directory through the synchronization instruction in the remote synchronization tool Rsync, so as to perform storage verification.

7. A data batch synchronization device, characterized in that: include: An acquisition module, used to acquire a source storage directory of data to be synchronized, and traverse directories of each level included in the source storage directory to generate a list of directories to be synchronized corresponding to each directory; A synchronization module is used to sequentially obtain at least two directories to be synchronized in the list of directories to be synchronized as the current synchronization processing batch, assign each directory to be synchronized in the current synchronization processing batch to a corresponding synchronization thread, and execute, through each synchronization thread, an operation of synchronizing the current hierarchical data in each directory to be synchronized to a target directory matching each directory to be synchronized; The batch synchronization module is used for, when it is determined that the batch synchronization condition is met, again executing the operation of obtaining at least two remaining directories to be synchronized in the list of directories to be synchronized as a new synchronization processing batch, and assigning each directory to be synchronized in the new synchronization processing batch to a corresponding synchronization thread, so as to perform batch synchronization processing on the data in each of the directories to be synchronized in each synchronization processing batch; Wherein, the synchronization module includes: A target directory construction unit, configured to construct a target directory matching the directory to be synchronized through the synchronization thread; A current-level data synchronization unit, configured to execute an operation of synchronizing the current-level data in the to-be-synchronized directory to the target directory through the synchronization thread; An extraction unit, used to extract the last level directory name and the directory path before the last level directory name in the source storage directory; A replacement unit is used to identify the directory path in the directory to be synchronized, and replace the directory path in the directory to be synchronized with a preset target storage directory to obtain the target directory; wherein the target storage directory is a root directory corresponding to the data to be synchronized on the target server.

8. A computer device comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data batch synchronization method as described in any one of claims 1 to 6 when executing the programs.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data batch synchronization method as described in any one of claims 1 to 6 is implemented.

Images