Data synchronization method and device, computer equipment and computer readable storage medium

Abstract

The invention discloses a data synchronization method and device, computer equipment and a computer readable storage medium. According to the data synchronization method and device, first data in to-be-synchronized data can be synchronized to a target data set through at least one data synchronization script, the first data comprises data, belonging to a target time interval of a target index, in the to-be-synchronized data set, and the at least one data synchronization script runs in at least one thread; checking the running state of the data synchronization script in the thread to determine at least one continued thread, then determining at least one continued synchronization script from the plurality of synchronization scripts according to the script identifier corresponding to the at least one data synchronization script and the number of the continued threads, and finally, synchronizing second data in the to-be-synchronized data to the target data set through the splicing synchronization script. Wherein the second data comprises data, belonging to the continuing time interval of the continuing index, in the to-be-synchronized data set. The data synchronization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] With the development of computer technology, data can be stored in a data set of a computer device. For application needs, there is a situation where the data in one data set is synchronized with another data set. The data set can be a database, a file system, etc. The data set can include multiple indexes (also called data tables). The index can save data and its storage time. Specifically, if the data in a database is to be synchronized with a file system, the existing technology usually requires manual operation to trigger the computer device to synchronize data for a specific index. After the data synchronization for the specific index is completed, manual operation is required again to synchronize the computer device with the data in another index. The entire data synchronization process is inefficient and cumbersome. Summary of the invention

[0003] The embodiments of the present application provide a data synchronization method, apparatus, computer device, and computer-readable storage medium, which can improve data synchronization efficiency.

[0004] The present application provides a data synchronization method, including:

[0005] Synchronizing first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval of the target index in the data set to be synchronized, the data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script is respectively run in at least one thread;

[0006] Check the running status of the data synchronization script in the thread to determine at least one continuing thread, and the data synchronization script of the continuing thread has finished running;

[0007] Determine at least one continuing synchronization script from the plurality of synchronization scripts according to the script identifier corresponding to each of the at least one data synchronization script and the number of continuing threads;

[0008] The second data in the data to be synchronized is synchronized to the target data set through the continuation synchronization script; wherein the second data includes the data in the data set to be synchronized that belongs to the continuation time interval of the continuation index.

[0009] Accordingly, an embodiment of the present application further provides a data synchronization device, including:

[0010] A running module, used to synchronize first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval of the target index in the data set to be synchronized, the data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script is respectively run in at least one thread;

[0011] A status module is used to check the running status of the data synchronization script in the thread to determine at least one continuing thread, and the data synchronization script of the continuing thread has finished running;

[0012] A continuation module, used to determine at least one continuation synchronization script from a plurality of synchronization scripts according to the script identifier of at least one data synchronization script and the number of continuation threads;

[0013] The synchronization module is used to synchronize the second data in the data to be synchronized to the target data set through the continuation synchronization script; wherein the second data includes the data in the data set to be synchronized that belongs to the continuation time interval of the continuation index.

[0014] In some embodiments of the present application, the data synchronization device further includes a document continuation module, wherein:

[0015] A continuation document module, used to save the script identifier of at least one data synchronization script in a preset continuation document;

[0016] At this time, the continuation module includes a first identification submodule, a continuation identification submodule and a continuation script submodule, wherein:

[0017] A first identification submodule, used to determine a first script identification from at least one script identification of a preset continuation document;

[0018] A continuation identification submodule, used to determine at least one continuation script identification according to the first script identification and the number of continuation threads;

[0019] The continuation script submodule is used to determine at least one continuation synchronization script corresponding to the continuation script identifier from multiple synchronization scripts.

[0020] In some embodiments of the present application, the script identifier of the data synchronization script includes a target index identifier and a target time interval, the target time interval includes a target day interval and a target time interval, and the first identification submodule is specifically used to:

[0021] determining a day-time relationship between target day intervals in at least one script identifier, and determining at least one initial script identifier from the at least one script identifier based on the day-time relationship;

[0022] Determine an identification relationship between target index identifications in at least one initial script identification, and determine at least one intermediate script identification from the initial script identification based on the identification relationship;

[0023] A temporal relationship between target time intervals in at least one intermediate script identifier is determined, and a first script identifier is determined from the intermediate script identifiers based on the temporal relationship.

[0024] In some embodiments of the present application, the connection identification submodule includes a connection time interval unit, a connection identification unit and a return unit, wherein:

[0025] A connection time interval unit, configured to determine the connection time interval based on a preset time interval granularity and a target time interval in the first script identifier;

[0026] A continuation identification unit, used to determine a continuation script identifier according to the continuation time interval and the target index identifier in the first script identifier;

[0027] The return unit is used to determine that the continuation script identifier is a new first script identifier, and return to execute the step of determining the continuation time interval based on the preset time interval granularity and the target time interval in the first script identifier, until the number of continuation script identifiers matches the number of continuation threads.

[0028] In some embodiments of the present application, the continuation identification unit includes a matching subunit and a non-matching subunit, wherein:

[0029] A matching subunit, configured to determine a continuation script identifier based on the continuation time interval, the target day interval of the first script identifier, and the target index identifier if the continuation time interval matches the preset time interval;

[0030] The mismatch subunit is used to determine the continuation index identifier based on the target index identifier in the first script identifier if the continuation time interval does not match the preset time interval, and to determine the continuation script identifier based on the continuation index identifier and the first script identifier.

[0031] In some embodiments of the present application, the data to be synchronized corresponds to an index identification file, and the index identification file includes at least one target index identification; the mismatch subunit is specifically used for:

[0032] If the continuation time interval does not match the preset time interval, determining the continuation index identifier based on the target index identifier in the first script identifier;

[0033] If the continuation index identifier belongs to the index identifier file, the continuation script identifier is determined based on the default time interval, the continuation index identifier and the target day interval in the first script identifier;

[0034] If the continuation index identifier does not belong to the index identifier file, the continuation date interval is determined based on the target date interval in the first script identifier, and the continuation script identifier is determined based on the continuation date interval, the default time interval and at least one target index identifier in the index identifier file.

[0035] In some embodiments of the present application, the state module is specifically used to:

[0036] After a preset time period, checking the running status of the data synchronization script in the thread to determine at least one running synchronization script and at least one continuing thread in the running state;

[0037] The data synchronization device also includes a determination module and a return module, wherein:

[0038] A determination module, used to determine the continued synchronization script and the running synchronization script as a new data synchronization script;

[0039] The return module is used to return to the step of checking the running status of the data synchronization script in the thread after a preset time period to determine at least one running synchronization script and at least one continuing thread in the running state until the data synchronization process for the data to be synchronized is completed.

[0040] In some embodiments of the present application, the data synchronization device further includes an information module and a generation module, wherein:

[0041] An information module, used to determine an index identification file and a total time interval corresponding to the data to be synchronized in the data set to be synchronized, wherein the index identification file includes at least one target index identification;

[0042] The generation module is used to obtain a synchronization template and generate multiple synchronization scripts based on at least one target index identifier, a total time interval and the synchronization template.

[0043] In some embodiments of the present application, the total time interval is determined by a first time point and a second time point, and the generation module includes an acquisition submodule, an index number submodule, a time point submodule, a return submodule and a storage submodule, wherein:

[0044] Get submodule, used to get synchronization template;

[0045] An index number submodule, used to determine the index number corresponding to each target index identifier in the index identifier file according to the position information of the target index identifier in the index identifier file;

[0046] A time point submodule, used to determine a target time interval and an intermediate time point based on a preset segmentation granularity and a first time point;

[0047] A return submodule, for determining the intermediate time point as a new first time point if the intermediate time point does not match the second time point, and returning to execute the step of determining the target time interval and the intermediate time point based on the preset segmentation granularity and the first time point;

[0048] The saving submodule is used to save the index number and the target time interval at a preset position of the synchronization template if the intermediate time point matches the second time point, so as to obtain multiple synchronization scripts.

[0049] In some embodiments of the present application, the index identification file includes multiple lines, each line stores a target index identification, and the index number submodule is specifically used for:

[0050] Determine the line number of the line where the target index mark is located in the index mark file;

[0051] The index number of the target index mark is obtained by calculation based on the row number and the preset value.

[0052] In some embodiments of the present application, the data synchronization device is also used to:

[0053] When the synchronization script determines that the data has been synchronized to the target data set, a synchronization log for the data is added to the preset log file;

[0054] If the synchronization script determines that the data has not been synchronized to the target dataset, resynchronize the data to the target dataset.

[0055] In some embodiments of the present application, the data synchronization device further includes:

[0056] The quantity statistics module is used to count the number of data to be synchronized in the data set to be synchronized, and the number of synchronization logs in the preset log file;

[0057] An analysis information module is used to generate synchronization analysis information based on the number of items to be synchronized and the number of synchronization logs;

[0058] The attribute information module is used to determine the attribute information of the data being synchronized in the data set to be synchronized, and send synchronization analysis information and attribute information to the target object.

[0059] In some embodiments of the present application, the synchronization analysis information includes a synchronization success rate and a synchronization result, and the analysis information module is specifically used to:

[0060] The number of synchronization targets and the number of synchronization logs are combined to obtain the synchronization success rate.

[0061] The number of items to be synchronized and the number of synchronization logs are compared, and the synchronization result is determined based on the comparison result.

[0062] In some embodiments of the present application, the attribute information includes the data synchronization status of the data, the target index to which the data belongs, and the target time interval. The attribute information module is specifically used to:

[0063] Get the target index and target time interval of the data being synchronized in the data set to be synchronized;

[0064] Query the script ID of the synchronous script running in the thread;

[0065] If the script identifier matches the target index and the target time interval, the data synchronization state is determined to be the first state;

[0066] If the script identifier does not match at least one of the target index and the target time interval, determining that the data synchronization state is the second state;

[0067] Send synchronization analysis information, data synchronization status, target index to which the data belongs, and target time interval to the target object.

[0068] Accordingly, an embodiment of the present application also provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the data synchronization method provided in the embodiment of the present application.

[0069] Accordingly, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to implement the steps in the data synchronization method provided in the embodiment of the present application.

[0070] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which are executed by a processor to implement the steps in the data synchronization method provided in the embodiment of the present application.

[0071] In an embodiment of the present application, the first data in the data to be synchronized can be synchronized to the target data set through at least one data synchronization script. The data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script runs in at least one thread respectively. For a data synchronization script, the corresponding first data is the data in the target time interval of the data set to be synchronized that belongs to the target index, and the target index and the target time interval serve as part of the script identifier of the data synchronization script.

[0072] The embodiment of the present application can check the running status of the data synchronization script in the thread. If the data synchronization script is in the running status, it can be determined that the data synchronization script is still performing data synchronization. If the data synchronization script is finished running, the data synchronized by the data synchronization script has been synchronized, and the thread where the data synchronization script is located is empty, and this thread is determined to be a continuation thread. Since the script identifier includes the target index and target time interval corresponding to the synchronization script, the target index and target time interval to which the data being synchronized or having been synchronized belong can be determined based on the script identifier of at least one data synchronization script. Then, combined with the number of continuation threads, at least one continuation synchronization script can be determined from multiple synchronization scripts, and the continuation synchronization script can be run in the continuation thread. Therefore, the present application can perform data synchronization efficiently and coherently, and effectively improve the efficiency of synchronizing the data to be synchronized to the target data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 is a scenario diagram of a data synchronization method provided in an embodiment of the present application;

[0075] Figure 2 It is a flowchart of a data synchronization method provided in an embodiment of the present application;

[0076] Figure 3 is another flowchart of the data synchronization method provided by an embodiment of the present application;

[0077] Figure 4 is another flowchart of the data synchronization method provided by an embodiment of the present application;

[0078] Figure 5 is another flowchart of the data synchronization method provided by an embodiment of the present application;

[0079] Figure 6 is another flowchart of the data synchronization method provided by an embodiment of the present application;

[0080] Figure 7 is another flowchart of the data synchronization method provided by an embodiment of the present application;

[0081] Figure 8 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0082] Fig. 9It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0084] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0085] It is understandable that in several embodiments of the present application, related data such as user information (such as data to be synchronized, etc.) is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0086] The embodiments of the present application provide a data synchronization method, apparatus, computer device and computer-readable storage medium. The data synchronization apparatus can be integrated into a data synchronization system, and the data synchronization system can be integrated into at least one computer device, which can include at least one of a terminal and a server.

[0087] The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart conference device, a smart screen, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0088] The data set in this application (including the data set to be synchronized, the target data set, etc.) may include a collection of data stored in multiple forms, such as a database, a file system, etc. The data set may include multiple indexes, and the indexes may have the same structure, such as the number and type of fields in the index. The index may store information such as data and the storage time of the data. The database may be Mysql (a database), Elasticsearch (a database), etc. The file system may include a distributed file system, such as a Hadoop distributed file system, etc. The data set may be set in at least one computer device, which may be determined according to the actual situation, and this should not be understood as a limitation to this application.

[0089] The data synchronization method of the present application can be integrated in a data synchronization system. At least one computer device integrated with the data synchronization system can be partially or completely identical to at least one computer device integrated with the data set to be synchronized. The two can also be completely different computer devices. The specific configuration can be flexibly based on the actual application scenario.

[0090] In an embodiment of the present application, a computer device can synchronize first data in the data to be synchronized to a target data set through at least one data synchronization script, the first data including data in a target time interval belonging to a target index in the data set to be synchronized, the data synchronization script can be determined from multiple synchronization scripts, at least one data synchronization script can be respectively run in at least one thread, the running status of the data synchronization script in the thread is checked to determine at least one continuation thread, the data synchronization script of the continuation thread has finished running, at least one continuation synchronization script is determined from multiple synchronization scripts according to the script identifier of at least one data synchronization script and the number of continuation threads, and second data in the data to be synchronized to the target data set is synchronized through the continuation synchronization script, the second data including data in a continuation time interval belonging to the continuation index in the data set to be synchronized.

[0091] For example, see Figure 1, the data set to be synchronized is set on server 1, and the target data set is set on server 2. Server 1 can run data synchronization scripts in 20 threads respectively. The script identifier of the data synchronization script includes a target index and a target time interval. The data synchronization script can be used to synchronize the data in the data set to be synchronized that belongs to the target index in the target time interval to the target data set. Server 1 can check the running status of the data synchronization script in the thread, and then determine the continuation threads of 5 data synchronization scripts that have finished running, and determine 5 continuation synchronization scripts from multiple synchronization scripts according to the script identifiers of the 20 data synchronization scripts and the number of continuation threads. The continuation synchronization script can be used to synchronize the data in the data set to be synchronized that belongs to the continuation index in the continuation time interval to the target data. The data in the continuation index in the continuation time interval is data that has not been synchronized. At least one continuation synchronization script runs in at least one continuation thread respectively.

[0092] Figure 1 This is an example of an application scenario of the data synchronization system of the present application, which is mainly used to introduce the data synchronization system of the present application. In the process of actually applying the technical solution described in the embodiment of the present application, the computer devices included in the data synchronization system and the steps performed by each computer device can be flexibly adjusted, and are not limited to Figure 1 The content described in .

[0093] The data synchronization method of the present application will be further described below in conjunction with the embodiments. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0094] Figure 2 A flow chart of the data synchronization method of the present application is shown as follows: Figure 2 , data synchronization methods may include:

[0095] 110. Synchronize the first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval of the data set to be synchronized that belongs to the target index, and the data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script runs in at least one thread respectively.

[0096] The data synchronization script may include a script for implementing a data synchronization function. Specifically, the data synchronization script may synchronize data in a to-be-synchronized data set to a target data set. The to-be-synchronized data set may include multiple indexes, and the index may store data and the storage time of the data. The data synchronization script may correspond to a target index and a target time interval. The data synchronization script may specifically synchronize data in the to-be-synchronized data set that belongs to the target index and whose storage time belongs to the target time interval to the target data set. The script identifier of the data synchronization script may include a target index and a target time interval.

[0097] The data synchronization script can be determined from multiple synchronization scripts. All the data to be synchronized in the data set to be synchronized can be synchronized to the target data set through multiple synchronization scripts. The data to be synchronized can be part or all of the data in the data set to be synchronized. It is known that the data set to be synchronized includes multiple indexes, and the index stores the data and its storage time. Therefore, the data to be synchronized in the data set to be synchronized can be determined by determining at least one target index identifier and several target time intervals (or the total time interval composed of several target time intervals).

[0098] In some embodiments of the present application, the index identification file can save at least one target index identification, and the index identification file and the total time interval corresponding to the data to be synchronized in the data set to be synchronized can be obtained first, and the synchronization template can also be obtained. The synchronization template may include a template for generating a synchronization script, and then multiple synchronization scripts can be generated based on at least one target index identification, the total time interval and the synchronization template.

[0099] Specifically, the index identification file and the total time interval may be obtained from a local computer device or other computer devices. The index identification file and the total time interval may be determined manually or generated by other business programs based on demand, and so on.

[0100] Based on the target index identifier, the total time interval and the synchronization template, there are many ways to generate multiple synchronization scripts. For example, the data set to be synchronized may include multiple indexes, and the index may include multiple sub-indexes. Multiple target sub-index identifiers can be determined according to the target index identifier, and the target sub-index identifiers and the total time interval are respectively saved in the preset position of the synchronization template to obtain multiple synchronization scripts. For example, index identifier file 1 is obtained, and index identifier file 1 includes target index identifier 1 and target index identifier 2. The total time interval A is obtained, and the index identifiers of the three sub-indexes of target index 1 are determined according to the target index identifier 1. : Target sub-index identifier 11, target sub-index identifier 12 and target sub-index identifier 13. According to target index identifier 2, the index identifiers of the two sub-indexes of target index 2 are determined: target sub-index identifier 21 and target sub-index identifier 22. The synchronization template includes a preset identifier position for storing index identifiers and a preset interval position for storing time intervals. Target sub-index identifier 11, target sub-index identifier 12, target sub-index identifier 13, target sub-index identifier 21 and target sub-index identifier 22 are respectively stored in the preset identifier positions, and the total time interval A is stored in the preset interval position to obtain 5 synchronization scripts.

[0101] In some embodiments of the present application, the total time interval can also be divided into multiple target time intervals, and then multiple synchronization scripts can be generated according to the multiple target time intervals and at least one target index identifier. For example, the total time interval can be determined by the first time point and the second time point, and the total time interval can be divided according to the preset segmentation granularity to obtain multiple target time intervals. The preset segmentation granularity can be determined according to actual needs, such as 1 hour, 2 hours and 3 minutes, 50 seconds, 2 days, etc. For example, the preset segmentation granularity can be 1 hour, and the total time interval can be 1:00-22:00 of Date 1, then 22 target time intervals can be obtained by segmentation, namely 1:00-2:00 of Date 1, ..., 21:00-22:00 of Date 1.

[0102] Specifically, the segmentation can be achieved in a cyclic manner. For example, a target time interval and an intermediate time point can be determined based on the preset segmentation granularity and the first time point. Specifically, the preset segmentation granularity can be added to the first time point to obtain the intermediate time point. The first time point and the intermediate time point determine the target time interval. For example, the first time point is 1 o'clock on date 1, and the preset segmentation granularity is 1 hour, then the intermediate time point is 2 o'clock on date 1, and the target time interval is 1 o'clock - 2 o'clock on date 1.

[0103] Then determine whether the middle time point matches the second time point. The match can be, for example, equality, a difference less than a preset threshold, etc., which can be flexibly configured according to actual needs. If not, it indicates that the total time interval can be divided again, and the middle time point can be determined as the new first time point. Return to execute the step of determining the middle time point and the target time interval based on the preset segmentation granularity and the first time point; if matched, it indicates that the total time interval has been divided and at least one target time interval is obtained. For example, if the intermediate time point 2:00 on Date 1 is less than the second time point 22:00 on Date 1, the intermediate time point 2:00 on Date 1 can be determined as the new first time point, and based on the preset segmentation granularity and this first time point, the intermediate time point 3:00 on Date 1 and the target time interval 2:00-3:00 on Date 1 are obtained, and it is determined again whether the intermediate time point matches the second time point, and this process is repeated until the intermediate time point 22:00 on Date 1 is obtained, and it is determined that this intermediate time point is equal to the second time point, and the loop is determined to end, and 22 target time intervals are obtained, namely 1:00-2:00 on Date 1, ..., 21:00-22:00 on Date 1.

[0104] The synchronization template includes a preset identification position for storing index identification and a preset interval position for storing time intervals. Finally, the target time intervals can be saved in the preset interval positions in sequence, and the target index identification can be saved in the preset identification positions in sequence to obtain multiple synchronization scripts. For example, 22 target time intervals are saved in the preset interval positions, and 2 target index identifications are saved in the preset identification positions, and finally 44 synchronization scripts are obtained.

[0105] In some embodiments of the present application, the index identifier may include at least one character. In order to enhance the connection between multiple target index identifiers, the target index identifier may be converted into an index number. The index number may include at least one character. The index number is determined based on specific rules, so that when one index number is known, other index numbers can be determined. For example, the target index identifiers can be sorted, and the sorting result of each target index identifier can be determined as the index number. For example, if the sorting result of target index identifier 5 among all target index identifiers is 032, then the index number is 032.

[0106] In some embodiments of the present application, the index number corresponding to the target index identifier can be determined based on the position information of the target index identifier in the index identifier file. For example, the index identifier file may include a table with M rows and N columns (M and N are both positive integers), each cell can save a target index identifier, and the position of the cell where the target index identifier is located can be determined as the index number.

[0107] For another example, the index identifier file may include multiple lines, each line may include a target index identifier, the line number of the line where the target index identifier is located may be determined, and then the line number and the preset value may be combined and calculated, such as the line number minus the preset value, and the obtained value is the index number of the target index identifier. For example, the line number of the line where the target index identifier 4 is located is 010, and the preset value is 1, and the line number minus the preset value may obtain the index number 009 of the target index identifier 4.

[0108] In some embodiments of the present application, in order to improve the efficiency and success rate of data synchronization, and to accurately record the data synchronization status, the synchronization script can also, when it is determined that the data has been synchronized to the target data set, add a synchronization log for this data in a preset log file; when the synchronization script is determined that the data has not been synchronized to the target data set, the data can be re-synchronized to the target data set. The synchronization script can also record the number of synchronization times for the data. If the number of synchronization times exceeds a preset threshold, data synchronization will no longer be performed for the data, and a failure log for the data will be recorded in a preset failure file.

[0109] 120. Check the running status of the data synchronization script in the thread to determine at least one continuing thread, and the data synchronization script of the continuing thread has finished running.

[0110] The embodiment of the present application includes multiple threads, and the running status of the data synchronization script in each thread can be determined respectively. If the data synchronization script in a thread ends, it can be determined that the thread is a continuation thread. If the data synchronization script in the thread is in a running state, the data synchronization script is determined to be a running synchronization script. Finally, at least one continuation thread and at least one running synchronization script are determined.

[0111] The application checks the running status of the data synchronization script in the thread and can be performed regularly at a set time interval. The set time interval can be flexibly set according to actual needs, such as 3 seconds, 1 minute, 50 milliseconds, etc.

[0112] 130. Determine at least one subsequent synchronization script from multiple synchronization scripts according to the script identifier of at least one data synchronization script and the number of subsequent threads.

[0113] Among them, the script identifier may include the target index identifier and target time interval corresponding to the data synchronization script. Through the script identifiers of multiple data synchronization scripts, at least one target index identifier and at least one target time interval corresponding to the data being synchronized or synchronized can be determined, and then the continuation script identifier is determined according to the rules between known index identifiers (such as determining the index number based on the position of the target index identifier in the index identifier file) and the rules between time intervals (such as dividing based on a preset segmentation granularity), and then the continuation script identifier is determined, and then the continuation synchronization script corresponding to the continuation script identifier is determined from multiple synchronization scripts, thereby automatically realizing the continuation of the synchronization script, and then performing data synchronization in a coherent and efficient manner.

[0114] Specifically, based on the script identifier of at least one data synchronization script and the number of continuation threads, the process of determining at least one continuation synchronization script from multiple synchronization scripts may include multiple methods. For example, the continuation script identifier may be determined, and at least one continuation synchronization script corresponding to the continuation script identifier may be determined from multiple synchronization scripts.

[0115] It should be noted that, in this process, the script identifier may include a target index identifier, and the target index identifier here may also be understood as an index number corresponding to the target index identifier.

[0116] In some embodiments of the present application, a first script identifier can be determined from the script identifier of at least one data synchronization script. The first script identifier can be used to determine the continuation script identifier. The script identifiers of multiple synchronization scripts can be determined based on an identification rule. The identification rule can make a front-end relationship exist between the script identifiers. The first script identifier is the script identifier on the boundary of the script identifiers of the multiple data synchronization scripts. Then, the continuation script identifier is determined based on the first script identifier and the number of continuation threads, and then at least one continuation synchronization script corresponding to the continuation script identifier is determined from the multiple synchronization scripts.

[0117] Specifically, the process of determining the first script identifier may include multiple methods. For example, multiple script identifiers may be sorted to obtain two boundary script identifiers at the boundary, and then the first script identifier may be selected from the boundary script identifiers.

[0118] In some embodiments of the present application, the target time interval may include a target time interval and a target day interval, and the script identifiers of multiple data synchronization scripts (referred to as multiple script identifiers) each include a target index identifier and a target time interval. Then, the daily time relationship between the target day intervals in the multiple script identifiers can be first determined. The daily time relationship can characterize the time relationship between each target day interval, and at least one initial script identifier can be determined from the multiple script identifiers based on the daily time relationship. If the number of initial script identifiers is 1, the initial script identifier is the first script identifier.

[0119] For example, the daily time relationship between the target day intervals in the 20 script identifiers can be determined. The daily time relationship can determine that the day interval closest to the current one is target day interval 3. Target day interval 3 only belongs to script identifier 3. It is determined that the initial script identifier only includes script identifier 3, and then script identifier 3 is determined to be the first script identifier.

[0120] If the number of initial script identifiers is greater than 1, determine the identification relationship between the target index identifiers in at least two initial script identifiers. The target index identifier can be determined based on a preset rule so that different index identifiers have a front-and-back relationship. The identification relationship can characterize the front-and-back relationship between the target index identifiers. Based on the identification relationship, at least one intermediate script identifier can be determined from at least two initial script identifiers. If the number of intermediate script identifiers is 1, the intermediate script identifier is the first script identifier.

[0121] For example, the daily time relationship between the target daily intervals in the 20 script identifiers can be determined. The daily time relationship can determine that the daily interval closest to the current one is target daily interval 8, and target daily interval 8 belongs to script identifier 8, script identifier 10, and script identifier 11. The identification relationship between the target index identifiers in script identifier 8, script identifier 10, and script identifier 11 is determined. The identification relationship can determine that the last index identifier is target index identifier 4, and target index identifier 4 only belongs to script identifier 10. Script identifier 10 is an intermediate script identifier, and script identifier 10 is further determined as the first script identifier.

[0122] If the number of intermediate script identifiers is greater than 1, determine the time relationship between the target time intervals in at least two intermediate script identifiers. The time relationship can characterize the temporal relationship between the target time intervals. Based on the time relationship, the first script identifier can be determined from at least two intermediate script identifiers.

[0123] For example, the identification relationship can determine that the last index identifier is the target index identifier 5, and the target index identifier 4 belongs to script identifier 8 and script identifier 11. The time relationship between the target time intervals in script identifier 8 and script identifier 11 is determined. The time relationship can determine the target time interval 2 closest to 23:00. The target time interval 2 belongs to script identifier 8, and script identifier 8 is determined as the first script identifier.

[0124] In some embodiments of the present application, the target time interval may include a target time interval and a target day interval, and the script identifiers of each of the multiple data synchronization scripts (referred to as multiple script identifiers) include a target index identifier and a target time interval. The day-time relationship between the target day intervals in the multiple script identifiers can also be determined, and based on the day-time relationship, at least one initial script identifier is determined from the multiple script identifiers, the identifier relationship between the target index identifiers in the at least one initial script identifier is determined, and based on the identifier relationship, at least one intermediate script identifier is determined from the initial script identifier, the time-time relationship between the target time intervals in the at least one intermediate script identifier is determined, and based on the time-time relationship, the first script identifier is determined from the intermediate script identifier.

[0125] In some embodiments of the present application, the script identifier may also include information determined based on set rules to characterize certain aspects of the synchronization script, such as time intervals, the amount of syncable data, etc. This information may also be introduced in the process of determining the first script identifier. Please refer to the above text for details, and flexibly expand it in combination with the actual application scenario. This application will not elaborate on this.

[0126] In some embodiments of the present application, a continuation time interval can be first determined based on a preset time interval granularity and a target time interval in a first script identifier, and then a continuation script identifier can be determined based on the continuation time interval and a target index identifier in the first script identifier, and then the continuation script identifier is determined as a new first script identifier, and the step of determining the continuation time interval based on the preset time interval granularity and the target time interval in the first script identifier is returned to execute until the number of continuation script identifiers matches the number of continuation threads, and the match here may include equal, less than, the difference is less than a preset threshold, etc., which can be flexibly determined according to actual conditions.

[0127] The preset time interval granularity here should match the preset segmentation granularity to ensure that the process of generating the synchronization script and the process of determining the continuation synchronization script are processed based on the same rules, thereby obtaining the continuation script identifier existing in the script identifiers of multiple synchronization scripts.

[0128] In some embodiments, based on the target index identifier in the continuation time interval and the first script identifier, the process of determining the continuation script identifier may include: first determining whether the continuation time interval matches the preset time interval, the match may include whether they are equal, whether the continuation time interval belongs to the preset time interval, whether the difference between the two is less than the preset time interval threshold, etc. The preset time interval may be a time interval directly set in advance, or a time interval determined in advance based on the total time interval, etc. The preset time intervals corresponding to different target day intervals may be the same or different, and can be flexibly handled according to actual conditions.

[0129] If they match, the continuation script identifier can be determined based on the continuation time interval, the day interval of the first script identifier and the target index identifier. For example, if the continuation time interval is 4:00-5:00 and the preset time interval is 0:00-24:00, it can be determined that the continuation time interval belongs to the preset time interval. If the two match, the target time interval in the first script identifier is replaced with the continuation time interval to obtain the continuation script identifier.

[0130] If there is no match, the continuation index identifier can be determined based on the target index identifier in the first script identifier, and then the continuation script identifier can be determined based on the continuation index identifier and the first script identifier. For example, if the continuation time interval is 24:00-25:00 and does not belong to the preset time interval 0:00-24:00, and the two do not match, then the target index identifier 004 in the first script identifier is obtained, and the continuation index identifier 005 is determined based on this, and then the continuation script identifier is determined based on the continuation index identifier 005 and the first script identifier.

[0131] In some embodiments of the present application, based on the continuation index identifier and the first script identifier, the process of determining the continuation script identifier may include first determining whether the continuation index identifier belongs to the index identifier file, and if so, determining the continuation script identifier based on the default time interval, the continuation index identifier, and the target day interval in the first script identifier. The default time interval may be pre-set, such as 0 hours to 1 hours. For example, it is determined that the continuation index identifier 005 belongs to the index identifier file, the target time interval in the first script identifier is replaced with the default time interval, and then the target index identifier therein is replaced with the continuation index identifier 005 to obtain the continuation index identifier.

[0132] If the continuation index identifier does not belong to the index identifier file, the continuation day interval can be determined based on the target day interval in the first script identifier, and then the continuation script identifier can be determined based on the continuation day interval, the default time interval and the target index identifier in the index identifier file. Specifically, it can be determined whether the continuation day interval belongs to the total time interval. If it does, the continuation script identifier is determined based on the continuation day interval, the default time interval and the first or last target index identifier in the index identifier file; if the continuation day interval does not belong to the total time interval, it is determined that all synchronization scripts are in operation or have ended, and no continuation script identifier is generated.

[0133] 140. Synchronize second data in the data to be synchronized to the target data set through a continuation synchronization script; wherein the second data includes data in the data set to be synchronized that belongs to a continuation time interval of the continuation index.

[0134] Therefore, the present application can directly and automatically perform data synchronization continuously without manual operation, thereby efficiently synchronizing the data to be synchronized in the data set to be synchronized to the target data set.

[0135] In some embodiments of the present application, the number of synchronization scripts is much larger than the number of threads. Therefore, the running status of the data synchronization script in the thread can be checked at a set time interval, and then the subsequent synchronization script can be determined and run in the subsequent thread. This process is repeated until all synchronization scripts are in a running state or have finished running, that is, the data synchronization process for all data to be synchronized is completed.

[0136] In some embodiments of the present application, the synchronous continuation script can also be used to execute the running status of the data synchronization script in the check thread to determine at least one continuation thread. The data synchronization script of the continuation thread has finished running. According to the script identifiers corresponding to the multiple data synchronization scripts and the number of continuation threads, at least one continuation synchronization script is determined from the multiple synchronization scripts, and the continuation synchronization script is run in the continuation thread. The synchronous continuation script is started regularly, that is, automatic continuation is realized, and data synchronization can be automatically and efficiently performed.

[0137] In some embodiments of the present application, relevant information of the data synchronization process may also be sent to the target object, so that the target object can better understand information such as the data synchronization progress and data synchronization effect.

[0138] For example, the script identifiers of the data synchronization scripts running in the thread can be obtained periodically, and these multiple script identifiers can be sent to the target object. Since the script identifier includes the target index identifier and the target time interval, the target object can understand the status of the data currently being synchronized through multiple script identifiers.

[0139] For example, the number of data to be synchronized in the data set to be synchronized and the number of synchronization logs in the preset log file can be counted, and synchronization analysis information can be generated based on the number of data to be synchronized and the number of synchronization logs, the attribute information of the data being synchronized in the data set to be synchronized can be determined, and the synchronization analysis information and attribute information can be sent to the target object.

[0140] Among them, the synchronization analysis information may include the synchronization success rate and the synchronization result. Specifically, the number of synchronization objects and the number of synchronization logs may be fused and calculated to obtain the synchronization success rate, and the number of synchronization objects and the number of synchronization logs may be compared, and the synchronization result may be determined based on the comparison result.

[0141] For example, the number to be synchronized is A, the number of synchronization logs is B, and the synchronization success rate is A / B. If A is equal to B, the synchronization result is determined to be successful. If A is not equal to B, the synchronization result is determined to be unsuccessful.

[0142] Among them, the attribute information may include the data synchronization status, the target index and target time interval to which the data belongs. Specifically, the script identifier of the synchronization script running in the thread can be queried. If the script identifier matches both the target index and the target time interval, the data synchronization status is determined to be the first state; if the script identifier does not match at least one of the target index and the target time interval, the data synchronization status is determined to be the second state.

[0143] For example, it is determined that the data being synchronized belongs to target index 1 and target time interval 2, and it is determined that the script identifier of the data synchronization script running in the thread includes target indexes 4-9 and target time intervals 2-6. Comparison shows that the script identifier does not match the target index 1 and target time interval 2, and the data synchronization status is determined to be a failure status.

[0144] In an embodiment of the present application, the first data in the data to be synchronized can be synchronized to the target data set through at least one data synchronization script. The data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script runs in at least one thread respectively. For a data synchronization script, the corresponding first data is the data in the target time interval of the data set to be synchronized that belongs to the target index, and the target index and the target time interval serve as part of the script identifier of the data synchronization script.

[0145] The embodiment of the present application can check the running status of the data synchronization script in the thread. If the data synchronization script is in the running status, it can be determined that the data synchronization script is still performing data synchronization. If the data synchronization script is finished running, the data synchronized by the data synchronization script has been synchronized, and the thread where the data synchronization script is located is empty, and this thread is determined to be a continuation thread. Since the script identifier includes the target index and target time interval corresponding to the synchronization script, the target index and target time interval to which the data being synchronized or having been synchronized belong can be determined based on the script identifier of at least one data synchronization script. Then, combined with the number of continuation threads, at least one continuation synchronization script can be determined from multiple synchronization scripts, and the continuation synchronization script can be run in the continuation thread. Therefore, the present application can perform data synchronization efficiently and coherently, and effectively improve the efficiency of synchronizing the data to be synchronized to the target data set.

[0146] The data synchronization method of the present application will be further described below in conjunction with the embodiments. Figure 3 A flow chart of the data synchronization method of the present application is shown as follows: Figure 3 Data synchronization methods can include:

[0147] 210. The computer device determines an index identification file and a total time interval corresponding to the data to be synchronized in the data set to be synchronized, where the index identification file includes at least one target index identification.

[0148] The index identifier file may include multiple lines, one line may store one target index identifier, and the index number corresponding to the target index identifier may be determined according to the line number of the line where the target index identifier is located. For example, the index number may be the line number minus the value 1. All index numbers are consecutive integers.

[0149] The total time interval can be determined by the start time point and the end time point. The minimum granularity of the total time interval can be hours, and the start time point can include the start day and the start hour. For example, the total time interval can be 3:00 on date 1 to 23:00 on date 2.

[0150] 220. The computer device obtains a synchronization template, and generates a plurality of synchronization scripts based on at least one target index identifier, a total time interval, and the synchronization template, wherein the synchronization scripts correspond to the index identifier and the time interval.

[0151] The synchronization template may include a preset identification position, a preset day interval position, and a preset time interval position, which are used to store the index number, the target day interval, and the target time interval, respectively.

[0152] Fill the start date and the minimum index number into the synchronization template, determine a time interval according to the start hour, and the time interval can be a time interval determined by adding 1 hour to the start hour, such as 3-4 hours, and fill the time interval into the synchronization template to obtain a synchronization script, and then determine a new time interval and fill it into the synchronization template to obtain a new synchronization script, and repeat this process until the time interval is 23-24 hours, keep the start date in the synchronization template unchanged, add 1 to the index number, and determine the time interval to be 0-1 hours, and obtain a new synchronization script, and repeat the process of determining the new time interval again to obtain a new synchronization script until the time interval is 23-24 hours, and repeat the process of changing the index number until the index number is the maximum index number, at this time, add 1 to the start date, and the index number returns to the minimum index number, and the time interval is 0-1 hours, and repeat this process until the new synchronization script includes the end date, end hour and maximum index number, and end this process to obtain multiple synchronization scripts. Each synchronization can be distributed to its corresponding execution directory so that the synchronization script can be quickly run later.

[0153] The data set to be synchronized in the embodiment of the present application may be a database, and the target data set may be a distributed file system. The database may include multiple indexes, and the index may store data and the storage time of the data, etc. The synchronization script may query the database for the data stored in the corresponding index of its corresponding target index identifier, and synchronize it to the distributed file system.

[0154] The synchronization script can also be used to retry when data synchronization fails, and to generate a failure log when the retries exceed 3 times. The synchronization script can also be used to generate a success log when data synchronization succeeds.

[0155] The script identifier of the synchronization script may include a target time interval and an index number. For example, the script identifier may be “es_to_tdbank_year-month-day-index number_hour.py”.

[0156] Steps 210-220 can be implemented by batch-producing synchronization scripts to obtain multiple synchronization scripts. The process can also be seen in Figure 4 .

[0157] 230. The computer device synchronizes the data whose data index identifier in the to-be-synchronized data set corresponds to the index and whose storage time is within the data time interval to the target data set through a data synchronization script. The data synchronization script is determined from multiple synchronization scripts, and the multiple data synchronization scripts are respectively run in multiple threads.

[0158] The data index identifier and data time interval here correspond to the data synchronization script.

[0159] The computer device runs the data synchronization scripts in the 20 threads respectively, and saves the script identifiers of the data synchronization scripts in the preset continuation document. The script identifiers in the preset continuation document are arranged in reverse order.

[0160] 240. The computer device checks the running status of the data synchronization script in the thread after a preset time period to determine a plurality of running synchronization scripts in a running state and at least one continuing thread, wherein the data synchronization script of the continuing thread has finished running.

[0161] The computer device starts the automatic continuation script every 3 seconds, checks the running status of the data synchronization script in the thread through the automatic continuation script, and determines the continuation thread in which the data synchronization script has finished running.

[0162] 250. The computer device determines a first script identifier according to the script identifiers corresponding to each of the multiple data synchronization scripts.

[0163] The first script identifier is obtained from the preset continuation document and is determined to be the first script identifier.

[0164] 260. The computer device determines at least one continuation script identifier according to the first script identifier and the number of continuation threads.

[0165] The logic for determining the continuation script identifier includes: the time interval increases from 00 to 23; when it reaches 23 hours, the index number is incremented, the time interval is reset to 00, and the cycle continues; when the index number increases to the maximum index number, when the hour increases to 23 hours, the day interval is incremented, the time interval is reset to 00, and the index number is the minimum index number, and the cycle continues until the maximum time interval (23 hours) of the maximum index number is reached, and the synchronization script of the end date is executed. This process can also be seen Figure 5 .

[0166] 270. The computer device determines at least one continuation synchronization script corresponding to the continuation script identifier from multiple synchronization scripts.

[0167] 280. The computer device synchronizes the data corresponding to the index of the continuation index identifier in the to-be-synchronized data set and whose continuation storage time is within the time interval to the target data set through the continuation synchronization script.

[0168] The continuation index identifier and the continuation saving time here both correspond to the continuation synchronization script.

[0169] The computer device can trigger the progress push script at a set time interval, and the progress push script can send the script identifier of the data synchronization script running in the thread to the target object, so that the target object can know the index and time interval of the currently synchronized data. For example, the computer device can send an email to the target mailbox through the progress push script, and the email content may include: script identifier 1, script identifier 2, script identifier 3, script identifier 4, script identifier 5 and script identifier 6.

[0170] See also Figure 6 The computer device can also periodically generate report information for the data synchronization process. The report information may include synchronization analysis information and data attribute information, specifically the number of data items in the database, the number of synchronization log items in the preset log file that records the synchronization log, and the index identifier and target time interval of the index to which the data currently being synchronized belongs.

[0171] For example, you can get the number of data entries in the database es_cnt: 1455, the number of data entries in the preset log file suc_cnt: 1300, calculate the synchronization success rate suc_per: a%, compare and get the synchronization result: False (indicating that the synchronization is not completed), get the index identifier of the index to which the data being synchronized belongs: adxx, and the index number 005, and the target time interval: 2-3 o'clock on date 1.

[0172] This application discloses Figure 7 In the processing flow, the total time interval of the index identification file can be read first. The index identification file can include multiple target index identifications. The target index identification can include an identification of the index to which the data to be synchronized belongs in the data set to be synchronized. Then, based on the multiple target index identifications and the total time interval, multiple synchronization scripts can be generated through the synchronization template. The synchronization script can be identified by the index identification, time interval and day interval, which indicates that the synchronization script can synchronize the data to be synchronized, which belongs to the index corresponding to the index identification and whose storage time matches the day interval and time interval. Then distribute each synchronization script to its corresponding execution directory so that the synchronization script can be run quickly later. First, multiple data synchronization scripts can be selected from multiple synchronization scripts. Through the data synchronization script, part of the data to be synchronized in the data set to be synchronized can be synchronized to the target data set. Multiple data synchronization scripts run in multiple threads respectively.

[0173] The embodiment of the present application can also push synchronization progress information to the target object through a progress push script. Specifically, the progress push script can be started periodically, and the computer device can determine whether the current time point is the time point for starting the progress push script. If so, the script identifiers of multiple data synchronization scripts currently running can be obtained through the progress push script, and an email including these multiple script identifiers can be sent to the target object.

[0174] The embodiment of the present application can also determine at least one continuation synchronization script from multiple synchronization scripts through automatic continuation scripts, and the continuation synchronization script can continue to synchronize part of the to-be-synchronized data in the to-be-synchronized data set to the target data set. Specifically, the Zidongge continuation script can be started regularly, and the computer device can determine whether the current time point is the time point for starting the progress push script. If so, the running status of the data synchronization script in each thread is determined, thereby determining a first number of continuation threads, and the data synchronization script in the continuation thread has finished running, and then judging the changed data in the script identifiers of the multiple data synchronization scripts. If the changed data is a day interval, the day-time relationship can be determined based on the day interval in the multiple script identifiers, and then the continuation script identifier is determined from the multiple synchronization scripts; if the changed data is a time interval, the time-time relationship can be determined based on the time interval in the multiple script identifiers, and then the continuation script identifier is determined from the multiple synchronization scripts; if the changed data is an index identifier, the identifier relationship can be determined based on the index identifier in the multiple script identifiers, and then the continuation script identifier is determined from the multiple synchronization scripts, and the data synchronization is continued through the continuation synchronization script.

[0175] Therefore, the embodiment of the present application automatically performs large-scale data synchronization and can also regularly push current data synchronization information to the target object, which can significantly improve data synchronization efficiency compared to the prior art.

[0176] In order to better implement the above method, the embodiment of the present application also provides a data synchronization device, such as Figure 8 As shown, the data synchronization device may include an operation module 310, a state module 320, a connection module 330 and a synchronization module 340, wherein:

[0177] The running module 310 is used to synchronize first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval of the target index in the data set to be synchronized, the data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script is respectively run in at least one thread;

[0178] A status module 320 is used to check the running status of the data synchronization script in the thread to determine at least one continuing thread, and the data synchronization script of the continuing thread has finished running;

[0179] A continuation module 330, configured to determine at least one continuation synchronization script from a plurality of synchronization scripts according to the script identifier of at least one data synchronization script and the number of continuation threads;

[0180] The synchronization module 340 is used to synchronize the second data in the data to be synchronized to the target data set through the continuation synchronization script; wherein the second data includes the data in the data set to be synchronized that belongs to the continuation time interval of the continuation index.

[0181] In some embodiments of the present application, the data synchronization device further includes a document continuation module, wherein:

[0182] A continuation document module, used to save the script identifier of at least one data synchronization script in a preset continuation document;

[0183] At this time, the continuation module includes a first identification submodule, a continuation identification submodule and a continuation script submodule, wherein:

[0184] A first identification submodule, used to determine a first script identification from at least one script identification of a preset continuation document;

[0185] A continuation identification submodule, used to determine at least one continuation script identification according to the first script identification and the number of continuation threads;

[0186] The continuation script submodule is used to determine at least one continuation synchronization script corresponding to the continuation script identifier from multiple synchronization scripts.

[0187] In some embodiments of the present application, the script identifier of the data synchronization script includes a target index identifier and a target time interval, the target time interval includes a target day interval and a target time interval, and the first identification submodule is specifically used to:

[0188] determining a day-time relationship between target day intervals in at least one script identifier, and determining at least one initial script identifier from a plurality of script identifiers based on the day-time relationship;

[0189] Determine an identification relationship between target index identifications in at least one initial script identification, and determine at least one intermediate script identification from the initial script identification based on the identification relationship;

[0190] A temporal relationship between target time intervals in at least one intermediate script identifier is determined, and a first script identifier is determined from the intermediate script identifiers based on the temporal relationship.

[0191] In some embodiments of the present application, the connection identification submodule includes a connection time interval unit, a connection identification unit and a return unit, wherein:

[0192] A connection time interval unit, configured to determine the connection time interval based on a preset time interval granularity and a target time interval in the first script identifier;

[0193] A continuation identification unit, used to determine a continuation script identifier according to the continuation time interval and the target index identifier in the first script identifier;

[0194] The return unit is used to determine that the continuation script identifier is a new first script identifier, and return to execute the step of determining the continuation time interval based on the preset time interval granularity and the target time interval in the first script identifier, until the number of continuation script identifiers matches the number of continuation threads.

[0195] In some embodiments of the present application, the continuation identification unit includes a matching subunit and a non-matching subunit, wherein:

[0196] A matching subunit, configured to determine a continuation script identifier based on the continuation time interval, the target day interval of the first script identifier, and the target index identifier if the continuation time interval matches the preset time interval;

[0197] The mismatch subunit is used to determine the continuation index identifier based on the target index identifier in the first script identifier if the continuation time interval does not match the preset time interval, and to determine the continuation script identifier based on the continuation index identifier and the first script identifier.

[0198] In some embodiments of the present application, the data to be synchronized corresponds to an index identification file, and the index identification file includes at least one target index identification; the mismatch subunit is specifically used for:

[0199] If the continuation time interval does not match the preset time interval, determining the continuation index identifier based on the target index identifier in the first script identifier;

[0200] If the continuation index identifier belongs to the index identifier file, the continuation script identifier is determined based on the default time interval, the continuation index identifier and the target day interval in the first script identifier;

[0201] If the continuation index identifier does not belong to the index identifier file, the continuation date interval is determined based on the target date interval in the first script identifier, and the continuation script identifier is determined based on the continuation date interval, the default time interval and at least one target index identifier in the index identifier file.

[0202] In some embodiments of the present application, the state module is specifically used to:

[0203] After a preset time period, checking the running status of the data synchronization script in the thread to determine at least one running synchronization script and at least one continuing thread in the running state;

[0204] The data synchronization device also includes a determination module and a return module, wherein:

[0205] A determination module, used to determine the continued synchronization script and the running synchronization script as a new data synchronization script;

[0206] The return module is used to return to the step of checking the running status of the data synchronization script in the thread after a preset time period to determine at least one running synchronization script and at least one continuing thread in the running state until the data synchronization process for the data to be synchronized is completed.

[0207] In some embodiments of the present application, the data synchronization device further includes an information module and a generation module, wherein:

[0208] An information module, used to determine an index identification file and a total time interval corresponding to the data to be synchronized in the data set to be synchronized, wherein the index identification file includes at least one target index identification;

[0209] The generation module is used to obtain a synchronization template and generate multiple synchronization scripts based on at least one target index identifier, a total time interval and the synchronization template.

[0210] In some embodiments of the present application, the total time interval is determined by a first time point and a second time point, and the generation module includes an acquisition submodule, an index number submodule, a time point submodule, a return submodule and a storage submodule, wherein:

[0211] Get submodule, used to get synchronization template;

[0212] An index number submodule, used to determine the index number corresponding to each target index identifier in the index identifier file according to the position information of the target index identifier in the index identifier file;

[0213] A time point submodule, used to determine a target time interval and an intermediate time point based on a preset segmentation granularity and a first time point;

[0214] A return submodule, for determining the intermediate time point as a new first time point if the intermediate time point does not match the second time point, and returning to execute the step of determining the target time interval and the intermediate time point based on the preset segmentation granularity and the first time point;

[0215] The saving submodule is used to save the index number and the target time interval at a preset position of the synchronization template if the intermediate time point matches the second time point, so as to obtain multiple synchronization scripts.

[0216] In some embodiments of the present application, the index identification file includes multiple lines, each line stores a target index identification, and the index number submodule is specifically used for:

[0217] Determine the line number of the line where the target index mark is located in the index mark file;

[0218] The index number of the target index mark is obtained by calculation based on the row number and the preset value.

[0219] In some embodiments of the present application, the data synchronization device may also be used to:

[0220] When the synchronization script determines that the data has been synchronized to the target data set, a synchronization log for the data is added to the preset log file;

[0221] If the synchronization script determines that the data has not been synchronized to the target dataset, resynchronize the data to the target dataset.

[0222] In some embodiments of the present application, the data synchronization device further includes:

[0223] The quantity statistics module is used to count the number of data to be synchronized in the data set to be synchronized, and the number of synchronization logs in the preset log file;

[0224] An analysis information module is used to generate synchronization analysis information based on the number of items to be synchronized and the number of synchronization logs;

[0225] The attribute information module is used to determine the attribute information of the data being synchronized in the data set to be synchronized, and send synchronization analysis information and attribute information to the target object.

[0226] In some embodiments of the present application, the synchronization analysis information includes a synchronization success rate and a synchronization result, and the analysis information module is specifically used to:

[0227] The number of synchronization targets and the number of synchronization logs are combined to obtain the synchronization success rate.

[0228] The number of items to be synchronized and the number of synchronization logs are compared, and the synchronization result is determined based on the comparison result.

[0229] In some embodiments of the present application, the attribute information includes the data synchronization status of the data, the target index to which the data belongs, and the target time interval. The attribute information module is specifically used to:

[0230] Get the target index and target time interval of the data being synchronized in the data set to be synchronized;

[0231] Query the script ID of the synchronous script running in the thread;

[0232] If the script identifier matches the target index and the target time interval, the data synchronization state is determined to be the first state;

[0233] If the script identifier does not match at least one of the target index and the target time interval, determining that the data synchronization state is the second state;

[0234] Send synchronization analysis information, data synchronization status, target index to which the data belongs, and target time interval to the target object.

[0235] In an embodiment of the present application, the first data in the data to be synchronized can be synchronized to the target data set through at least one data synchronization script. The data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script runs in at least one thread respectively. For a data synchronization script, the corresponding first data is the data in the target time interval of the data set to be synchronized that belongs to the target index, and the target index and the target time interval serve as part of the script identifier of the data synchronization script.

[0236] The embodiment of the present application can check the running status of the data synchronization script in the thread. If the data synchronization script is in the running status, it can be determined that the data synchronization script is still performing data synchronization. If the data synchronization script is finished running, the data synchronized by the data synchronization script has been synchronized, and the thread where the data synchronization script is located is empty, and this thread is determined to be a continuation thread. Since the script identifier includes the target index and target time interval corresponding to the synchronization script, the target index and target time interval to which the data being synchronized or having been synchronized belong can be determined based on the script identifier of at least one data synchronization script. Then, combined with the number of continuation threads, at least one continuation synchronization script can be determined from multiple synchronization scripts, and the continuation synchronization script can be run in the continuation thread. Therefore, the present application can perform data synchronization efficiently and coherently, and effectively improve the efficiency of synchronizing the data to be synchronized to the target data set.

[0237] The present application also provides a computer device, such as Fig. 9 As shown, it shows a schematic diagram of the structure of a computer device involved in an embodiment of the present application. The computer device may be a terminal or a server, etc. Specifically:

[0238] The computer device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Fig. 9 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0239] The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing computer programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the computer device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and computer programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0240] The memory 402 can be used to store computer programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0241] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0242] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0243] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more computer programs into the memory 402 according to the following instructions, and the processor 401 will run the application program stored in the memory 402 to achieve various functions, as follows:

[0244] Synchronize the first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval belonging to the target index in the data set to be synchronized, and the data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script is respectively run in at least one thread; check the running status of the data synchronization script in the thread to determine at least one continuation thread, and the data synchronization script of the continuation thread has finished running; determine at least one continuation synchronization script from multiple synchronization scripts according to the script identifier corresponding to each of the at least one data synchronization script and the number of continuation threads; synchronize the second data in the data to be synchronized to the target data set through the continuation synchronization script; wherein the second data includes data in the continuation time interval belonging to the continuation index in the data set to be synchronized.

[0245] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0246] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0247] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any data synchronization method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0248] Synchronize the first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval belonging to the target index in the data set to be synchronized, and the data synchronization script is determined from multiple synchronization scripts, and at least one data synchronization script is respectively run in at least one thread; check the running status of the data synchronization script in the thread to determine at least one continuation thread, and the data synchronization script of the continuation thread has finished running; determine at least one continuation synchronization script from multiple synchronization scripts according to the script identifier corresponding to each of the at least one data synchronization script and the number of continuation threads; synchronize the second data in the data to be synchronized to the target data set through the continuation synchronization script; wherein the second data includes data in the continuation time interval belonging to the continuation index in the data set to be synchronized.

[0249] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0250] Since the computer program stored in the computer-readable storage medium can execute the steps in any data synchronization method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data synchronization method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0251] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional implementations of the above data synchronization method.

[0252] The above is a detailed introduction to a data synchronization method, device, computer equipment and computer-readable storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data synchronization method, characterized in that: include: Synchronize first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval of the data set to be synchronized and belongs to the target index, the data synchronization script is determined from multiple synchronization scripts, and the at least one data synchronization script is respectively run in at least one thread; Checking the running status of the data synchronization script in the thread to determine at least one continuing thread, wherein the data synchronization script of the continuing thread has finished running; Determine at least one continuing synchronization script from the plurality of synchronization scripts according to the script identifier corresponding to each of the at least one data synchronization script and the number of continuing threads; The second data in the data to be synchronized is synchronized to the target data set through the continuation synchronization script; wherein the second data includes data in the data set to be synchronized that belongs to the continuation time interval of the continuation index.

2. The method according to claim 1, characterized in that After synchronizing the first data in the data to be synchronized to the target data set by using at least one data synchronization script, the method further includes: Saving the script identifier of at least one of the data synchronization scripts in a preset continuation document respectively; The step of determining at least one subsequent synchronization script from the plurality of synchronization scripts according to the script identifier of at least one data synchronization script and the number of the subsequent threads includes: Determine a first script identifier from at least one of the script identifiers of the preset continuation document; Determining at least one continuation script identifier according to the first script identifier and the number of the continuation threads; At least one continuation synchronization script corresponding to the continuation script identifier is determined from the multiple synchronization scripts.

3. The method according to claim 2, characterized in that The script identifier of the data synchronization script includes a target index identifier and the target time interval, the target time interval includes a target day interval and a target time interval, and determining the first script identifier from at least one of the script identifiers of the preset continuation document includes: Determining a daily time relationship between target day intervals in at least one of the script identifiers, and determining at least one initial script identifier from at least one of the script identifiers based on the daily time relationship; Determine an identification relationship between target index identifications in at least one of the initial script identifications, and determine at least one intermediate script identification from the initial script identifications based on the identification relationship; A time relationship between target time intervals in at least one of the intermediate script identifiers is determined, and based on the time relationship, a first script identifier is determined from the intermediate script identifiers.

4. The method according to claim 3, characterized in that The step of determining at least one continuation script identifier according to the first script identifier and the number of continuation threads includes: Determine the connection time interval based on the preset time interval granularity and the target time interval in the first script identifier; Determine a continuation script identifier according to the continuation time interval and the target index identifier in the first script identifier; The continuation script identifier is determined to be a new first script identifier, and the step of determining the continuation time interval based on the preset time interval granularity and the target time interval in the first script identifier is returned to execute until the number of the continuation script identifiers matches the number of the continuation threads.

5. The method according to claim 4, characterized in that The determining the continuation script identifier according to the continuation time interval and the target index identifier in the first script identifier includes: If the continuation time interval matches the preset time interval, determining a continuation script identifier based on the continuation time interval, the target day interval of the first script identifier, and the target index identifier; If the continuation time interval does not match the preset time interval, a continuation index identifier is determined based on the target index identifier in the first script identifier, and a continuation script identifier is determined based on the continuation index identifier and the first script identifier.

6. The method according to claim 5, characterized in that The data to be synchronized corresponds to an index identification file, and the index identification file includes at least one target index identification; The determining the continuation script identifier based on the continuation index identifier and the first script identifier includes: If the continuation index identifier belongs to the index identifier file, determining a continuation script identifier based on a default time interval, the continuation index identifier, and a target day interval in the first script identifier; If the continuation index identifier does not belong to the index identifier file, the continuation day interval is determined based on the target day interval in the first script identifier, and the continuation script identifier is determined based on the continuation day interval, the default time interval and at least one target index identifier in the index identifier file.

7. The method according to claim 1, characterized in that The checking the running status of the data synchronization script in the thread to determine at least one continuing thread includes: After a preset time period, checking the running status of the data synchronization script in the thread to determine at least one running synchronization script and at least one continuing thread in a running state; After synchronizing the second data in the data to be synchronized to the target data set through the continuation synchronization script, the method further includes: Determine that the continuation synchronization script and the running synchronization script are new data synchronization scripts; Return to the step of checking the running status of the data synchronization script in the thread after a preset time period to determine at least one running synchronization script and at least one continuing thread in a running state until the data synchronization process for the data to be synchronized is completed.

8. The method according to claim 1, characterized in that: Before synchronizing the first data in the data to be synchronized to the target data set by using at least one data synchronization script, the method further includes: Determine an index identification file and a total time interval corresponding to the data to be synchronized in the data set to be synchronized, wherein the index identification file includes at least one target index identification; A synchronization template is obtained, and a plurality of synchronization scripts are generated based on the at least one target index identifier, the total time interval and the synchronization template.

9. The method according to claim 8, characterized in that The total time interval is determined by a first time point and a second time point, and the generating a plurality of synchronization scripts based on the at least one target index identifier, the total time interval and the synchronization template includes: Determine the index number corresponding to each target index identifier in the index identifier file according to the position information of the target index identifier in the index identifier file; Determine a target time interval and an intermediate time point based on a preset segmentation granularity and the first time point; If the intermediate time point does not match the second time point, determine the intermediate time point as a new first time point, and return to the step of determining the target time interval and the intermediate time point based on the preset segmentation granularity and the first time point; If the intermediate time point matches the second time point, the index number and the target time interval are saved in a preset position of the synchronization template to obtain multiple synchronization scripts.

10. The method according to claim 9, characterized in that The index identification file includes a plurality of lines, each line stores a target index identification, and determining the index number corresponding to each target index identification in the index identification file according to the position information of the target index identification in the index identification file includes: Determine the row number of the row where the target index identifier is located in the index identifier file; A calculation is performed based on the row number and a preset value to obtain an index number of the target index identifier.

11. The method according to claim 10, characterized in that The method further comprises: When it is determined by the synchronization script that the data has been synchronized to the target data set, adding a synchronization log for the data to a preset log file; When it is determined through the synchronization script that the data has not been synchronized to the target data set, the data is resynchronized to the target data set.

12. The method according to claim 11, characterized in that The method further comprises: Counting the number of to-be-synchronized data in the to-be-synchronized data set and the number of synchronization logs in the preset log file; Generate synchronization analysis information based on the number to be synchronized and the number of synchronization logs; Determine the attribute information of the data being synchronized in the data set to be synchronized, and send the synchronization analysis information and the attribute information to the target object.

13. The method according to claim 12, characterized in that The synchronization analysis information includes a synchronization success rate and a synchronization result. The synchronization analysis information is generated based on the number of items to be synchronized and the number of synchronization logs, including: The number to be synchronized and the number of synchronization logs are combined and calculated to obtain a synchronization success rate; The number to be synchronized and the number of synchronization logs are compared, and a synchronization result is determined according to the comparison result.

14. The method according to claim 13, characterized in that The attribute information includes the data synchronization state of the data, the target index and the target time interval to which the data belongs, and the determining of the attribute information of the data being synchronized in the data set to be synchronized includes: Obtain the target index and target time interval of the data being synchronized in the data set to be synchronized; Query the script identifier of the synchronization script running in the thread; If the script identifier matches both the target index and the target time interval, determining that the data synchronization state is the first state; If the script identifier does not match at least one of the target index and the target time interval, it is determined that the data synchronization state is the second state.

15. The method according to claim 1, characterized in that The method further comprises: Script identifiers of respective data synchronization scripts are obtained, and the plurality of script identifiers are sent to a target object.

16. A data synchronization device, characterized in that: include: An operation module is used to synchronize first data in the data to be synchronized to the target data set through at least one data synchronization script; wherein the first data includes data in the target time interval of the data set to be synchronized and belongs to the target index, the data synchronization script is determined from multiple synchronization scripts, and the at least one data synchronization script is respectively run in at least one thread; A status module, used for checking the running status of the data synchronization script in the thread to determine at least one continuing thread, wherein the data synchronization script of the continuing thread has finished running; A continuation module, configured to determine at least one continuation synchronization script from the plurality of synchronization scripts according to the script identifier of at least one data synchronization script and the number of the continuation threads; A synchronization module is used to synchronize the second data in the data to be synchronized to the target data set through the continuation synchronization script; wherein the second data includes the data in the data set to be synchronized that belongs to the continuation time interval of the continuation index.

17. A computer device, characterized in that: It comprises a memory and a processor; the memory stores an application, and the processor is used to run the application in the memory to execute the steps in the data synchronization method described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the data synchronization method according to any one of claims 1 to 15.

Images