A method and device for intelligent data synchronization

Abstract

The present invention discloses a method and device for intelligently synchronizing data, the method comprising: detecting a read instruction, obtaining target data from a database according to the read instruction, and reading a synchronization rule corresponding to the target data from a designer; performing a lock-taking operation on the target data to obtain a lock-taking result; judging whether the lock-taking result is used to indicate that the target data has completed the lock-taking operation; when the judgment result is yes, performing an extraction operation on the target data in combination with the corresponding synchronization rule, obtaining the target data and storing it in a data center. It can be seen that the present invention can complete the data synchronization operation in an intelligent manner, synchronize the data to the data center in combination with the synchronization rule, and extract the data to be queried from the data center, which is conducive to reducing the situation where the database performance is affected by the use of existing data query methods for data query, thereby improving the accuracy and efficiency of data query.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for intelligently synchronizing data. Background Art

[0002] In the information age, users are accustomed to querying data through databases. The content of the queried data is gradually diversified and the data in the database is constantly expanded, making it more accurate for users to query data. However, the existing data query methods have the problem of easily having a significant performance impact on the original database, such as reducing the efficiency of data query, thereby slowing down the processing efficiency of other normal businesses and poor user query experience. It can be seen that the existing technology has defects that need to be solved urgently. Summary of the invention

[0003] The technical problem to be solved by the content of the present invention is to provide a method and device for intelligently synchronizing data, which can complete the data synchronization operation in an intelligent manner, synchronize the data to the data center in combination with the synchronization rules, and extract the data to be queried from the data center, which is conducive to reducing the situation where the database performance is affected by the use of existing data query methods for data query, thereby improving the accuracy and efficiency of data query.

[0004] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a method for intelligently synchronizing data, the method comprising:

[0005] Detecting a read instruction triggered for target data, acquiring the target data from a database according to the read instruction, and reading synchronization rules corresponding to the target data from a designer; the synchronization rules corresponding to the target data include a storage location change rule for the target data, an extraction method rule for the target data, and a time length rule for completing an extraction operation for the target data, and are used to perform an extraction operation on the target data;

[0006] Performing a lock-taking operation on the target data to obtain a lock-taking result of the target data;

[0007] Determining whether the unlock result is used to indicate that the unlock operation has been completed on the target data;

[0008] When the unlock result indicates that the unlock operation has been completed for the target data, an extraction operation of the target data is performed in combination with a synchronization rule corresponding to the target data to obtain the target data, and the target data is stored in a data center.

[0009] As an optional implementation, in the first aspect of the present invention, the triggering mode of the read instruction triggered for the target data includes a timing triggering mode and / or a manual triggering mode and / or a dependent triggering mode;

[0010] And, the determining whether the unlock result is used to indicate that the unlock operation has been completed on the target data includes:

[0011] When the target data is in a synchronization state, determining the unlock result is used to indicate that the unlock operation has not been completed for the target data;

[0012] When the concurrent number of the execution queue of the target number extraction operation is greater than or equal to the set concurrent number threshold, the lock acquisition result is determined to indicate that the lock acquisition operation has not been completed for the target data.

[0013] As an optional implementation, in the first aspect of the present invention, after performing the target data extraction operation in combination with the synchronization rule corresponding to the target data to obtain the target data, the method further includes:

[0014] Recording information about operations related to extracting the target data; the information about operations related to extracting the target data includes details of situations that occur during the process of extracting the target data, synchronization rules corresponding to the target data and the extraction of the target data, and the time when the extraction of the target data is completed;

[0015] Execute an unlocking operation of the target data;

[0016] When the unlocking operation is completed, determining whether the target data has any associated dependent tasks;

[0017] If it is determined that the target data has the associated dependent task, a dependency trigger instruction is generated, and an operation of extracting the dependent data of the target data is executed according to the dependency trigger instruction.

[0018] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0019] Determine whether all data tables in the data set have been configured with a corresponding single historical synchronization rule; all data tables in the data set include all data tables in the target data and a dependent table corresponding to any data table of all the data tables in the target data;

[0020] If it is determined that all data tables in the data set are not configured with a corresponding single historical synchronization rule, a configuration operation of the corresponding target synchronization rule is performed on all data tables in the first data table; the first data table is a data table in the data set that is not configured with a corresponding single historical synchronization rule;

[0021] Determine whether the single historical synchronization rule corresponding to all data tables in the second data table is consistent with the target synchronization rule corresponding to all data tables in the second data table; the second data table is a data table in which the corresponding single historical synchronization rule has been configured in all the data tables in the data set;

[0022] If it is determined that the single historical synchronization rule corresponding to all the data tables in the second data table is inconsistent with the target synchronization rule corresponding to all the data tables in the second data table, a configuration operation of the target synchronization rule corresponding to all the data tables in the second data table is performed on all the data tables in the second data table;

[0023] The target synchronization rule is saved to the designer.

[0024] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0025] If it is determined that all data tables in the data set have been configured with a corresponding single historical synchronization rule, then determining whether the single historical synchronization rule corresponding to all data tables in the data set is consistent with the target synchronization rule corresponding to all data tables in the data set;

[0026] If it is determined that the single historical synchronization rule corresponding to all the data tables in the data set is not consistent with the target synchronization rule corresponding to all the data tables in the data set, a configuration operation of the target synchronization rule corresponding to all the data tables in the third data table is performed on all the data tables in the third data table; the third data table is a data table in which a single historical synchronization rule corresponding to the third data table exists among all the data tables in the data set, but the corresponding single historical synchronization rule is inconsistent with the target synchronization rule corresponding to the third data table;

[0027] The target synchronization rule is saved to the designer.

[0028] As an optional implementation, in the first aspect of the present invention, after determining that the unlock result indicates that the unlock operation has not been completed on the target data, the method further includes:

[0029] Putting the extraction operation of the target data into a waiting queue, and performing corresponding operations on the target data;

[0030] The performing of corresponding operations on the target data includes:

[0031] Generate a loop query instruction, and re-execute the lock-taking operation on the target data according to the loop query instruction to obtain the lock-taking result of the target data;

[0032] Alternatively, a trigger instruction for terminating the execution of the operation related to extracting the target data is detected, and information about the execution of the operation related to extracting the target data is recorded; the information about the execution of the operation related to extracting the target data includes details of the situation occurring during the execution of the operation of extracting the target data, the synchronization rules corresponding to the target data and the extraction completion time of the target data;

[0033] Alternatively, when it is detected that the time length of waiting for the operation of extracting the target data is greater than or equal to a set threshold, the relevant operation of ending the extraction of the target data is executed.

[0034] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0035] detecting a processing instruction of the target data;

[0036] Execute a corresponding processing operation according to the processing instruction of the target data; the corresponding processing operation is to execute a preview wide table operation or to execute a configuration synchronization rule operation;

[0037] If the corresponding processing operation is to perform the preview wide table operation, it is determined whether a related table exists in the designer; the related table is a wide table obtained according to the setting requirements;

[0038] If it is determined that the related table already exists in the database, displaying the related table;

[0039] If it is determined that the related table does not exist in the database, it is prompted that the related table does not exist in the database.

[0040] A second aspect of the present invention discloses an intelligent data synchronization device, the device comprising:

[0041] A detection module, used for detecting a read instruction triggered for target data;

[0042] an acquisition module, used to acquire the target data from a database according to the read instruction and to acquire synchronization rules corresponding to the target data from a designer; the synchronization rules corresponding to the target data include a storage location change rule of the target data, an extraction method rule of the target data, and a time length rule for completing the target data extraction operation, and is used to perform an extraction operation on the target data;

[0043] An unlocking module is used to execute an unlocking operation on the target data to obtain an unlocking result of the target data;

[0044] A judging module, used for judging whether the unlocking result is used to indicate that the unlocking operation has been completed for the target data;

[0045] The data extraction module is used to perform the extraction operation of the target data in combination with the synchronization rule corresponding to the target data when the unlock result indicates that the unlock operation has been completed for the target data, obtain the target data, and store the target data in the data center.

[0046] As an optional implementation, in the second aspect of the present invention, the triggering mode of the read instruction triggered for the target data includes a timing triggering mode and / or a manual triggering mode and / or a dependent triggering mode;

[0047] And, the specific method of determining whether the unlock result is used to indicate that the unlock operation has been completed on the target data includes:

[0048] When the target data is in a synchronization state, determining the unlock result is used to indicate that the unlock operation has not been completed for the target data;

[0049] When the concurrent number of the execution queue of the target number extraction operation is greater than or equal to the set concurrent number threshold, the lock acquisition result is determined to indicate that the lock acquisition operation has not been completed for the target data.

[0050] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0051] A recording module is used for, after the data extraction module performs an extraction operation of the target data in combination with the synchronization rule corresponding to the target data, recording information of the relevant operations of extracting the target data after obtaining the target data; the information of the relevant operations of extracting the target data includes details of the situation occurring during the extraction operation of the target data, the target data and the synchronization rule corresponding to the extraction of the target data, and the time when the extraction of the target data is completed;

[0052] A data unlocking module, used to perform an unlocking operation on the target data;

[0053] The judging module is further configured to judge whether the target data has any associated dependent tasks when the unlocking operation is completed;

[0054] The data extraction module is further configured to generate a dependency trigger instruction when the determination module determines that the target data has the associated dependency task, and execute an operation of extracting the dependency data of the target data according to the dependency trigger instruction.

[0055] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0056] The judgment module is further used to judge whether all data tables in the data set have been configured with a corresponding single historical synchronization rule; all data tables in the data set include all data tables in the target data and a dependent table corresponding to any data table of all the data tables in the target data;

[0057] A configuration module, configured to execute a configuration operation of a corresponding target synchronization rule on all data tables in a first data table when the judgment module determines that all data tables in the data set are not configured with a corresponding single historical synchronization rule; the first data table is a data table in the data set for which no corresponding single historical synchronization rule is configured;

[0058] The judgment module is further used to judge whether the single historical synchronization rule corresponding to all data tables in the second data table is consistent with the target synchronization rule corresponding to all data tables in the second data table; the second data table is a data table in which the corresponding single historical synchronization rule has been configured in all the data tables in the data set;

[0059] The configuration module is further configured to perform a configuration operation of the target synchronization rule corresponding to all the data tables in the second data table on all the data tables in the second data table when the judgment module judges that the single historical synchronization rule corresponding to all the data tables in the second data table is inconsistent with the target synchronization rule corresponding to all the data tables in the second data table;

[0060] A saving module is used to save the target synchronization rule to the designer.

[0061] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0062] The judgment module, when it is judged that all data tables in the data set have been configured with a corresponding single historical synchronization rule, is further used to judge whether the single historical synchronization rule corresponding to all data tables in the data set is consistent with the target synchronization rule corresponding to all data tables in the data set;

[0063] The configuration module is further used for performing a configuration operation of the target synchronization rule corresponding to all data tables in the third data table on all data tables in the third data table when the judgment module judges that the single historical synchronization rule corresponding to all data tables in the data set is not consistent with the target synchronization rule corresponding to all data tables in the data set; the third data table is a data table in which a single historical synchronization rule corresponding to the third data table exists among all data tables in the data set, but the corresponding single historical synchronization rule is inconsistent with the target synchronization rule corresponding to the third data table;

[0064] The saving module is also used to save the target synchronization rules to the designer.

[0065] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0066] A waiting processing module, configured to put the extraction operation of the target data into a waiting queue and perform a corresponding operation on the target data after the judging module judges that the unlocking result indicates that the unlocking operation has not been completed on the target data;

[0067] The specific manner in which the waiting processing module performs corresponding operations on the target data includes:

[0068] Generate a loop query instruction, and re-execute the lock-taking operation on the target data according to the loop query instruction to obtain the lock-taking result of the target data;

[0069] Alternatively, a trigger instruction for terminating the execution of the operation related to extracting the target data is detected, and information about the execution of the operation related to extracting the target data is recorded; the information about the execution of the operation related to extracting the target data includes details of the situation occurring during the execution of the operation of extracting the target data, the synchronization rules corresponding to the target data and the extraction completion time of the target data;

[0070] Alternatively, when it is detected that the time length of waiting for the operation of extracting the target data is greater than or equal to a set threshold, the relevant operation of ending the extraction of the target data is executed.

[0071] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0072] The detection module is further used to detect the processing instructions of the target data;

[0073] An execution module, used to execute a corresponding processing operation according to the processing instruction of the target data; the corresponding processing operation is to execute a preview wide table operation or to execute a configuration synchronization rule operation;

[0074] The judging module is further configured to judge whether a relevant table exists in the designer when the executing module performs the corresponding processing operation of performing the preview wide table operation; the relevant table is a wide table obtained according to the setting requirements;

[0075] A wide table processing module, configured to display the related table when the judging module judges that the related table already exists in the database;

[0076] The wide table processing module is further configured to prompt that the relevant table does not exist in the database when the judging module judges that the relevant table does not exist in the database.

[0077] The third aspect of the present invention discloses another intelligent data synchronization device, the device comprising:

[0078] A memory storing executable program code;

[0079] a processor coupled to the memory;

[0080] The processor calls the executable program code stored in the memory to execute the intelligent data synchronization method disclosed in the first aspect of the present invention.

[0081] The fourth aspect of the present invention discloses a computer storable medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the intelligent data synchronization method disclosed in the first aspect of the present invention.

[0082] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0083] In an embodiment of the present invention, a read instruction triggered for target data is detected, and the target data is read from the database according to the read instruction, and the synchronization rule corresponding to the target data is read from the designer; the synchronization rule corresponding to the target data is used to limit the extraction operation when executing the target data extraction operation, and the limitation includes limiting the location change of storing the target data, limiting the method of data extraction, and limiting the time to complete the data extraction operation; performing a lock operation on the target data to obtain the lock result of the target data; judging the lock result; when the lock result indicates that the lock operation has been completed for the target data, the extraction operation of the target data is performed in combination with the synchronization rule corresponding to the target data, the target data is obtained and stored in the data center. It can be seen that the implementation of the present invention can complete the data synchronization operation in an intelligent manner, synchronize the data to the data center in combination with the synchronization rule, and extract the data to be queried from the data center, which is conducive to reducing the situation that the database performance is affected by the use of existing data query methods for data query, thereby improving the accuracy and efficiency of data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0085] Figure 1 It is a flowchart of an intelligent data synchronization method disclosed in an embodiment of the present invention;

[0086] Figure 2 It is a flowchart of another intelligent data synchronization method disclosed in an embodiment of the present invention;

[0087] Figure 3 It is a structural diagram of an intelligent data synchronization device disclosed in an embodiment of the present invention;

[0088] Figure 4 It is a structural schematic diagram of another intelligent data synchronization device disclosed in an embodiment of the present invention;

[0089] Figure 5 It is a structural schematic diagram of another intelligent data synchronization device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0091] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0092] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0093] The present invention discloses a method and device for intelligent data synchronization, which can adopt a timing trigger mode, a manual trigger mode or a dependent trigger mode and combine with synchronization rules to perform data extraction operations, synchronize data to a data center through data extraction operations, and when a user performs data query, the data synchronization system directly performs a query operation for the required data from the data center, and when the query data has dependent data, the query data and the dependent data of the query data are jointly sent back to the user, in addition, it also provides a synchronization rule configuration function and a wide table preview function, and sets a logical and reasonable implementation operation method, which can improve the accuracy of data query and the completion rate of executing data extraction operations, thereby facilitating the improvement of the intelligence of executing data synchronization operations. The following are detailed descriptions.

[0094] Embodiment 1

[0095] See also Figure 1 , Figure 1 1 is a flow chart of an intelligent data synchronization method disclosed in an embodiment of the present invention. Figure 1 The described method can be applied to a data processing terminal or a big data analysis system, and the embodiments of the present invention do not limit this. Figure 1 As shown, the intelligent data synchronization method may include the following operations:

[0096] 101. Detect a read instruction triggered for target data, obtain the target data from a database according to the read instruction, and read a synchronization rule corresponding to the target data from a designer.

[0097] In the embodiment of the present invention, the triggered read instruction is specifically used to trigger the reading of the data extraction task sent by the data synchronization system, thereby completing the acquisition of the target data and the corresponding synchronization rules. Optionally, the synchronization rules used when the target data is subjected to the data extraction operation are determined according to the specific extraction operation requirements (such as the specified target data storage before change and storage after change, storage format of the target data, etc.) and / or the target data's own limiting conditions (such as the target data's capacity, target data's type, and target data's dependent tasks, etc.).

[0098] 102. Perform a lock operation on the target data to obtain a lock result of the target data.

[0099] In the embodiment of the present invention, the lock acquisition operation is specifically to use the self-developed ZK public lock to perform the lock acquisition operation on the target data extraction task, and the self-developed ZK public lock is used to manage the data extraction tasks in a first-come-first-served order to ensure that each data extraction task is executed. Further, the target data can be various types of data, which is not limited in the embodiment of the present invention.

[0100] 103. Determine whether the lock acquisition result is used to indicate that the lock acquisition operation has been completed for the target data.

[0101] In the embodiment of the present invention, determining the lock acquisition result specifically refers to determining whether the data synchronization system successfully creates the node. Further, when it is determined that the data synchronization system successfully creates the node, it means that Zookeeper has completed the lock acquisition operation; or, when it is determined that the data synchronization system has not successfully created the node, it means that Zookeeper has not completed the lock acquisition operation.

[0102] 104. When the lock acquisition result indicates that the lock acquisition operation has been completed for the target data, an extraction operation of the target data is performed in combination with a synchronization rule corresponding to the target data to obtain the target data, and the target data is stored in a data center.

[0103] In the embodiment of the present invention, specifically, when Zookeeper completes the lock-taking operation, the task of extracting target data is included in the execution queue using the task management software. Optionally, the task management software (such as RabbitMQ) is a software with a message queue framework and capable of implementing current limiting, delayed waiting, and distributed task management functions. The embodiment of the present invention does not limit the specific task management software used for the relevant operations of extracting target data.

[0104] Optionally, the target data extraction operation is specifically to use an open source ETL tool to perform target data backup, extraction, and rollback operations. Specifically, the open source ETL tool can complete data extraction and job scheduling frameworks for various data types, such as Kettle, Talend, etc. The embodiment of the present invention does not limit the specific open source ETL tool selected.

[0105] Optionally, the target data can be stored in the data center by manually triggering the data storage operation, or by the system setting to automatically trigger the data storage operation after detecting the completion of the data extraction operation, or by the system setting a specified conditional instruction to trigger the data storage operation, which is not limited in the embodiment of the present invention. Further, the operator can set a specified time length to limit the interval between the system detecting the completion of the data extraction operation and the automatic triggering of the data storage operation, and the specific time length value of the interval occurrence time length is not limited in the embodiment of the present invention.

[0106] Furthermore, the data center is used to store target data obtained through data extraction operations. Optionally, the implementation tool of the data center may be Mysql, Hive, or other systems capable of performing data management and analysis, which is not limited in the embodiments of the present invention.

[0107] It can be seen that the method of intelligently synchronizing data described in the embodiment of the present invention can complete the data synchronization operation in an intelligent manner, synchronize the data to the data center in combination with the synchronization rules, and extract the data to be queried from the data center, which is conducive to reducing the situation where the database performance is affected by the use of existing data query methods for data query, thereby improving the accuracy and efficiency of data query.

[0108] In an optional embodiment, the triggering mode of the read instruction triggered by the target data includes a timing triggering mode and / or a manual triggering mode and / or a dependent triggering mode; further, in step 103, determining whether the unlocking result is used to indicate that the unlocking operation has been completed for the target data may include:

[0109] When the target data is in the synchronization state, determining the lock acquisition result is used to indicate that the lock acquisition operation of the target data has not been completed;

[0110] When the concurrent number of the execution queue of the target number extraction operation is greater than or equal to the set concurrent number threshold, the lock acquisition result is determined to indicate that the lock acquisition operation of the target data has not been completed.

[0111] In the embodiment of the present invention, a timed trigger method is used to trigger the reading and extraction of data. Specifically, the task of reading and extracting data is triggered according to a set time interval and / or a set number of timed triggers, and the timed trigger setting is performed using a mainstream timed task scheduling framework (such as Quartz). Optionally, the set time interval can be a regular or irregular setting, which is not limited in the embodiment of the present invention; the specific value of the set number of timed triggers is not limited in the embodiment of the present invention.

[0112] The data extraction is triggered by a manual trigger. Specifically, the data synchronization system detects the trigger instruction sent by the user terminal and triggers the execution of the data extraction operation. Optionally, the user terminal may include but is not limited to at least one of a mobile terminal, an electronic device application software (such as a WeChat applet), a PC website, a designated client device, and a manual counter, which is not limited in the embodiment of the present invention.

[0113] The dependent trigger method is used to trigger the reading and extraction of data. Specifically, it is determined whether the target data has a dependent task. If it is determined that the target data has a dependent task, the dependent task is read and extracted. Optionally, the method of determining the target data dependent task can be to manually set the dependent data of the target data, or the system can automatically bind the target data and its dependent data according to the dependency setting conditions, which is not limited in the embodiment of the present invention.

[0114] Specifically, the concurrent number of the task of extracting target data is limited by the message queue framework of the task management software. The concurrent number is specifically the maximum number of connections of RabbitMQ. Optionally, the maximum number of connections (i.e., the concurrent number threshold) is set according to demand, and can be a threshold set by artificial calculation or a threshold set by RabbitMQ by default. The embodiment of the present invention does not limit this.

[0115] It can be seen that this optional embodiment can trigger the execution of data extraction operations in a variety of triggering methods and ensure that all extraction tasks are executed, reducing some unnecessary cumbersome operations and improving the intelligence of executing data extraction operations and the rationality of processing tasks.

[0116] In another optional embodiment, in the above steps, after determining that the unlock result indicates that the unlock operation has not been completed on the target data, the method may further include the following steps:

[0117] Put the extraction operation of target data into the waiting queue, and perform corresponding operations on the target data;

[0118] The corresponding operations are performed on the target data, including:

[0119] Generate a loop query instruction, and re-execute the lock operation on the target data according to the loop query instruction to obtain the lock result of the target data;

[0120] Alternatively, a trigger instruction for terminating the execution of operations related to extracting target data is detected, and information about the execution of operations related to extracting target data is recorded; the information about the execution of operations related to extracting target data includes details of situations that occur during the execution of the operations for extracting target data, synchronization rules corresponding to the target data and the target data for extracting, and the time when the extraction of the target data is completed;

[0121] Alternatively, when it is detected that the time length of waiting for the operation of extracting the target data is greater than or equal to a set threshold, the relevant operation of ending the extraction of the target data is executed.

[0122] In the embodiment of the present invention, specifically, the task management software is used to put the target data extraction operation task that has not completed the lock acquisition operation into the waiting queue. Optionally, the task management software (such as RabbitMQ) is a software with a message queue framework and capable of implementing current limiting, delayed waiting and distributed task management functions. The specific task management software used for the relevant operations of extracting target data is not limited in the embodiment of the present invention.

[0123] Optionally, when the data synchronization system detects a loop query instruction, it performs a check on the completion of the lock operation of the target data at every set interval, and performs corresponding operations according to the completion of the lock operation of the target data. Specifically, when the lock operation of the target data is completed through the loop query, the loop query ends, and the target data extraction task enters the execution queue.

[0124] Optionally, when the target data is found through a cyclic query to be an uncompleted lock operation, the target data extraction task returns to the waiting queue, and further, the execution operation selection of the target data extraction task is detected. Optionally, the execution operation selection situation can be to continue to query the completion status of the lock extraction operation of the target data extraction task, or to detect a termination instruction to terminate the lock extraction operation, or to detect a wait timeout instruction to end the lock extraction operation, which is not limited by the embodiment of the present invention. Specifically, the completion status of the lock extraction operation of the target data extraction task is cyclically queried according to the set interval duration. Optionally, the interval duration can be a fixed value, or the interval duration can be extended or shortened according to the number of detections; the change in the interval duration can be regular or irregular, which is not limited by the embodiment of the present invention.

[0125] Further, when it is detected that the waiting time for the target data extraction task to be executed is greater than or equal to the timeout threshold, the target data extraction operation is terminated. Optionally, the specific value of the timeout threshold can be set to any value manually, or can be selected within the threshold range originally specified by the system, which is not limited in the embodiment of the present invention.

[0126] It can be seen that this optional embodiment can provide multiple ways of processing data extraction tasks waiting to be executed, and can customize the limiting conditions for executing the operation, which is conducive to improving the diversification and intelligence of executing data extraction operations.

[0127] In yet another optional embodiment, after performing the target data extraction operation in step 104 in combination with the synchronization rule corresponding to the target data, the method may further include the following operations:

[0128] Recording information related to the operation of extracting target data; the information related to the operation of extracting target data includes details of the situation that occurs during the operation of extracting target data, synchronization rules corresponding to the target data and the extracted target data, and the time when the extraction of target data is completed;

[0129] Execute the unlocking operation of the target data;

[0130] When the unlock operation is completed, determine whether the target data has any associated dependent tasks;

[0131] If it is determined that the target data has associated dependent tasks, a dependent trigger instruction is generated, and an operation of extracting dependent data of the target data is executed according to the dependent trigger instruction.

[0132] In the embodiment of the present invention, specifically, the unlocking operation of the target data is executed, and the unlocking operation of the target data is detected to be completed, and the operation of extracting the target data is completed. It should be noted that after the target data completes the unlocking operation, if the target data needs to be extracted again, the Zookeeper lock operation needs to be performed on the target data again; if the target data is not unlocked after the previous data extraction operation is completed, the next extraction operation cannot extract the target data again.

[0133] Optionally, the details of the situation that occurred during the extraction of the target data are recorded, and the details of the situation include but are not limited to the fact that the extraction operation has been completed, an abnormality occurred during the extraction operation but has been resolved, and an abnormality occurred during the extraction operation and cannot be resolved, etc., which are not limited in the embodiments of the present invention. Further, when an abnormality occurs during the extraction operation, the data synchronization system detects the type of abnormal problem and automatically performs corresponding abnormal repair on the abnormal problem; if the automatic abnormal repair method cannot solve the abnormal problem, a prompt message is sent to prompt the administrator to perform manual operation to repair the abnormal problem.

[0134] It can be seen that this optional embodiment can provide a variety of optional methods for handling operation exceptions and record and extract operation information, which is conducive to facilitating subsequent backtracking operations and tracking operations.

[0135] Embodiment 2

[0136] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of another intelligent data synchronization method disclosed in an embodiment of the present invention. Figure 2 The described method can be applied to a data processing terminal or a big data analysis system, and the embodiments of the present invention do not limit this. Figure 2 As shown, the intelligent data synchronization method may include the following operations:

[0137] 201. Detect a read instruction triggered for target data, obtain the target data from a database according to the read instruction, and read a synchronization rule corresponding to the target data from a designer.

[0138] 202. Perform a lock operation on the target data to obtain a lock result of the target data.

[0139] 203. Determine whether the lock acquisition result is used to indicate that the lock acquisition operation has been completed for the target data.

[0140] 204. When the lock acquisition result indicates that the lock acquisition operation has been completed for the target data, an extraction operation of the target data is performed in combination with a synchronization rule corresponding to the target data to obtain the target data, and the target data is stored in the data center.

[0141] In the embodiment of the present invention, for other descriptions of step 201 to step 204, please refer to the detailed description of step 101 to step 104 in the first embodiment, and the embodiment of the present invention will not be repeated here.

[0142] 205. Determine whether all data tables in the data set have been configured with a corresponding single historical synchronization rule.

[0143] In an embodiment of the present invention, optionally, before performing configuration judgment on all data tables in a data set, the data synchronization system detects a data table filtering instruction, and the filtering instruction is used to determine all data tables and their dependent tables in the data set that need to be extracted from the database according to actual needs. Alternatively, all data tables and their dependent tables in the data set that need to be extracted are determined through a new data set operation and an add operation. Furthermore, the add operation can be to create a new data set and add a data set, or to create a new data set and add a data source and add a table, which is not limited in the embodiment of the present invention.

[0144] It should be noted that the operation of creating a new data set is an optional operation. When a new data set has been previously created in the data synchronization system and the data set is suitable for the data table type of this data extraction operation, the previously established data set can be directly called and modified to obtain the data table required for this data extraction operation; further, the establishment of the dependent table is similar to the establishment of the data table.

[0145] Further, after completing the screening operation or the data table establishment operation, all data tables in the data set and their dependent tables are displayed. Optionally, the configuration order determination method for configuring a single synchronization rule operation for all data tables in the data set and their dependent tables may include manually determining the configuration order, determining the configuration order according to the data table type and / or the first letter of the type name, determining the configuration order in a forward or reverse direction according to the establishment time, and randomly determining the configuration order, which is not limited in the embodiment of the present invention.

[0146] 206. If it is determined that all data tables in the data set are not configured with the corresponding single historical synchronization rule, a configuration operation of the corresponding target synchronization rule is performed on all data tables in the first data table.

[0147] In the embodiment of the present invention, optionally, when it is determined that not all data tables have been configured with a corresponding single historical synchronization rule, a prompt message is displayed indicating that the first data table has not been configured with a corresponding target synchronization rule, and the operator can choose to perform an operation of configuring the corresponding synchronization rule for the first data table, or can choose to directly terminate the configuration operation. Further, when the operator chooses to perform an operation of configuring the corresponding synchronization rule for the first data table, the corresponding configuration operation is performed.

[0148] 207. Determine whether the single historical synchronization rule corresponding to all the data tables in the second data table is consistent with the target synchronization rule corresponding to all the data tables in the second data table.

[0149] In an embodiment of the present invention, a single historical synchronization rule and a target synchronization rule corresponding to all data tables in the second data table are displayed. Optionally, whether the single historical synchronization rule is consistent with the target synchronization rule can be determined manually by comparison or by a data synchronization system combined with relevant programs, which is not limited in the embodiment of the present invention.

[0150] 208. If it is determined that the single historical synchronization rule corresponding to all data tables in the second data table is inconsistent with the target synchronization rule corresponding to all data tables in the second data table, a configuration operation of the target synchronization rule corresponding to all data tables in the second data table is performed on all data tables in the second data table.

[0151] Optionally, a method for determining the configuration order for executing corresponding target synchronization rule configuration operations on all data tables in the second data table may include manually determining the configuration order, determining the configuration order according to the data table type and / or the first letter of the type name, determining the configuration order in forward or reverse order according to the establishment time, and randomly determining the configuration order, which is not limited in the embodiments of the present invention.

[0152] 209. Save the target synchronization rules to the designer.

[0153] In an embodiment of the present invention, all data table configuration synchronization rule operations in a data set are logged, and the synchronization rules that complete the configuration operations are saved in the designer. The log operations and the designer operations are saved to facilitate subsequent backtracking operations or tracking operations.

[0154] Optionally, the logging operation includes, but is not limited to, recording the executor of the synchronization rule configuration operation, recording the operating behavior of the executor during the synchronization rule configuration operation, recording the program running status and the system and machine status during the synchronization rule configuration operation, and recording the information of events occurring during the operation (such as the time, location, level and content of the event), which is not limited in the embodiments of the present invention.

[0155] It can be seen that the embodiments of the present invention can perform corresponding judgment operations according to different conditions. Further, corresponding operations can be performed according to different judgment results obtained by the judgment operation, thereby realizing the configuration operation of the synchronization rules, and recording the operation information for subsequent tracing or tracking, which is beneficial to improving the reliability and rationality of the data extraction operation, thereby improving the efficiency of completing the data extraction operation.

[0156] In an optional embodiment, after executing step 205, the intelligent data synchronization method may further include the following operations:

[0157] If it is determined that all data tables in the data set have been configured with a corresponding single historical synchronization rule, then it is determined whether the single historical synchronization rules corresponding to all data tables in the data set are consistent with the target synchronization rules corresponding to all data tables in the data set;

[0158] If it is determined that the single historical synchronization rule corresponding to all data tables in the data set is not consistent with the target synchronization rule corresponding to all data tables in the data set, a configuration operation of the target synchronization rule corresponding to all data tables in the third data table is performed on all data tables in the third data table; the third data table is a data table in which a single historical synchronization rule corresponding to the third data table exists among all data tables in the data set, but the corresponding single historical synchronization rule is inconsistent with the target synchronization rule corresponding to the third data table;

[0159] Save the target synchronization rule to the designer.

[0160] Further optionally, when it is determined that all data tables in the data set have been configured with a corresponding single historical synchronization rule, all data tables in the data set, their corresponding single historical synchronization rules, and corresponding target synchronization rules are displayed to determine whether the corresponding single historical synchronization rule is consistent with the corresponding target synchronization rule. Optionally, the judgment operation can be a manual comparison and judgment, or a data synchronization system combined with a related program to perform a comparison and judgment, which is not limited in the embodiments of the present invention.

[0161] Optionally, a method for determining the configuration order for executing corresponding target synchronization rule configuration operations on all data tables in the third data table may include manually determining the configuration order, determining the configuration order according to the data table type and / or the first letter of the type name, determining the configuration order in forward or reverse order according to the establishment time, and randomly determining the configuration order, which is not limited in the embodiments of the present invention.

[0162] Optionally, log the synchronization rule configuration operations for all data tables in the dataset, and save the synchronization rules that have completed the configuration operations in the designer. The log operations and the designer operations are used to facilitate subsequent backtracking or tracking operations.

[0163] Further optionally, the logging operation includes but is not limited to recording the executor of the synchronization rule configuration operation, recording the operating behavior of the executor during the synchronization rule configuration operation, recording the program running status and the system and machine status during the synchronization rule configuration operation, and recording the information of events occurring during the operation (such as the time, location, level and content of the event), which is not limited in the embodiments of the present invention.

[0164] It can be seen that this optional embodiment can also perform corresponding judgment operations according to different conditions. Further, corresponding operations can be performed according to different judgment results obtained by the judgment operation, thereby realizing the configuration operation of the synchronization rules, and recording the operation information for subsequent tracing or tracking, which is beneficial to improving the feasibility and rationality of the data extraction operation, thereby improving the efficiency of completing the data extraction operation.

[0165] In another optional embodiment, the intelligent data synchronization method may further include the following operations:

[0166] Detecting processing instructions for target data;

[0167] Execute corresponding processing operations according to the processing instructions of the target data; the corresponding processing operations are executing a preview wide table operation or executing a configuration synchronization rule operation;

[0168] If the corresponding processing operation is to preview a wide table, it is determined whether a related table exists in the designer; the related table is a wide table obtained according to the setting requirements;

[0169] If it is determined that the relevant table already exists in the database, the relevant table is displayed;

[0170] If it is determined that the relevant table does not exist in the database, a prompt is given that the relevant table does not exist in the database.

[0171] Further optionally, the required wide table is obtained according to the setting requirements, and the setting requirements include the original rules for setting the wide table and / or the flexible rules for setting the wide table according to actual human needs. Optionally, the original rules for setting the wide table may include but are not limited to the number of data tables included in the wide table, the number of fields included in the wide table, and each data table in the wide table must have a unique identifier, etc., which are not limited in the embodiments of the present invention.

[0172] Optionally, the flexible rules for wide table settings may be to stipulate that all data tables in the wide table must be associated together according to indicators, dimensions, and attributes related to the business subject, or to stipulate that the wide table settings must comply with different normalization requirements, or to set up rules for wide tables based on wide table requirements specified by the operator, which is not limited in the embodiments of the present invention.

[0173] It can be seen that this optional embodiment can also select the function of executing the configuration synchronization rule operation or the preview wide table operation, and reasonably limit the operation steps for realizing the function, which is conducive to improving functionality, rationality and intelligence.

[0174] Embodiment 3

[0175] See also Figure 3 , Figure 3 is a schematic diagram of the structure of another intelligent data synchronization device disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a data processing terminal or a big data analysis system, which is not limited in the embodiments of the present invention. Figure 3 As shown, the intelligent data synchronization device may include:

[0176] The detection module 301 is used to detect a read instruction triggered for target data.

[0177] The acquisition module 302 acquires the target data from the database according to the read instruction and acquires the synchronization rules corresponding to the target data from the designer; the synchronization rules corresponding to the target data include the storage location change rules of the target data, the extraction method rules of the target data and the time required to complete the target data extraction operation, and is used to perform the extraction operation on the target data.

[0178] The lock-taking module 303 is used to perform a lock-taking operation on the target data to obtain a lock-taking result of the target data.

[0179] The judgment module 304 is used to judge whether the unlock result is used to indicate that the unlock operation has been completed on the target data.

[0180] The data extraction module 305 is used to perform an extraction operation on the target data in combination with the synchronization rule corresponding to the target data when the unlock result indicates that the unlock operation has been completed on the target data, obtain the target data, and store the target data in the data center.

[0181] It can be seen that the implementation Figure 3 The described device can complete data synchronization operations in an intelligent manner, synchronize data to a data center in combination with synchronization rules, and extract the data to be queried from the data center, which is conducive to reducing the situation where database performance is affected by using existing data query methods for data query, thereby improving the accuracy and efficiency of data query.

[0182] In this optional embodiment, as an optional implementation, the triggering mode of the read instruction triggered by the target data includes a timing triggering mode and / or a manual triggering mode and / or a dependent triggering mode;

[0183] Furthermore, the specific manner in which the determination module 304 determines whether the unlock result is used to indicate that the unlock operation has been completed on the target data includes:

[0184] When the target data is in the synchronization state, determining the lock acquisition result is used to indicate that the lock acquisition operation of the target data has not been completed;

[0185] When the concurrent number of the execution queue of the target number extraction operation is greater than or equal to the set concurrent number threshold, the lock acquisition result is determined to indicate that the lock acquisition operation of the target data has not been completed.

[0186] It can be seen that this optional embodiment can also trigger the execution of data extraction operations in a variety of triggering methods, and ensure that all extraction tasks are executed, reducing some unnecessary cumbersome operations, and improving the intelligence of executing data extraction operations and the rationality of processing tasks.

[0187] In yet another optional embodiment, Figure 4 As shown, the device may also include:

[0188] The recording module 306 is used to execute the target data extraction operation in the data extraction module 305 in combination with the synchronization rules corresponding to the target data, and after obtaining the target data, record the information of the relevant operations of extracting the target data; the information of the relevant operations of extracting the target data includes the details of the situations occurring during the execution of the target data extraction operation, the target data and the synchronization rules corresponding to the target data extraction, and the time when the extraction of the target data is completed.

[0189] The data unlocking module 307 is used to execute the unlocking operation of the target data.

[0190] The determination module 304 is further configured to determine whether the target data has any associated dependent tasks when the unlocking operation is completed.

[0191] The data extraction module 305 is also used to generate a dependency trigger instruction when the determination module 304 determines that the target data has an associated dependency task, and execute an operation of extracting the dependency data of the target data according to the dependency trigger instruction.

[0192] In this optional embodiment, when the data extraction module 305 completes the above-mentioned unlocking result indicating that the target data has completed the unlocking operation, the target data extraction operation is performed in combination with the synchronization rule corresponding to the target data to obtain the target data. After the target data is stored in the data center, the data unlocking module 307 can be triggered to perform the above-mentioned unlocking operation of the target data.

[0193] It can be seen that implementation Figure 4The described device can also provide a variety of optional methods for handling operational anomalies, and record and extract operational information, which is conducive to facilitating subsequent backtracking and tracking operations.

[0194] In yet another optional embodiment, Figure 4 As shown, the device may also include:

[0195] The judgment module 304 is also used to judge whether all data tables in the data set have been configured with corresponding single historical synchronization rules; all data tables in the data set include all data tables in the target data and dependent tables corresponding to any data table of all data tables in the target data.

[0196] Configuration module 308 is used to perform the configuration operation of the corresponding target synchronization rule on all data tables in the first data table when the judgment module 304 determines that not all data tables in the data set have been configured with the corresponding single historical synchronization rule; the first data table is a data table in the data set that is not configured with the corresponding single historical synchronization rule.

[0197] The judgment module 304 is also used to judge whether the single historical synchronization rule corresponding to all data tables in the second data table is consistent with the target synchronization rule corresponding to all data tables in the second data table; the second data table is a data table in which the corresponding single historical synchronization rule has been configured in all data tables in the data set.

[0198] The configuration module 308 is also used to perform a configuration operation on all the data tables in the second data table corresponding to the target synchronization rule for all the data tables in the second data table when the judgment module 304 determines that the single historical synchronization rule corresponding to all the data tables in the second data table is inconsistent with the target synchronization rule corresponding to all the data tables in the second data table.

[0199] The saving module 309 is used to save the target synchronization rules to the designer.

[0200] It can be seen that implementation Figure 4 The described device can also perform corresponding judgment operations according to different conditions. Further, corresponding operations can be performed according to different judgment results obtained from the judgment operation, thereby realizing the configuration operation of the synchronization rules, and recording the operation information for subsequent tracing or tracking, which is conducive to improving the feasibility and rationality of the data extraction operation, thereby improving the efficiency of completing the data extraction operation.

[0201] In yet another optional embodiment, Figure 4 As shown, the device may also include:

[0202] The judgment module 304 is also used to judge whether the single historical synchronization rule corresponding to all data tables in the data set is consistent with the target synchronization rule corresponding to all data tables in the data set when it is judged that all data tables in the data set have been configured with a corresponding single historical synchronization rule.

[0203] The configuration module 308 is also used to perform a configuration operation of the target synchronization rule corresponding to all data tables in the third data table on all data tables in the third data table when the judgment module 304 determines that the single historical synchronization rule corresponding to all data tables in the data set is not consistent with the target synchronization rule corresponding to all data tables in the data set; the third data table is a data table in which there is a single historical synchronization rule corresponding to the third data table among all data tables in the data set, but the corresponding single historical synchronization rule is inconsistent with the target synchronization rule corresponding to the third data table.

[0204] The saving module 309 is also used to save the target synchronization rules to the designer.

[0205] It can be seen that implementation Figure 4 The described device can also perform corresponding judgment operations according to different conditions. Further, corresponding operations can be performed according to different judgment results obtained from the judgment operation, thereby realizing the configuration operation of the synchronization rules, and recording the operation information for subsequent tracing or tracking, which is conducive to improving the feasibility and rationality of the data extraction operation, thereby improving the efficiency of completing the data extraction operation.

[0206] In yet another optional embodiment, Figure 4 As shown, the device may also include:

[0207] The waiting processing module 310 is used to put the extraction operation of the target data into the waiting queue and perform corresponding operations on the target data after the judging module 304 judges that the lock acquisition result indicates that the lock acquisition operation has not been completed on the target data.

[0208] The specific manner in which the waiting processing module 310 performs corresponding operations on the target data includes:

[0209] Generate a loop query instruction, and re-execute the lock operation on the target data according to the loop query instruction to obtain the lock result of the target data;

[0210] Alternatively, a trigger instruction for terminating the execution of operations related to extracting target data is detected, and information about the execution of operations related to extracting target data is recorded; the information about the execution of operations related to extracting target data includes details of situations that occur during the execution of the operations for extracting target data, synchronization rules corresponding to the target data and the target data for extracting, and the time when the extraction of the target data is completed;

[0211] Alternatively, when it is detected that the time length of waiting for the operation of extracting the target data is greater than or equal to a set threshold, the relevant operation of ending the extraction of the target data is executed.

[0212] It can be seen that implementation Figure 4 The described device can also provide multiple ways of processing data extraction tasks waiting to be executed, and can customize the limiting conditions for executing the operation, which is conducive to improving the diversification and intelligence of executing data extraction operations.

[0213] In yet another optional embodiment, Figure 4 As shown, the device may also include:

[0214] The detection module 301 is also used to detect the processing instructions of the target data.

[0215] The execution module 311 is used to execute a corresponding processing operation according to the processing instruction of the target data; the corresponding processing operation is to execute a preview wide table operation or to execute a configuration synchronization rule operation.

[0216] The judging module 304 is also used to judge whether a related table exists in the designer when the executing module 311 performs a corresponding processing operation to perform a preview wide table operation; the related table is a wide table obtained according to setting requirements.

[0217] The wide table processing module 312 is used to display the related table when the determination module 304 determines that the related table already exists in the database.

[0218] The wide table processing module 312 is also used to prompt that the relevant table does not exist in the database when the judging module 304 judges that the relevant table does not exist in the database.

[0219] It can be seen that implementation Figure 4 The described device can also choose to execute the function of configuring synchronization rule operations or executing the preview wide table operation, and reasonably limit the operation steps for realizing the functions, which is conducive to improving functionality, rationality and intelligence.

[0220] Embodiment 4

[0221] See also Figure 5 , Figure 5 is a schematic diagram of the structure of another intelligent data synchronization device disclosed in an embodiment of the present invention. Figure 5 The described device can be applied to a data processing terminal or a big data analysis system, which is not limited in the embodiments of the present invention. Figure 5 As shown, the device may include:

[0222] A memory 401 storing executable program codes;

[0223] a processor 402 coupled to the memory 401;

[0224] Furthermore, it may also include an input interface 403 and an output interface 404 coupled to the processor 402;

[0225] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the method for intelligently synchronizing data described in the first or second embodiment.

[0226] Embodiment 5

[0227] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the method for intelligently synchronizing data described in the first or second embodiment.

[0228] Embodiment 6

[0229] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the method for intelligently synchronizing data described in Embodiment 1 or Embodiment 2.

[0230] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0231] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0232] Finally, it should be noted that the method and device for intelligent data synchronization disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments.

Claims

1. A method for intelligent data synchronization, characterized in that: The method comprises: Detecting a read instruction triggered for target data, acquiring the target data from a database according to the read instruction, and reading synchronization rules corresponding to the target data from a designer; the synchronization rules corresponding to the target data include a storage location change rule for the target data, an extraction method rule for the target data, and a time length rule for completing an extraction operation for the target data, and are used to perform an extraction operation on the target data; Performing a lock-taking operation on the target data to obtain a lock-taking result of the target data; Determining whether the unlock result is used to indicate that the unlock operation has been completed on the target data; When the unlock result indicates that the unlock operation has been completed for the target data, an extraction operation of the target data is performed in combination with a synchronization rule corresponding to the target data to obtain the target data, and the target data is stored in a data center; And, the method further comprises: Determine whether all data tables in the data set have been configured with a corresponding single historical synchronization rule; all data tables in the data set include all data tables in the target data and a dependent table corresponding to any data table of all the data tables in the target data; If it is determined that all data tables in the data set are not configured with a corresponding single historical synchronization rule, a configuration operation of the corresponding target synchronization rule is performed on all data tables in the first data table; the first data table is a data table in the data set that is not configured with a corresponding single historical synchronization rule; Determine whether the single historical synchronization rule corresponding to all data tables in the second data table is consistent with the target synchronization rule corresponding to all data tables in the second data table; the second data table is a data table in which the corresponding single historical synchronization rule has been configured in all the data tables in the data set; If it is determined that the single historical synchronization rule corresponding to all the data tables in the second data table is inconsistent with the target synchronization rule corresponding to all the data tables in the second data table, a configuration operation of the target synchronization rule corresponding to all the data tables in the second data table is performed on all the data tables in the second data table; Saving the target synchronization rule to the designer; And, the method further comprises: If it is determined that all data tables in the data set have been configured with a corresponding single historical synchronization rule, then determining whether the single historical synchronization rules corresponding to all data tables in the data set are consistent with the target synchronization rules corresponding to all data tables in the data set; If it is determined that the single historical synchronization rule corresponding to all the data tables in the data set is not consistent with the target synchronization rule corresponding to all the data tables in the data set, a configuration operation of the target synchronization rule corresponding to all the data tables in the third data table is performed on all the data tables in the third data table; the third data table is a data table in which a single historical synchronization rule corresponding to the third data table exists among all the data tables in the data set, but the corresponding single historical synchronization rule is inconsistent with the target synchronization rule corresponding to the third data table; The target synchronization rule is saved to the designer.

2. The method for intelligent data synchronization according to claim 1, characterized in that: The triggering mode of the read instruction triggered for the target data includes a timing triggering mode and / or a manual triggering mode and / or a dependent triggering mode; And, the determining whether the unlock result is used to indicate that the unlock operation has been completed on the target data includes: When the target data is in a synchronization state, determining the unlock result is used to indicate that the unlock operation has not been completed for the target data; When the concurrent number of the execution queue of the target data extraction operation is greater than or equal to the set concurrent number threshold, the lock acquisition result is determined to indicate that the lock acquisition operation has not been completed for the target data.

3. The method for intelligent data synchronization according to claim 1, characterized in that: After performing the target data extraction operation in combination with the synchronization rule corresponding to the target data to obtain the target data, the method further includes: Recording information about operations related to extracting the target data; the information about operations related to extracting the target data includes details of situations that occur during the process of extracting the target data, synchronization rules corresponding to the target data and the extraction of the target data, and the time when the extraction of the target data is completed; Execute an unlocking operation of the target data; When the unlocking operation is completed, determining whether the target data has any associated dependent tasks; If it is determined that the target data has the associated dependent task, a dependency trigger instruction is generated, and an operation of extracting the dependent data of the target data is executed according to the dependency trigger instruction.

4. The method for intelligent data synchronization according to claim 1, characterized in that: After determining that the unlock result indicates that the unlock operation has not been completed on the target data, the method further includes: Putting the extraction operation of the target data into a waiting queue, and performing corresponding operations on the target data; The performing of corresponding operations on the target data includes: Generate a loop query instruction, and re-execute the lock-taking operation on the target data according to the loop query instruction to obtain the lock-taking result of the target data; Alternatively, a trigger instruction for terminating the execution of the operation related to extracting the target data is detected, and information about the execution of the operation related to extracting the target data is recorded; the information about the execution of the operation related to extracting the target data includes details of the situation occurring during the execution of the operation of extracting the target data, the synchronization rules corresponding to the target data and the extraction completion time of the target data; Alternatively, when it is detected that the time length of waiting for the operation of extracting the target data is greater than or equal to a set threshold, the relevant operation of ending the extraction of the target data is executed.

5. The method for intelligent data synchronization according to claim 1, characterized in that: The method further comprises: detecting a processing instruction of the target data; Execute a corresponding processing operation according to the processing instruction of the target data; the corresponding processing operation is to execute a preview wide table operation or to execute a configuration synchronization rule operation; If the corresponding processing operation is to perform the preview wide table operation, it is determined whether a related table exists in the designer; the related table is a wide table obtained according to the setting requirements; If it is determined that the related table already exists in the database, displaying the related table; If it is determined that the related table does not exist in the database, it is prompted that the related table does not exist in the database.

6. An intelligent data synchronization device, characterized in that: The device comprises: A detection module, used for detecting a read instruction triggered for target data; an acquisition module, used to read the target data from a database according to the read instruction and to acquire synchronization rules corresponding to the target data from a designer; the synchronization rules corresponding to the target data include changes in the storage location of the target data, a method for extracting the target data, and a time required to complete an operation of extracting the target data, and are used to limit the operation of extracting the target data; An unlocking module is used to execute an unlocking operation on the target data to obtain an unlocking result of the target data; A judging module, used for judging whether the unlocking result is that the unlocking operation has been completed; A data extraction module, configured to, when the unlock result indicates that the unlock operation has been completed for the target data, perform an extraction operation on the target data in combination with a synchronization rule corresponding to the target data, obtain the target data, and store the target data in a data center; And, the device further comprises: The judgment module is further used to judge whether all data tables in the data set have been configured with a corresponding single historical synchronization rule; all data tables in the data set include all data tables in the target data and a dependent table corresponding to any data table of all the data tables in the target data; A configuration module, configured to execute a configuration operation of a corresponding target synchronization rule on all data tables in a first data table when the judgment module determines that all data tables in the data set are not configured with a corresponding single historical synchronization rule; the first data table is a data table in the data set for which no corresponding single historical synchronization rule is configured; The judgment module is further used to judge whether the single historical synchronization rule corresponding to all data tables in the second data table is consistent with the target synchronization rule corresponding to all data tables in the second data table; the second data table is a data table in which the corresponding single historical synchronization rule has been configured in all the data tables in the data set; The configuration module is further configured to perform a configuration operation of the target synchronization rule corresponding to all the data tables in the second data table on all the data tables in the second data table when the judgment module judges that the single historical synchronization rule corresponding to all the data tables in the second data table is inconsistent with the target synchronization rule corresponding to all the data tables in the second data table; A saving module, used for saving the target synchronization rule to the designer; And, the device further comprises: The judgment module, when it is judged that all data tables in the data set have been configured with a corresponding single historical synchronization rule, is further used to judge whether the single historical synchronization rule corresponding to all data tables in the data set is consistent with the target synchronization rule corresponding to all data tables in the data set; The configuration module is further used for performing a configuration operation of the target synchronization rule corresponding to all data tables in the third data table on all data tables in the third data table when the judgment module judges that the single historical synchronization rule corresponding to all data tables in the data set is not consistent with the target synchronization rule corresponding to all data tables in the data set; the third data table is a data table in which a single historical synchronization rule corresponding to the third data table exists among all data tables in the data set, but the corresponding single historical synchronization rule is inconsistent with the target synchronization rule corresponding to the third data table; The saving module is also used to save the target synchronization rules to the designer.

7. An intelligent data synchronization device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the intelligent data synchronization method as described in any one of claims 1-5.

8. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the method for intelligently synchronizing data as described in any one of claims 1-5.

Images