Data synchronization method and system, and computer readable storage medium

Abstract

The present disclosure relates to a data synchronization method and system, and a computer-readable storage medium. The method comprises: obtaining initial source data from a source database; generating fingerprint data of the initial source data to obtain target source data containing the fingerprint data; using the fingerprint data as the primary key of the initial source data; synchronizing the target source data to a target database, so that the target database stores the target source data after the primary key verification passes. In this embodiment, by generating fingerprint data with unique characteristics for the initial source data, the problem of repeated synchronization during data synchronization can be solved, thereby improving the accuracy and efficiency of data synchronization.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data synchronization method and system, and a computer-readable storage medium. Background Art

[0002] Existing heterogeneous data source offline synchronization tools are committed to achieving stable and efficient data synchronization between various heterogeneous data sources including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, etc. As an offline data synchronization framework, the heterogeneous data source offline synchronization tool itself is built with the Framework+plugin architecture, abstracting the reading and writing of data sources into Reader / Writer plug-ins and incorporating them into the entire synchronization framework.

[0003] Considering that the data to be synchronized comes from different data sources, and the data from different data sources may use different types of primary keys (such as string-type uuid, database auto-increment numeric primary keys, or primary keys with custom rules). In actual applications, some source data read by Reader plug-ins do not have primary keys. In this case, it is necessary to use the relational database to generate auto-increment primary keys, which will cause duplicate insertion of data during incremental data synchronization. In addition, since the Reader plug-in and the Writer plug-in may be in different nodes, and the existing offline synchronization tools for heterogeneous data sources do not have a security processing mechanism, it is impossible to detect data that has been tampered with during the data synchronization transmission process. Summary of the invention

[0004] The present disclosure provides a data synchronization method and system, and a computer-readable storage medium to address the deficiencies of related technologies.

[0005] According to a first aspect of an embodiment of the present disclosure, a data synchronization method is provided, which is applied to a data synchronization system, and includes:

[0006] Obtaining initial source data from a source database;

[0007] Generate fingerprint data of the initial source data to obtain target source data containing the fingerprint data; the fingerprint data is used as the primary key of the initial source data;

[0008] The target source data is synchronized to a target database, so that the target database stores the target source data after the primary key verification passes.

[0009] Optionally, generating fingerprint data of the initial source data includes:

[0010] Call the preset fingerprint generation model;

[0011] The initial source data is input into the fingerprint generation model as input data of the fingerprint generation model to obtain the fingerprint data.

[0012] Optionally, the preset fingerprint generation model is implemented using at least one of a message digest algorithm, a secure hash algorithm, a message authentication code algorithm, and a key-based message authentication code algorithm.

[0013] Optionally, synchronizing the target source data to a target database includes:

[0014] Generate verification fingerprint data based on initial source data in the target source data;

[0015] When the primary key in the target source data is identical to the verification fingerprint data, the target source data is sent to the target database.

[0016] Optionally, the method further comprises:

[0017] The target database obtains the primary key in the target source data and matches the primary key of the stored data;

[0018] When the primary key of the target source data exists in the target database, updating the target source data to the target database;

[0019] When the primary key of the target source data does not exist in the target database, the target source data is inserted into the target database.

[0020] Optionally, synchronizing the target source data to a target database includes:

[0021] Acquire first fingerprint data of data columns other than the newly added data columns and second fingerprint data of data columns including the newly added data columns;

[0022] Matching the first fingerprint data with a primary key of the target database;

[0023] When the primary key of the target source data exists in the target database, the target source data and the second fingerprint data are updated to the target database.

[0024] Optionally, the method further comprises:

[0025] Get historical task information;

[0026] Counting the column combinations in the target source data used in the historical task information;

[0027] Fingerprint data is generated according to the column data of the column combination, and the fingerprint data is synchronously stored in the target database as a secondary key of the target source data.

[0028] According to a second aspect of an embodiment of the present disclosure, there is provided a data synchronization system, including a source database, a target database and a data synchronization device;

[0029] The data synchronization device is used to obtain initial source data from the source database and generate fingerprint data of the initial source data to obtain target source data containing the fingerprint data; the fingerprint data is used as the primary key of the initial source data; and the target source data is synchronized to the target database so that the target database stores the target source data after the primary key verification is passed.

[0030] Optionally, the data synchronization device includes a Framework module, a fingerprint data generation module, a read plug-in and a write plug-in; the Framework module is connected to the read plug-in and the write plug-in respectively,

[0031] The reading plug-in is used to read the initial source data to be synchronized from the source database;

[0032] The fingerprint data generating module is used to generate fingerprint data of the initial source data and use the fingerprint data as the primary key of the initial source data;

[0033] The Framework module is used to forward the initial source data and the primary key as target source data to the write plug-in;

[0034] The write plug-in is used to write the target source data into the target database.

[0035] Optionally, the fingerprint data generation module is integrated into the reading plug-in and / or Framework module.

[0036] Optionally, the fingerprint data generation module is integrated into the write plug-in to generate verification fingerprint data based on the initial source data in the target source data; the write plug-in is also used to compare the primary key in the target source data with the verification fingerprint data, and when the primary key is the same as the verification fingerprint data, the target source data is sent to the target database.

[0037] Optionally, the plug-in written in the data synchronization device is also used to: obtain first fingerprint data of data columns other than the newly added data columns and second fingerprint data of data columns including the newly added data columns; match the first fingerprint data and the primary key of the target database; when the primary key of the target source data exists in the target database, update the target source data and the second fingerprint data to the target database.

[0038] Optionally, the data synchronization device is also used to obtain historical task information; count the column combinations in the target source data used in the historical task information; generate fingerprint data based on the column data of the column combinations, and synchronously store the fingerprint data as a secondary key of the target source data in the target database.

[0039] According to a third aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.

[0040] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0041] As can be seen from the above embodiments, the solution provided by the embodiments of the present disclosure can obtain initial source data from the source database; then, generate fingerprint data of the initial source data to obtain target source data containing the fingerprint data; the fingerprint data is used as the primary key of the initial source data; then, synchronize the target source data to the target database, so that the target database stores the target source data after the primary key verification passes. In this way, in this embodiment, by generating fingerprint data with unique characteristics for the initial source data, the problem of repeated synchronization during data synchronization can be solved, thereby improving the accuracy and efficiency of data synchronization.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0044] Figure 1 It is a block diagram of a data synchronization system according to an exemplary embodiment.

[0045] Figure 2 is a block diagram of another data synchronization system according to an exemplary embodiment.

[0046] Figure 3 The figure is a flow chart of a data synchronization method according to an exemplary embodiment.

[0047] Figure 4 The figure is a schematic diagram of an application scenario of a data synchronization system according to an exemplary embodiment.

[0048] Figures 5 to 8 The figure is a schematic diagram showing the effect of configuring a task according to an exemplary embodiment.

[0049] Fig. 9 The figure is a flow chart showing a data synchronization according to an exemplary embodiment. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described exemplarily below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the attached claims. It should be noted that the features in the following embodiments and implementations may be combined with each other without conflict.

[0051] Existing heterogeneous data source offline synchronization tools are dedicated to achieving stable and efficient data synchronization between various heterogeneous data sources including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, etc. As an offline data synchronization framework, the heterogeneous data source offline synchronization tool itself is built with the Framework+plugin architecture, abstracting the data source reading and writing into Reader / Writer plug-ins and incorporating them into the entire synchronization framework.

[0052] Considering that the data to be synchronized comes from different data sources, and the data from different data sources may use different types of primary keys (such as string-type uuid, database auto-increment numeric primary keys, or primary keys with custom rules). In actual applications, some source data read by Reader plug-ins do not have primary keys. In this case, it is necessary to use the relational database to generate auto-increment primary keys, which will cause duplicate insertion of data during incremental data synchronization. In addition, since the Reader plug-in and the Writer plug-in may be in different nodes, and the existing offline synchronization tools for heterogeneous data sources do not have a security processing mechanism, it is impossible to detect data that has been tampered with during the data synchronization transmission process.

[0053] To solve the above technical problems, the embodiments of the present disclosure provide a data synchronization method and system, and a computer-readable storage medium. Figure 1 is a block diagram of a data synchronization system according to an exemplary embodiment. Figure 1, the data synchronization system includes a source database, a target database and a data synchronization device. The data synchronization device is connected to the source database and the target database respectively, and the data synchronization device is used to obtain the initial source data from the source database and generate fingerprint data of the initial source data to obtain the target source data containing the fingerprint data; the fingerprint data is used as the primary key of the initial source data; and the target source data is synchronized to the target database so that the target database stores the target source data after the primary key verification is passed. In this embodiment, by generating fingerprint data with unique characteristics for the initial source data, the problem of repeated synchronization in the data synchronization process can be solved, and the accuracy and synchronization efficiency of data synchronization can be improved.

[0054] It should be noted that the source of the source data can be one source database or multiple source databases, that is, the number of source databases can be set according to the specific scenario and is not limited here. Similarly, the target source data can also be synchronized to different target databases. Therefore, the number of target databases can be one or more, which can be set according to the specific scenario and is not limited here. For the convenience of description, the present disclosure directly takes one source database and one target database as an example to describe the solutions of each embodiment.

[0055] In this embodiment, the data synchronization device can be implemented by using a heterogeneous data source offline synchronization tool, that is, the data synchronization device can be constructed by using a Framework+plugin architecture. Figure 2 is a block diagram of another data synchronization system according to an exemplary embodiment. Figure 2 , the data synchronization device in the data synchronization system may include a Framework module, a fingerprint data generation module, a reading plug-in and a writing plug-in. The Framework module is connected to the reading plug-in and the writing plug-in respectively. The reading plug-in is used to read the initial source data to be synchronized from the source database; the fingerprint data generation module is used to generate the fingerprint data of the initial source data and use the fingerprint data as the primary key of the initial source data; the Framework module is used to forward the initial source data and the primary key as the target source data to the writing plug-in; the writing plug-in is used to write the target source data to the target database. In this way, the data synchronization device can achieve the effect of synchronizing the data in the source database to the target database without duplication.

[0056] In this embodiment, the fingerprint data may be a message digest. In this case, the fingerprint data generation module may include a preset fingerprint generation model, which may include but is not limited to a message digest algorithm (Message Digest, MD), a secure hash algorithm (Secure Hash Algorithm, SHA) and a message authentication code algorithm (Message Authentication Code, MAD). The technicians may select a suitable algorithm according to the specific scenario. If the fingerprint data can be generated, the corresponding scheme falls within the protection scope of this disclosure.

[0057] In one example, the fingerprint generation model is a key-based message authentication code algorithm (Hash-basedMessage Authentication Code, HMAC). The HMAC algorithm takes a message M and a key K as input, and generates a fixed-length message digest as output. The message M is the initial source data, and the fixed-length message digest is the fingerprint data. In this way, the fingerprint data is a message digest that can prevent data from being tampered with during transmission, and can be used as a unique identifier to compare with the primary key in the target database to avoid duplicate synchronization of data. In addition, in this example, using fingerprint data to generate fingerprint data from the initial source data can unify the primary key type of existing data, avoid repeated insertion of data, and improve synchronization efficiency.

[0058] This embodiment only describes the function of the data synchronization device in transmitting the target source data. In actual applications, it serves as a data transmission channel for both the reading plug-in and the writing plug-in, and can also handle buffering, data flow control, concurrent processing, data conversion and other functions. The corresponding function can be selected according to the specific scenario. When the target source data can be transmitted normally, the corresponding scheme falls within the protection scope of this disclosure.

[0059] It should be noted that Figure 2 In the data synchronization system shown, the fingerprint data generation module is located between the reading plug-in and the Framework module. This is because the fingerprint data generation module can be integrated into the reading plug-in and can also be integrated into the Framework module, that is, the fingerprint data generation module is integrated into the reading plug-in and / or the Framework module. It can be selected according to the specific scenario, and the corresponding solution falls within the protection scope of this disclosure.

[0060] In one embodiment, considering that the reading plug-in is usually set to different source databases, the fingerprint data generation module can be integrated into the reading plug-in. Plug-ins for different data sources need to follow the framework's agreement on plug-ins, so that each plug-in completes the common steps of data operations, that is, the segmentation of concurrent tasks and the reading and sending of data. In this embodiment, the reading plug-in can call the startRead method of the reading plug-in after the task is started. The startRead method uses the recordSender interface of the framework as a parameter. This startRead method connects to the data source according to the task configuration and reads the data to be synchronized, and then uniformly encapsulates the data to be synchronized into a record object Record object of the framework standard, and sends the record object to the Framework framework for processing. In the process of the reading plug-in encapsulating the read data into the Record object, the fingerprint data generation module can generate fingerprint data for the data to be synchronized, and the generated fingerprint data is encapsulated in the Record object as the primary key column, that is, the reading plug-in can directly obtain the target source data. In other words, the reading plug-in can generate the fingerprint data of the initial source data after reading the initial source data. At this time, the reading plug-in can directly obtain the target source data and upload the target source data to the Framework module. In this way, the Framework module does not need to generate fingerprint data, which can reduce the amount of data processed by the Framework module. Furthermore, when the fingerprint data is generated by the reading plug-in, the fingerprint data can be verified in the Framework module or the writing plug-in to determine whether the original source data has been tampered with, which is beneficial to improving the security of the data synchronization process.

[0061] In another embodiment, the read plug-in can call the startRead method of the read plug-in after the task is started. The startRead method uses the recordSender interface of the framework as a parameter. This method connects to the data source and reads the data according to the configuration, and then encapsulates the data into a unified record object Record object of the framework. The fingerprint data generation module can be used as a recordSender interface method. This interface method is called during the processing of the Record object by the Framework module to generate a unified fingerprint for the transmitted data. At this time, the fingerprint data generation module can generate fingerprint data of the initial source data transmitted by different read plug-ins. In this way, the fingerprint data generation module only needs to be deployed in the Framework module, and does not need to be generated in each read plug-in, which can reduce the workload of the read plug-in. In addition, generating fingerprint data of the initial source data in the Framework module can reduce the possibility of tampering with data transmission in subsequent processes, which is conducive to improving the security of the data synchronization process.

[0062] In one embodiment, after the writing plug-in task is started, the startWrite method of the writing plug-in is called. The startWrite method uses the recordReceive interface of the framework as a parameter. This method receives the record object Record object from the Framework framework, connects to the data source according to the task configuration and writes the target source data. At this time, the fingerprint data generation module can obtain the initial source data in the target source data, such as the column data in the Record object, and generate the verification fingerprint data. It is understandable that the generation method of the verification fingerprint data in the writing plug-in is the same as that of the fingerprint data in the reading plug-in. Then, the writing plug-in can compare the primary key column and the verification fingerprint data in the Record object, and when the primary key column is the same as the verification fingerprint data, it is determined to send the target source data to the target database; when the primary key is different from the verification fingerprint data, it is determined that the target source data has been tampered with, and there is no need to synchronize the target source data. In this way, this embodiment can perform security verification on the target source data by integrating the fingerprint data generation module in the writing plug-in, thereby improving the security of the data synchronization process.

[0063] It should be noted that, in combination with the functions and deployment location of the above-mentioned fingerprint data generation module, when the fingerprint data generation module is integrated in the read plug-in, the fingerprint data generation module can be integrated in the Framework module and / or the write plug-in, and the latter's integrated fingerprint data generation module performs security verification on the target source data. Alternatively, the fingerprint data generation module is integrated in the read plug-in and / or the Framework module, and the fingerprint data generation module is integrated in the write plug-in, and the fingerprint data generation module in the write plug-in performs security verification on the target source data. When the fingerprint data generation module is deployed in the read plug-in, it can give developers greater flexibility, such as specifying which data is used as input for generating fingerprints, that is, only using the specified data column to generate the primary key, and only verifying the specified data column. In this way, the corresponding write plug-in also needs to be consistent with the fingerprint generation implementation of the read plug-in. When the fingerprint data generation module is deployed in the Framework module, the developer cannot decide the data range for generating fingerprints, and the fingerprint generation becomes a fixed and unified process, and all read plug-ins use unified rules to generate fingerprints. In other words, the fingerprint data generated by the fingerprint data generation module can be used as a primary key or as verification fingerprint data, and the location of deployment can be selected according to the specific scenario, and the corresponding scheme falls within the protection scope of the present disclosure.

[0064] In one scenario, the initial source data includes four columns of data, namely name, height, weight and age. During the reading process and the verification process, the fingerprint data generation module can generate fingerprint data and write it into the target source data or perform initial source data verification. In another scenario, if a certain business needs to add a column of "gender" data. In order to match the overlap, the fingerprint data generation module can first generate fingerprint data for the first four columns of data, and the fingerprint data is used as the first fingerprint data; then, the fingerprint data generation module can generate fingerprint data for the original source data and the newly added source data, that is, generate fingerprint data for the first five columns of data, and the fingerprint data is used as the second fingerprint data. Then, the writing plug-in can use the first fingerprint data for verification and matching during the writing process. When the same fingerprint data exists in the target database, the newly added data column is inserted into the row corresponding to the first fingerprint data, and the second fingerprint data is inserted into the corresponding row; or, the target source data can be directly used to replace the data corresponding to the first fingerprint data in the target database.

[0065] In another scenario, before writing to the target database, the data synchronization system can preprocess the target source data as follows: the data synchronization system can obtain historical task information and count the column combinations in the target source data used by the historical task information. Then, the data synchronization system can group the data columns of the target source data based on the above column combinations. For example, the four columns of name, height, weight and age can be divided into column combinations such as {name, height}, {name, weight}, {name, age}, {name, age, height}, {name, age, weight}, and {name, age, height, weight}. Afterwards, the data synchronization system can generate different fingerprint data based on each column combination, and these fingerprint data will be synchronously stored in the target database as the secondary key of the target source data, so as to facilitate the user's needs to use different data columns when creating new tasks and the needs to insert data into different data column combinations in the target database. In this way, in this scenario, one preprocessing can provide fingerprint data for subsequent multiple data synchronizations, reduce the data processing volume of the subsequent synchronization process, and help improve the efficiency of data synchronization. In addition, this scenario can be applied to the de-redundancy, simplification and security processing of large-scale data, provide data support for users to create tasks, and reduce the management difficulty of the data synchronization system.

[0066] In this embodiment, after the write plug-in writes the target source data to the target database, the target database can read the primary key from the target database and compare it with the primary key it has stored. When the primary key in the target source data already exists in the target database, it means that the target source data has been stored in the target database, and the target database can update or replace the target source data. When the primary key in the target source data does not exist in the target database, the target database can insert the target source data. In this way, this embodiment can achieve the effect of synchronizing the initial source data in the source database to the target database.

[0067] So far, the solution provided by the embodiment of the present disclosure can obtain the initial source data from the source database; then, generate the fingerprint data of the initial source data to obtain the target source data containing the fingerprint data; the fingerprint data is used as the primary key of the initial source data; then, synchronize the target source data to the target database, so that the target database stores the target source data after the primary key verification passes. In this way, in this embodiment, by generating fingerprint data with unique characteristics for the initial source data, the problem of repeated synchronization in the data synchronization process can be solved, and the accuracy and efficiency of data synchronization can be improved.

[0068] Combination Figure 1 and Figure 2 The data synchronization system shown in the figure, the embodiment of the present disclosure also provides a data synchronization method, Figure 3 is a flow chart of a data synchronization method according to an exemplary embodiment. Figure 3 , a data synchronization method, comprising steps 31 to 33.

[0069] In step 31, initial source data is obtained from a source database.

[0070] In this embodiment, the data synchronization device can obtain configuration information, determine the source database and the source data to be synchronized in the source database, and the target database according to the configuration information, and then the reading plug-in in the data synchronization device can obtain the initial source data from the source database.

[0071] In step 32, fingerprint data of the initial source data is generated to obtain target source data including the fingerprint data; the fingerprint data is used as the primary key of the initial source data.

[0072] In this embodiment, the fingerprint data generation module in the data synchronization device can generate fingerprint data of the initial source data. For example, the data synchronization device can call a preset fingerprint generation model, and then input the initial source data into the fingerprint generation model as input data of the fingerprint generation model to obtain fingerprint data. Afterwards, the data synchronization device can use the initial source data and the fingerprint data as target source data, that is, obtain target source data containing fingerprint data. In this example, the above fingerprint data is used as the primary key of the initial source data, and the primary key can be used for subsequent security verification and primary key comparison during the storage process.

[0073] It is understandable that the fingerprint data generation module in the data synchronization device can be integrated into the reading plug-in and the Framework module to generate fingerprint data used as the primary key, and can also be integrated into the Framework module and the writing plug-in to generate verification fingerprint data for verification. The specific content can be found in the solution of the data synchronization system, which will not be repeated here.

[0074] In step 33, the target source data is synchronized to the target database, so that the target database stores the target source data after the primary key verification passes.

[0075] In this embodiment, a fingerprint data generation module may be deployed in the data synchronization device, and the data synchronization device may generate verification fingerprint data based on the initial source data in the target source data. Then, when the primary key in the target source data is the same as the verification fingerprint data, the data synchronization device may send the target source data to the target database.

[0076] In this embodiment, the target database can obtain the primary key in the target source data and match the primary key of the stored data. When the primary key of the target source data exists in the target database, the target database can update the target source data to the target database; when the primary key of the target source data does not exist in the target database, the target database can insert the target source data into the target database. In this embodiment, through the update and insert operations, it can be ensured that the data synchronized to the target database is not repeated.

[0077] In this embodiment, synchronizing the target source data to the target database includes:

[0078] Acquire first fingerprint data of data columns other than the newly added data columns and second fingerprint data of data columns including the newly added data columns;

[0079] Matching the first fingerprint data with a primary key of the target database;

[0080] When the primary key of the target source data exists in the target database, the target source data and the second fingerprint data are updated to the target database.

[0081] In this embodiment, the method further includes:

[0082] Get historical task information;

[0083] Counting the column combinations in the target source data used in the historical task information;

[0084] Fingerprint data is generated according to the column data of the column combination, and the fingerprint data is synchronously stored in the target database as a secondary key of the target source data.

[0085] It should be noted that the method shown in this embodiment matches the content of the system embodiment, and reference may be made to the content of the above system embodiment, which will not be repeated here.

[0086] The working principle of the data synchronization system is described below in conjunction with an embodiment. Figure 4 The data synchronization system includes: WEB container, GateWay gateway, Service container, Exexutor container, external data source and execution engine.

[0087] The WEB container includes a data source management module, a task configuration module, a job management module, and a system management module. The data source management module is used to configure and manage the connection information of the data source, such as IP address, port, cluster configuration parameters, and authentication method, manage all data sources created by users, provide common search, editing, and deletion methods, and can perform connectivity tests and external permission settings on data sources. The task configuration module is used to manage user-configured tasks. Users can combine existing data sources to create exchange tasks. The created tasks will be mounted under the corresponding project. Scheduled execution tasks and historical data reruns can be configured. The job management module is used to list all executed jobs under user-related tasks, including job call time, completion time, execution parameters, execution nodes, and completion status. You can click on the detailed log to view specific execution details. The system management module is used to manage system users and permissions and manage service execution nodes.

[0088] The WEB container can provide a visual UI interface for interaction with users, which is convenient for users to configure and manage each link in the data synchronization process. For example, users can interact with the data source management module, so that the data source management module calls the SERVICE service API interface to add, delete, modify or query the source data source and the target data source. These data sources can be understood as candidate source data sources and target data sources, that is, the data source of the reading plug-in and the writing target of the writing plug-in. Users can interact with the task configuration module, so that the task configuration module can specify the source data source of the data to be synchronized and the target data source of the data to be written and configure the field correspondence between the data to be synchronized. These task configuration information can be saved to the database through the SERVICE service API interface for subsequent reading plug-ins and writing plug-ins. Users can interact with the job management module, which can add, delete, modify, query and execute the configured tasks. For example, when clicking to execute a task, the SERVICE service API interface will be called. The SERVICE service reads the task information to obtain the task-related configuration. The SERVICE service will call the Exexutor service to actually execute the task. The Exexutor service starts the reading plug-in and the writing plug-in according to the task information to complete the synchronization of the task data. Users can interact with the system management module, which can manage users, permissions, etc.

[0089] The GateWay gateway communicates with the Service container and the Exexutor container, including the user unified authentication component, the user single sign-on component, the container permission configuration component, the authentication credential generator, the credential refresh entry, and the service routing module. The GateWay gateway is used for authentication and routing between service containers, authentication of external requests, and interaction with external authentication servers, and maintains the routing link of the entire platform. As the only entry for SERVICE services and Exexutor services, the GateWay gateway uniformly receives requests from the WEB container and completes login verification and distribution of the requests.

[0090] The SERVICE container includes a data permission management module, an execution node monitoring management module, a load balancer, a job queue scheduler, an RPC call module, a task configuration module, a data source management module, and a task job information status management module. The SERVICE service provided by the SERVICE container can manage multiple groups of API interfaces related to user use, and is the entry point for users to create and configure data sources, configure and execute data exchange tasks, and other business operations. The SERVICE container receives requests from the front-end WEB container and processes the requests, including adding, deleting, modifying, and querying operations on the database, as well as scheduling tasks by calling the Exexutor container service.

[0091] The Exexutor container includes a task execution management container, a task executor, a resource allocation module, a task job subprocess, a job daemon thread, a job log, a job information callback interface, and job runtime information. The Executor service provided by the Exexutor container is a container that actually executes data exchange tasks. It is used to connect to the execution engine. It also maintains multiple listening threads for task jobs, monitors job logs, timeouts, and job resource allocation. The Exexutor container receives requests from the WEB container and processes the requests, including queries on task job status and task job logs. The Exexutor container also maintains a heartbeat connection with the SERVICE container. When the WEB container starts a task, the SERVICE service calls the Exexutor container service by passing the task information. The Exexutor container service starts the data synchronization execution engine (i.e., the Framework module) according to the task configuration information. The execution engine starts the read and write plug-ins to run the data synchronization job.

[0092] The execution engine includes a Hadoop / Hive environment and a heterogeneous data source offline synchronization tool (i.e., the data synchronization device in the above embodiment). The heterogeneous data source offline synchronization tool includes a data source read-write plug-in (i.e., the read plug-in and the write plug-in in the above embodiment), a data transmission channel (i.e., a Framework module), a checkpoint module, a data postprocessor, a channel speed change strategy group, etc. The execution engine is used to read the initial source data to be synchronized from an external data source and generate fingerprint data to obtain the target source data, and then write it to the external data source, i.e., the target database, through a write plug-in. The read plug-in and the write plug-in obtain the corresponding relationship between the source database and the target database and the source data and the target data according to the task configuration information for data synchronization. These configuration information are configured through the UI interface provided by the user through the WEB container and are saved in the database by calling the SERVICE container service; when the task is started, the SERVICE container service reads the task configuration information and passes it to the Exexutor container service, and the Exexutor container service finally passes the configuration information to the execution engine for the read-write plug-in to synchronize data according to the configuration information.

[0093] Figure 4 The data synchronization factor shown works as follows:

[0094] (1) When using the data synchronization system, the user can open the interactive interface and enter the data management center in the interactive interface, which includes a data management module. The user can trigger the data management module to configure the source data source (i.e., source database) and the target data source (i.e., target database). For example, the user clicks on "Create a new data source". After the user fills in the data source information and clicks "Save", a request for adding a new data source is generated. The request calls the API interface in the Service to store the data source information in the database. The interface returns a response message that the data source has been added successfully, and a prompt message indicating that the data source has been saved successfully is displayed on the UI interface.

[0095] (2) Trigger the task configuration module and enter the data integration management task configuration page. Create a new data synchronization task, fill in the task name, task project and other basic information, and click Next to configure the source data source. Select any data source, such as the rest data source, configure the rest API interface address, request method and request parameters, and test the availability of the data source connection. The effect is as follows: Figure 5 shown.

[0096] (3) Click Next to configure the target data source. Select the postgresql data source. The effect is as follows Figure 6 shown.

[0097] (4) Click Next to configure the correspondence between the source field and the target field. The fields in the source data source correspond to the fields and types in the target data source one by one. That is, specify which column in the source data corresponds to which column in the target data source. The field correspondence can only be one-to-one, not one-to-many or many-to-one, and there are no unmatched field columns. The effect is as follows: Figure 7 shown.

[0098] (5) Click Next, configure the data synchronization speed and click Save; call the service API interface to save the configured data synchronization task information to the database. The effect is as follows Figure 8 shown.

[0099] (6) Trigger the job management module and enter the task management interface. At this point, you can find the task you just configured and click Execute. At this point, you can call the service API to execute the task.

[0100] (7) The Service service calls the API interface of the Executor service to pass the task information to the Executor service for execution.

[0101] (8) The Executor service starts the offline synchronization tool for heterogeneous data sources (i.e. Figure 4 Data synchronization device) to perform data synchronization process, see Fig. 9,include:

[0102] (a) Parse the configuration files, including the task configuration file job.json, the engine configuration file core.json, and the read-write plug-in configuration file plugin.json.

[0103] (b) Merge the parsed configuration information into the task configuration information required by the task.

[0104] (c) Start the engine through the Engine.start() engine start method according to the task configuration information.

[0105] (d) The engine creates a task container JobContainer and starts it through the task container startup method JobContainer.start(). The task container startup method JobContainer.start() executes the task pre-processing method preHandler(), the task initialization method init(), the read-write plug-in ready method prepare(), the task splitting method split(), the task scheduling method schedule(), the task post-processing method post(), and the task post-processing method postHandle() in sequence.

[0106] (e) The task initialization method init() initializes the reading plug-in reader and the writing plug-in writer according to the task configuration information.

[0107] (f) The task ready method prepare() implements the class loading of the read-write plug-in by calling the read-write plug-in ready method prepare().

[0108] (g) The task splitting method split() adjusts the number of channels through the channel number adjustment method adjustChannelNumber(), and simultaneously performs the finest-grained splitting of the reading plug-in reader and the writing plug-in writer.

[0109] (h) Channel counting is mainly based on the rate limit of bytes and records (i.e. the source data to be synchronized or the initial source data). If the user does not set the number of channels, the first step of the task splitting method split() needs to calculate the size of the channel.

[0110] (i) The mergeReaderAndWriterTaskConfigs() method in the split() method is responsible for merging the read and write subtask configurations among the reader plug-in, the writer plug-in, and the transformer to generate the subtask configuration.

[0111] (j) The task scheduling method schedule() generates a subtask group taskGroup object according to the subtask task configuration generated by the task splitting method split(). It is configured according to the number of subtasks task and the number of subtasks task supported by a single subtask group taskGroup. The number of subtask groups taskGroup can be obtained by dividing the two.

[0112] (k) The task scheduling method schdule() is executed internally through the task scheduling method schedule() of the scheduler AbstractScheduler, and then continues to execute the startAllTaskGroup() method to create all subtask tasks related to the subtask group container TaskGroupContainer. The subtask group container runner TaskGroupContainerRunner is responsible for running the subtask group container TaskGroupContainer to execute the assigned subtask task.

[0113] (l) The sub-task group container execution service taskGroupContainerExecutorService starts a fixed thread pool to execute the sub-task group container runner TaskGroupContainerRunner object. The execution method run() of TaskGroupContainerRunner calls the sub-task group container start method taskGroupContainer.start() method, creates a sub-task executor TaskExecutor for each channel channel, and starts the task through taskExecutor.doStart().

[0114] The (m)recordSender data fingerprint interface method uses the HMAC algorithm to generate a message digest (i.e., fingerprint data) of a record, thereby generating fingerprint data for the source data to be synchronized and obtaining the target source data.

[0115] (n) The writing plug-in synchronizes the target source data into the target database. Before synchronization, the target source data may be subjected to security verification.

[0116] In this way, the present disclosure adds an interface method for generating data fingerprints to the offline synchronization tool for heterogeneous data sources. Before the reading plug-in reads the data record and completes the processing and sends it to the writing plug-in, the interface method for generating data fingerprints is called to generate a summary string for the data record (the message digest or fingerprint data in the above embodiment). This summary string is used as the identifier and check value of the data record, so that the data synchronized to the target library has a unified data identifier, thereby avoiding the problem of inconsistent data identifier IDs of different data sources; and the effect of data tamper-proofing and avoiding the problem of repeated data insertion are achieved.

[0117] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program, and the executable computer program can be executed by a processor to implement the following. Figure 3 The method of the embodiment shown. The readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0118] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0119] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that: Applied to data synchronization systems, including: Obtaining initial source data from a source database; Generate fingerprint data of the initial source data to obtain target source data containing the fingerprint data; the fingerprint data is used as the primary key of the initial source data; Synchronize the target source data to a target database, so that the target database stores the target source data after the primary key verification passes; The method further comprises: Acquire historical task information; count the column combinations in the target source data used in the historical task information; generate fingerprint data according to the column data of the column combination, and synchronously store the fingerprint data as a secondary key of the target source data in the target database.

2. The method according to claim 1, characterized in that Generating fingerprint data of the initial source data includes: Call the preset fingerprint generation model; The initial source data is input into the fingerprint generation model as input data of the fingerprint generation model to obtain the fingerprint data.

3. The method according to claim 2, characterized in that The preset fingerprint generation model is implemented using at least one of a message digest algorithm, a secure hash algorithm, a message authentication code algorithm, and a key-based message authentication code algorithm.

4. The method according to claim 1, characterized in that: Synchronizing the target source data to the target database includes: Generate verification fingerprint data based on initial source data in the target source data; When the primary key in the target source data is identical to the verification fingerprint data, the target source data is sent to the target database.

5. The method according to claim 4, characterized in that The method further comprises: The target database obtains the primary key in the target source data and matches the primary key of the stored data; When the primary key of the target source data exists in the target database, updating the target source data to the target database; When the primary key of the target source data does not exist in the target database, the target source data is inserted into the target database.

6. The method according to claim 1, characterized in that Synchronizing the target source data to the target database includes: Acquire first fingerprint data of data columns other than the newly added data columns and second fingerprint data of data columns including the newly added data columns; Matching the first fingerprint data with a primary key of the target database; When the primary key of the target source data exists in the target database, the target source data and the second fingerprint data are updated to the target database.

7. A data synchronization system, characterized in that: It includes a source database, a target database and a data synchronization device; The data synchronization device is used to obtain the initial source data from the source database and generate fingerprint data of the initial source data to obtain the target source data including the fingerprint data; the fingerprint data is used as the primary key of the initial source data; and synchronizing the target source data to a target database, so that the target database stores the target source data after the primary key verification passes; The data synchronization device is also used to obtain historical task information; Counting the column combinations in the target source data used in the historical task information; generating fingerprint data according to the column data of the column combinations, and synchronously storing the fingerprint data in the target database as a secondary key of the target source data.

8. The system according to claim 7, characterized in that The data synchronization device comprises a Framework module, a fingerprint data generation module, a reading plug-in and a writing plug-in; the Framework module is connected to the reading plug-in and the writing plug-in respectively, The reading plug-in is used to read the initial source data to be synchronized from the source database; The fingerprint data generating module is used to generate fingerprint data of the initial source data and use the fingerprint data as the primary key of the initial source data; The Framework module is used to forward the initial source data and the primary key as target source data to the write plug-in; The write plug-in is used to write the target source data into the target database.

9. The system according to claim 8, characterized in that The fingerprint data generation module is integrated into the reading plug-in and / or Framework module.

10. The system according to claim 8, characterized in that The fingerprint data generation module is integrated into the write plug-in, and is used to generate verification fingerprint data based on the initial source data in the target source data; the write plug-in is also used to compare the primary key in the target source data with the verification fingerprint data, and when the primary key and the verification fingerprint data are the same, send the target source data to the target database.

11. The system according to claim 7, characterized in that The plug-in written in the data synchronization device is also used to: obtain first fingerprint data of data columns except the newly added data columns and second fingerprint data of data columns including the newly added data columns; match the first fingerprint data and the primary key of the target database; when the primary key of the target source data exists in the target database, update the target source data and the second fingerprint data to the target database.

12. A non-transitory computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 6 can be implemented.

Images