Data synchronization method, device and equipment, storage medium and computer program product

Abstract

The embodiment of the invention discloses a data synchronization method. The method comprises the steps that primary key information of a to-be-synchronized data table in a source database is determined; determining a target data table used for synchronously storing data in the to-be-synchronized data table in a target database; determining data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and synchronous configuration; based on the data processing parameters and the synchronization configuration, processing data in the to-be-synchronized data table to obtain a target synchronization data set; and synchronizing the target synchronous data set to the target data table. The embodiment of the invention also discloses a data synchronization device and equipment, a storage medium and a computer program product.

Description

Technical Field

[0001] The present application relates to the field of big data application technology, and in particular to a data synchronization method, apparatus, device, storage medium and computer program product. Background Art

[0002] With the development of Internet technology, a large amount of data is generated in the process of Internet application. How to effectively utilize this data and convert it into effective economic or research value has become a hot topic in current research. The method of obtaining this data is usually to collect and store data at the data generation end, and then aggregate and synchronize it to the data aggregation end. However, at present, when synchronizing data from some databases to other databases, for example, when using Flink to synchronize relational databases to other destination databases, the synchronized data needs to be processed by data conversion, such as extraction, transformation, and loading before synchronization operations are performed. In this implementation process, ETL consumes a huge amount of central processing unit (CPU), resulting in a decrease in task execution rate, while reducing the data extraction capability, resulting in a slow data synchronization rate. Summary of the invention

[0003] In view of this, the embodiments of the present application hope to provide a data synchronization method, apparatus, equipment, storage medium and computer program product, which solve the problem of slow data synchronization efficiency when synchronizing data between databases, and propose a data synchronization method, which reduces the amount of data conversion processing tasks during the data synchronization process, ensures the data synchronization rate, and improves the task processing efficiency.

[0004] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0005] The present application provides a data synchronization method, the method comprising:

[0006] Determine the primary key information of the data table to be synchronized in the source database;

[0007] Determine a target data table in a target database for synchronously storing the data in the data table to be synchronized;

[0008] Determine data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration;

[0009] Based on the data processing parameters and the synchronization configuration, the data in the to-be-synchronized data table is processed to obtain a target synchronization data set;

[0010] Synchronize the target synchronization data set to the target data table.

[0011] In the above solution, determining the primary key information of the data table to be synchronized in the source database includes:

[0012] Determine the data table to be synchronized in the source database;

[0013] Querying the table creation statement of the data table to be synchronized to obtain the first table field information of the data table to be synchronized;

[0014] The primary key information is obtained based on the first table field information.

[0015] In the above solution, determining the data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration includes:

[0016] Based on the data table to be synchronized, the target data table and the synchronization configuration, determining the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized; wherein the data processing parameters include the data objects to be converted and the data objects to be directly synchronized;

[0017] Based on the primary key information, determine a first splitting parameter and a second splitting parameter; wherein the first splitting parameter is used to perform data splitting on the data corresponding to the data object to be converted in the data table to be synchronized, and the second splitting parameter is used to perform data splitting on the data corresponding to the direct synchronization data object in the data table to be synchronized, and the data processing parameters include the first splitting parameter and the second splitting parameter.

[0018] In the above solution, determining the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized based on the data table to be synchronized, the target data table and the synchronization configuration includes:

[0019] Obtaining second table field information of the target data table;

[0020] Determine at least one first data object to be synchronized in the to-be-synchronized data table in the synchronization configuration and at least one second data object to be synchronized in the target data table;

[0021] Determine a first data type of each of the first data objects based on first table field information of the data table to be synchronized;

[0022] Determine the second data type of each of the second data objects based on the second table field information;

[0023] Determine, from at least one of the first data objects, a first data object whose first data type is different from the second data type of the corresponding second data object configured in the synchronization configuration, to obtain the object to be converted;

[0024] From at least one of the first data objects, an object configured in the synchronization configuration other than the object to be converted is determined to obtain the direct synchronization data object.

[0025] In the above solution, determining the first splitting parameter and the second splitting parameter based on the primary key information includes:

[0026] Determine the parallelism of task execution;

[0027] Based on the primary key information and the task execution parallelism, the first splitting parameter and the second splitting parameter are determined.

[0028] In the above solution, the data in the to-be-synchronized data table is processed based on the data processing parameters and the synchronization configuration to obtain a target synchronization data set, including:

[0029] Splitting the data corresponding to the data object to be converted from the to-be-synchronized data table according to the first splitting parameter to obtain at least one first data set;

[0030] Performing conversion processing on each of the first data sets according to the conversion processing configuration configured in the synchronization configuration to obtain a corresponding second data set; wherein the target synchronization data set includes the second data set corresponding to each of the first data sets;

[0031] caching the corresponding second data set;

[0032] Splitting the data corresponding to the direct synchronization data object from the to-be-synchronized data table according to the second splitting parameter to obtain at least one third data set;

[0033] Each of the third data sets is cached; wherein the target synchronization data set includes at least one of the third data sets.

[0034] In the above solution, synchronizing the target synchronization data set to the target data table includes:

[0035] Determine the parallelism of task execution;

[0036] According to the task execution parallelism, each of the second data set and the third data set is synchronized in parallel with the target data table.

[0037] The present application provides a data synchronization device, the device comprising: a first determination unit, a second determination unit, a processing unit and a synchronization unit; wherein:

[0038] The first determining unit is used to determine the primary key information of the data table to be synchronized in the source database;

[0039] The first determining unit is further used to determine a target data table in a target database for synchronously storing the data in the data table to be synchronized;

[0040] The second determining unit is used to determine the data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration;

[0041] The processing unit is used to process the data in the to-be-synchronized data table based on the data processing parameters and the synchronization configuration to obtain a target synchronization data set;

[0042] The synchronization unit is used to synchronize the target synchronization data set to the target data table.

[0043] The present application provides a data synchronization device, the device comprising at least: a memory, a processor and a communication bus; wherein:

[0044] The memory is used to store executable instructions;

[0045] The communication bus is used to realize the communication connection between the processor and the memory;

[0046] The processor is used to execute the data synchronization program stored in the memory to implement the steps of the data synchronization method as described in any one of the above items.

[0047] The present application provides a storage medium, on which a data synchronization program is stored. When the data synchronization program is executed by a processor, the steps of the data synchronization method as described in any one of the above items are implemented.

[0048] The present application provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data synchronization methods when executed by a processor.

[0049] The data synchronization method, apparatus, device, storage medium and computer program product provided by the embodiments of the present application, after determining the primary key information of the data table to be synchronized in the source database, determining the target data table in the target database for synchronously storing the data in the data table to be synchronized, based on the data table to be synchronized, the target data table, the primary key information and the synchronization configuration, determine the data processing parameters, based on the data processing parameters and the synchronization configuration, process the data in the data table to be synchronized to obtain the target synchronization data set, and synchronize the target synchronization data set to the target data table. In this way, by analyzing the primary key information and the data table to be synchronized in the source database, the target data table in the target database and the synchronization configuration, after determining to obtain the data processing parameters, the data processing parameters and the synchronization configuration are used to process the data in the data table to be synchronized to obtain the target synchronization data set for synchronization to the target data table, which solves the problem of slow data synchronization efficiency when synchronizing data between databases at present, and proposes a data synchronization method, which reduces the amount of tasks for data conversion processing during data synchronization, ensures the data synchronization rate, and improves the task processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a data synchronization method according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The embodiment of the present application provides a data synchronization method, which is applied to a data synchronization device, referring to Figure 1 As shown, the method comprises the following steps:

[0056] Step 101: Determine the primary key information of the data table to be synchronized in the source database.

[0057] In the embodiment of the present application, the source database is a database that needs to be synchronized, for example, it can be a relational database. Table analysis is performed on the data table to be synchronized that needs to be synchronized in the source database to determine the primary key information included in the data table to be synchronized. The primary key information of the data table includes a combination of one or more columns, and its value can uniquely identify each row in the table. It can be used to enforce the entity integrity of the table, mainly for foreign key associations with other tables, as well as modification and deletion of text records.

[0058] The data synchronization device is capable of realizing communication between the source database and the target database, and can realize data synchronization processing operations of synchronizing the data in the source database to the target database. For example, Flink can be used to realize data synchronization processing.

[0059] Step 102: Determine a target data table in the target database for synchronously storing data in the data table to be synchronized.

[0060] In the embodiment of the present application, the target database is a database for storing the data in the source database to be synchronized. When the data in the source database is synchronized, the data in the source database is synchronized to the target data table included in the target database. The target database can be pre-specified, and similarly, the target data table is also pre-specified.

[0061] Step 103: Determine data processing parameters based on the data table to be synchronized, the target data table, the primary key information and the synchronization configuration.

[0062] In the embodiment of the present application, the synchronization configuration is a pre-configured configuration parameter used to indicate the relevant configuration parameters in the data synchronization process, for example, including the data to be synchronized, the location to be synchronized to the corresponding target data table, etc. The data table to be synchronized, the target data table, the synchronization configuration, and the primary key information of the data table to be synchronized are analyzed to determine the processing method for the synchronized data in the synchronization configuration process, and obtain the data processing parameters.

[0063] Step 104: Based on the data processing parameters and synchronization configuration, the data in the to-be-synchronized data table is processed to obtain a target synchronization data set.

[0064] In the embodiment of the present application, according to the data processing parameters and based on the synchronization configuration, the data in the to-be-synchronized data table is analyzed and processed to obtain a target synchronization data set.

[0065] Step 105: Synchronize the target synchronization data set to the target data table.

[0066] In the embodiment of the present application, the processed target synchronization data set is synchronized to the target data table of the target database. Correspondingly, the corresponding data in the data table to be synchronized in the source database can be deleted or retained, which can be determined by the actual application scenario, such as the remaining storage resources of the source database, or the storage data processing rules of the source database, and is not specifically limited here.

[0067] Based on the foregoing embodiment, in other embodiments of the present application, step 101 of determining the primary key information of the data table to be synchronized in the source database may be implemented by the following steps:

[0068] Determine the data table to be synchronized in the source database;

[0069] Query the table creation statement of the data table to be synchronized to obtain the first table field information of the data table to be synchronized;

[0070] Based on the field information of the first table, primary key information is obtained.

[0071] In an embodiment of the present application, determining the data table to be synchronized in the source database can be achieved through pre-configured table identification information of the data table to be synchronized. The table identification information is information used to uniquely identify the corresponding data table, for example, it can be the table name, serial number, etc. of the data table. The table field information contains the table attribute parameters of the data table, such as the column name, column type, and column precision included in the corresponding data table. The preset query method can be used to query the table creation statement of the data table to be synchronized to obtain the first table field information of the data table to be synchronized. The preset query method can be implemented, for example, by using a structured query language (SQL) statement query method. The first table field information is analyzed to determine the primary key indicated in the first table field information to obtain the primary key information.

[0072] Based on the foregoing embodiment, in other embodiments of the present application, step 103 determines the data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration by the following steps:

[0073] Based on the data table to be synchronized, the target data table and the synchronization configuration, determine the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized; wherein the data processing parameters include the data objects to be converted and the data objects to be directly synchronized;

[0074] Based on the primary key information, determine the first splitting parameter and the second splitting parameter; wherein the first splitting parameter is used to perform data splitting on the data corresponding to the data object to be converted in the synchronized data table, and the second splitting parameter is used to perform data splitting on the data corresponding to the data object directly synchronized in the synchronized data table, and the data processing parameters include the first splitting parameter and the second splitting parameter.

[0075] In an embodiment of the present application, according to the synchronization configuration, the data objects in the data table to be synchronized that need to be synchronized are determined, and the data objects in the target data table to be synchronized are determined; the data table to be synchronized is analyzed to determine the object type of the data objects in the data table to be synchronized that need to be synchronized, and similarly, the target data table is analyzed to determine the object type of the data objects in the target data table to be synchronized, and the data objects to be converted that need to be converted and the directly synchronized data objects that do not need to be converted are determined according to the data types corresponding to the determined data table to be synchronized and the target data table, and according to the determined primary key information, the first splitting parameter for splitting the data corresponding to the object to be converted and the second splitting parameter for splitting the data corresponding to the directly synchronized data object are determined, so that the data can be split to improve the data processing efficiency. It should be noted that the first splitting parameter and the second splitting parameter can be the same or different, which can be determined by the actual application scenario.

[0076] Based on the foregoing embodiment, in other embodiments of the present application, the steps are based on the data table to be synchronized, the target data table and the synchronization configuration, and determining the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized can be achieved by the following steps:

[0077] Get the second table field information of the target data table;

[0078] Determine at least one first data object to be synchronized in the to-be-synchronized data table in the synchronization configuration and at least one second data object to be synchronized in the target data table;

[0079] Determine the first data type of each first data object based on the first table field information of the data table to be synchronized;

[0080] Determine the second data type of each second data object based on the second table field information;

[0081] Determine, from at least one first data object, a first data object whose first data type is different from a second data type of a corresponding second data object configured in the synchronization configuration, and obtain an object to be converted;

[0082] From the at least one first data object, objects configured in the synchronization configuration other than the object to be converted are determined to obtain a direct synchronization data object.

[0083] In an embodiment of the present application, the table creation statement of the target data table is queried, and the table creation statement of the target data table is analyzed to obtain the second table field information of the target data table. At least one first data object that needs to be synchronized in the data table to be synchronized is determined from the synchronization configuration, and at least one corresponding second data object in the target data table is determined at the same time. The correspondence between at least one first data object and at least one second data object is: synchronize the data corresponding to the first data object to the corresponding second data object. Exemplarily, the synchronization configuration is configured to synchronize the three columns of data (a, b, c) in the data table to be synchronized to (e, f, g) in the target data table, where a is synchronized to e, b is synchronized to f, and c is synchronized to g. From the first table field information of the data table to be synchronized, the first data type of each first data object is determined, and from the second table field information of the target data table, the second data type of each second data object is determined; wherein the data type can be a character, a numeric value, a Boolean, etc., and the specific character can include a character type and a string type, and the numeric value can include an integer type, a floating point type, etc. The data type of the data in the data table can be determined by the actual situation, and no specific decision is made here. Compare and analyze the first data type of at least one first data object and the second data type of the corresponding second data object to determine whether the first data type of each first data object is the same as the second data type of the corresponding second data object. If they are the same, the corresponding first data object can be determined as a direct synchronization data object. If they are different, the corresponding first data object can be determined as an object to be converted.

[0084] Based on the foregoing embodiment, in other embodiments of the present application, the step of determining the first splitting parameter and the second splitting parameter based on the primary key information can be implemented by the following steps:

[0085] Determine the parallelism of task execution;

[0086] Based on the primary key information and the task execution parallelism, a first splitting parameter and a second splitting parameter are determined.

[0087] In the embodiment of the present application, the task execution parallelism is a pre-configured number of tasks that can be executed in parallel at the same time when synchronizing data between the data table to be synchronized and the target data table. After determining the task execution parallelism, the data of each object to be converted and each directly synchronized data object can be split according to the primary key information and the task execution parallelism. Exemplarily, the primary key information is used as a reference for splitting, the amount of data corresponding to the primary key information is counted, and then the amount of data corresponding to the primary key information is scored as a multiple of the task execution parallelism, so that the corresponding amount of data can be synchronized each time according to the task execution parallelism.

[0088] Based on the above embodiment, in other embodiments of the present application, step 104 processes the data in the to-be-synchronized data table based on the data processing parameters and the synchronization configuration to obtain the target synchronization data set, which can be implemented by the following steps:

[0089] Splitting data corresponding to the data object to be converted from the to-be-synchronized data table according to a first splitting parameter to obtain at least one first data set;

[0090] Perform conversion processing on each first data set according to the conversion processing configuration configured in the synchronization configuration to obtain a corresponding second data set; wherein the target synchronization data set includes the second data set corresponding to each first data set;

[0091] caching a corresponding second data set;

[0092] Splitting the data corresponding to the direct synchronization data object from the to-be-synchronized data table according to the second splitting parameter to obtain at least one third data set;

[0093] Each third data set is cached; wherein the target synchronization data set includes at least one third data set.

[0094] In an embodiment of the present application, from the table of data to be synchronized, the conversion processing configuration includes conversion processing requirements for data that need to be converted. It should be noted that when the data corresponding to the data object to be converted is split according to the first splitting parameter and the data corresponding to the direct synchronization data object is split according to the second splitting parameter, it can be executed in parallel according to the task execution parallelism. After the data is split, the data is cached, which can reduce the consumption of the CPU when the data is read again. In this way, by distinguishing the synchronization objects, the objects that need to be converted and the objects that do not need to be converted, the data processing pressure is reduced.

[0095] Based on the above embodiment, in other embodiments of the present application, step 105 of synchronizing the target synchronization data set to the target data table can be implemented by the following steps:

[0096] Determine the parallelism of task execution;

[0097] According to the task execution parallelism, each second data set and third data set value target data table are synchronized in parallel.

[0098] In the embodiment of the present application, the data sets in the target synchronization data set are synchronized according to the task execution parallelism, and multiple parallel tasks are executed simultaneously, thereby improving the synchronization rate.

[0099] Based on the foregoing embodiments, an embodiment of the present application provides a data synchronization method, which is mainly based on Flink to implement the synchronization process. First, some corresponding terms are introduced to facilitate understanding of the content of this embodiment.

[0100] Flink: is an open source stream processing framework that supports efficient, scalable, and fault-tolerant stream and batch processing. The core of Flink is a distributed data stream engine that can perform real-time computing on large-scale data sets. Flink provides a rich set of application programming interfaces (APIs) and tools that can be used to build various types of stream processing and batch processing applications. Flink's features include: Efficient stream processing: Flink's stream processing engine can achieve millisecond-level latency while supporting high-throughput data processing; Scalability: Flink can run on large-scale clusters and support dynamic expansion and reduction of cluster size; Fault tolerance: Flink can ensure the correctness of data processing in the event of node failure or network anomalies; Multiple APIs: Flink provides multiple APIs, including DataStreamAPI, DataSet API, and Table API, which can meet the needs of different types of applications; Support for multiple data sources and data formats: Flink can process multiple data sources and data formats, including files, Kafka, Hadoop, Cassandra, etc. Flink has a wide range of applications in real-time data processing, streaming ETL, real-time machine learning, and other fields.

[0101] ETL: is a process commonly used in data warehouses and data integration. It includes the following three main steps: Extraction: Get data from different data sources. The data source can be a relational database, file, Web service, API, etc. In this step, the data is extracted and usually saved in its original, unprocessed form; Transformation: Clean, integrate and transform the extracted data. The purpose of this step is to convert the data into a format and structure suitable for the target system. Common transformation operations include data cleaning, data filtering, data merging, data calculation, data format conversion, etc.; Loading: Load the transformed data into the target system, usually a data warehouse or data integration platform. During the loading process, the data is stored in the data structure of the target system for subsequent data analysis and query operations.

[0102] The main goal of ETL is to integrate scattered and heterogeneous data sources into a unified data warehouse to support the decision-making analysis and business needs of the enterprise. Through the process of extraction, transformation and loading, ETL can clean and integrate data, improve data quality, and make data consistent and available between different systems. Furthermore, the ETL process may also involve other operations such as data verification, data supplementation, and data segmentation. The specific implementation method and tool selection depend on the specific needs and technical environment.

[0103] Based on the above description, an implementation process of the corresponding data synchronization method can refer to Figure 2 As shown, at least the following steps are included:

[0104] Step a11, start.

[0105] Step a12: Determine the table field information of the source database.

[0106] The SQL statement is used to query the table creation statement of the source database table in the source database to provide relevant information for the subsequent processing process. The table field information of the source database table is obtained according to the table creation statement. The table field information can be a list, which at least includes information such as column name, column type, and column precision.

[0107] Step a13: query the primary key information of the source database table based on the table field information.

[0108] Among them, most relational database tables have primary key information, which is usually used to ensure the correctness and consistency of data and improve data access efficiency. It is also the basis for realizing data association and query.

[0109] Step a14: determine the table field information of the target data table in the target database.

[0110] The purpose of querying the field information of the target table is to determine whether there is a user-defined ETL operation when synchronizing from the source database to the target database. The queried field information of the target table is also a list, which contains information such as column name, column type, and column precision.

[0111] Step a15: Obtain ETL transformation rules.

[0112] For example, if the source database table has [a, b, c] fields, but the target data table has [d, e, f] fields, and the data types of the two do not match, the user configures the synchronization of field a data to field f, field b data to field e, and field c data to field d. At this time, the conversion rules are read from the configuration file and cached to facilitate subsequent reading, ensuring that the configuration is not repeatedly read from the configuration file during subsequent task execution.

[0113] Step a16: Based on the ETL transformation rules, determine whether ETL is required.

[0114] Among them, to determine whether the synchronized data needs to be ETL operated, the field type comparison and field quantity comparison can be performed. If the field types of the source database table and the target data table do not need to be converted and the number of fields is exactly the same in the configured ETL synchronization rules, it is determined that ETL is not required, corresponding to the aforementioned direct synchronization of data objects. If the field types of the source database table and the target data table do not need to be converted and the number of fields is not exactly the same, it means that ETL operation is required, corresponding to the aforementioned data object to be converted. At this time, the fields in the source database table that require ETL operation when synchronizing data can be cached to reduce the CPU resource consumption when traversing all rows and columns when the corresponding data is converted. For data that does not require ETL, the conversion operation can be directly skipped during synchronization, which can speed up the synchronization rate.

[0115] Step a17: perform query splitting and conversion processing on the source database table based on the primary key information.

[0116] Among them, the query is optimized according to the primary key type. For example, when the primary key name is id and the type is an integer type, the data corresponding to the primary key in the source database table is split according to the user-configured synchronization task parallelism (N). Correspondingly, when splitting, the query statement SELECT*FROM table WHERE id%N=[0,N-1] can be used to implement it. If the type is a string type, the query statement SELECT*FROM table WHERE hash(id)%N=[0,N-1] can be used to implement it. In this way, when querying and splitting from the source database table, the data in the table can be read in parallel through N SQL statements, so that the reading rate will be increased by N times. This is also one of the means to optimize the synchronization task. At this time, when splitting data, the data that needs to be processed by ETL and the data that does not need to be processed by ETL are split and processed in parallel according to the Flink parallelism. The data that needs to be processed by ETL is transformed according to the ETL transformation rules to obtain the transformed data and cache it.

[0117] For example, during the conversion process, the data that needs to be converted can be cached. For example, in the case of case insensitivity, it needs to be converted to lowercase for processing. At this time, the key data that has appeared can be cached. After the cache is aligned, it can be read directly from the cache, which can effectively reduce data traversal, thereby reducing the consumption of CPU resources so that CPU resources can be used in data synchronization.

[0118] Step a18: Synchronize data based on Flink parallelism.

[0119] Among them, Flink parallelism refers to the number of tasks that can be executed in parallel in a Flink application. In Flink, each task can be assigned to one or more parallel instances for execution. The setting of parallelism affects the performance and resource utilization of the Flink application.

[0120] Furthermore, Flink parallelism can be set in the code or in the Flink configuration file. Specifically, when Flink parallelism is set in the code, it can be implemented using the setParallelism() method. When setting Flink parallelism in the configuration file, it can be implemented using the parallelism.default property.

[0121] In this way, when writing synchronization data to the target data table, the data is written according to the split parallelism, which can speed up the writing speed and thus improve the data synchronization rate of the entire link.

[0122] Step a19, end.

[0123] The data synchronization method provided by the embodiment of the present application determines the primary key information of the data table to be synchronized in the source database, determines the target data table in the target database for synchronously storing the data in the data table to be synchronized, determines the data processing parameters based on the data table to be synchronized, the target data table, the primary key information and the synchronization configuration, processes the data in the data table to be synchronized based on the data processing parameters and the synchronization configuration to obtain the target synchronization data set, and synchronizes the target synchronization data set to the target data table. In this way, by analyzing the primary key information and the data table to be synchronized in the source database, the target data table in the target database and the synchronization configuration, and determining the data processing parameters, the data in the data table to be synchronized is processed using the data processing parameters and the synchronization configuration to obtain the target synchronization data set for synchronization to the target data table, which solves the problem of slow data synchronization efficiency when synchronizing data between databases at present, and proposes a data synchronization method, which reduces the amount of tasks for data conversion processing during data synchronization, ensures the data synchronization rate, and improves the task processing efficiency.

[0124] Based on the above embodiments, the present application provides a data synchronization device. The data synchronization device 2 can be applied to Figure 1 And the corresponding embodiments, refer to Figure 3 As shown, the data synchronization device 2 includes: a first determination unit 21, a second determination unit 22, a processing unit 23 and a synchronization unit 24; wherein:

[0125] A first determining unit 21 is used to determine the primary key information of the data table to be synchronized in the source database;

[0126] The first determining unit 21 is further used to determine a target data table in the target database for synchronously storing the data in the data table to be synchronized;

[0127] A second determining unit 22, configured to determine data processing parameters based on the data table to be synchronized, the target data table, the primary key information and the synchronization configuration;

[0128] A processing unit 23 is used to process the data in the to-be-synchronized data table based on the data processing parameters and the synchronization configuration to obtain a target synchronization data set;

[0129] The synchronization unit 24 synchronizes the target synchronization data set to the target data table.

[0130] In other embodiments of the present application, when the first determination unit executes the step of determining the primary key information of the data table to be synchronized in the source database, it includes: a first determination module, a query module and a obtaining module; wherein:

[0131] A first determination module is used to determine the data table to be synchronized in the source database;

[0132] A query module is used to query the table creation statement of the data table to be synchronized, and obtain the first table field information of the data table to be synchronized;

[0133] The obtaining module is used to obtain primary key information based on the field information of the first table.

[0134] In other embodiments of the present application, when the first determination execution step determines the data processing parameters based on the data table to be synchronized, the target data table, the primary key information and the synchronization configuration, it also includes: a second determination module; wherein:

[0135] A second determination module is used to determine the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized based on the data table to be synchronized, the target data table and the synchronization configuration; wherein the data processing parameters include the data objects to be converted and the data objects to be directly synchronized;

[0136] The second determination module is also used to determine the first splitting parameter and the second splitting parameter based on the primary key information; wherein the first splitting parameter is used to perform data splitting on the data corresponding to the data object to be converted in the synchronized data table, and the second splitting parameter is used to perform data splitting on the data corresponding to the directly synchronized data object in the synchronized data table, and the data processing parameters include the first splitting parameter and the second splitting parameter.

[0137] In other embodiments of the present application, the second determination unit includes: a first acquisition module and a third determination module; wherein:

[0138] A first acquisition module is used to acquire the second table field information of the target data table;

[0139] A third determination module is used to determine at least one first data object to be synchronized in the to-be-synchronized data table and at least one second data object to be synchronized in the target data table in the synchronization configuration;

[0140] The third determination module is further used to determine the first data type of each first data object based on the first table field information of the data table to be synchronized;

[0141] The third determination module is further used to determine the second data type of each second data object based on the second table field information;

[0142] The third determination module is further used to determine, from at least one first data object, a first data object whose first data type is different from a second data type of a corresponding second data object configured in the synchronization configuration, to obtain an object to be converted;

[0143] The third determination module is further used to determine the objects configured in the synchronization configuration except the object to be converted from the at least one first data object, and obtain the direct synchronization data object.

[0144] In the embodiment of the present application, the second determination module is specifically used to implement the following steps:

[0145] Determine the parallelism of task execution;

[0146] Based on the primary key information and the task execution parallelism, a first splitting parameter and a second splitting parameter are determined.

[0147] In other embodiments of the present application, the processing unit includes: a second acquisition module, a processing module and a cache module; wherein:

[0148] A second acquisition module is used to split the data corresponding to the data object to be converted from the to-be-synchronized data table according to the first splitting parameter to obtain at least one first data set;

[0149] A processing module, configured to perform conversion processing on each first data set according to the conversion processing configuration configured in the synchronization configuration to obtain a corresponding second data set; wherein the target synchronization data set includes the second data set corresponding to each first data set;

[0150] A cache module, used for caching the corresponding second data set;

[0151] A second acquisition module is used to split the data corresponding to the direct synchronization data object from the to-be-synchronized data table according to the second splitting parameter to obtain at least one third data set;

[0152] The cache module is used to cache each third data set; wherein the target synchronization data set includes at least one third data set.

[0153] In other embodiments of the present application, the synchronization unit includes: a fourth determination module and a synchronization module; wherein:

[0154] The fourth determination module is used to determine the parallelism of task execution;

[0155] The synchronization mode is used to synchronize each second data set and the third data set value target data table in parallel according to the task execution parallelism.

[0156] It should be noted that the information interaction process between the units and modules in this embodiment can refer to the information interaction process described in the aforementioned method embodiment, which will not be repeated here.

[0157] The data synchronization device provided by the embodiment of the present application determines the primary key information of the data table to be synchronized in the source database, determines the target data table in the target database for synchronously storing the data in the data table to be synchronized, determines the data processing parameters based on the data table to be synchronized, the target data table, the primary key information and the synchronization configuration, processes the data in the data table to be synchronized based on the data processing parameters and the synchronization configuration to obtain the target synchronization data set, and synchronizes the target synchronization data set to the target data table. In this way, by analyzing the primary key information and the data table to be synchronized in the source database, the target data table in the target database and the synchronization configuration, and determining the data processing parameters, the data in the data table to be synchronized is processed using the data processing parameters and the synchronization configuration to obtain the target synchronization data set for synchronization to the target data table, which solves the problem of slow data synchronization efficiency when synchronizing data between databases at present, and proposes a data synchronization method, which reduces the amount of data conversion processing tasks during data synchronization, ensures the data synchronization rate, and improves the task processing efficiency.

[0158] Based on the above embodiments, the embodiments of the present application provide a data synchronization device, which can be applied to Figure 1 And the corresponding embodiments, refer to Figure 4 As shown, the data synchronization device 3 at least includes: a memory 31, a processor 32 and a communication bus 33; wherein:

[0159] A memory 31, used for storing executable instructions;

[0160] A communication bus 33, used to realize the communication connection between the processor and the memory;

[0161] The processor 32 is used to execute the data synchronization program stored in the memory to implement the following Figure 1 The corresponding embodiments correspond to the steps of the data synchronization method.

[0162] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, referred to as a storage medium, which stores one or more data synchronization methods, and the one or more data synchronization methods can be executed by one or more processors to implement the following. Figure 1 The data synchronization method provided by the corresponding embodiment will not be described in detail here.

[0163] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium, referred to as a storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement reference Figure 1 The implementation process of the method provided in the corresponding embodiment will not be repeated here.

[0164] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0165] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, ..., an air conditioner, or a network communication link device, etc.) to execute the methods described in each embodiment of the present application.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0170] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data synchronization method, characterized in that: The method comprises: Determine the primary key information of the data table to be synchronized in the source database; Determine a target data table in a target database for synchronously storing the data in the data table to be synchronized; Determine data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration; Based on the data processing parameters and the synchronization configuration, the data in the to-be-synchronized data table is processed to obtain a target synchronization data set; Synchronize the target synchronization data set to the target data table.

2. The method according to claim 1, characterized in that The step of determining the primary key information of the data table to be synchronized in the source database includes: Determine the data table to be synchronized in the source database; Querying the table creation statement of the data table to be synchronized to obtain the first table field information of the data table to be synchronized; The primary key information is obtained based on the first table field information.

3. The method according to claim 1, characterized in that The determining of data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration includes: Based on the data table to be synchronized, the target data table and the synchronization configuration, determining the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized; wherein the data processing parameters include the data objects to be converted and the data objects to be directly synchronized; Based on the primary key information, determine a first splitting parameter and a second splitting parameter; wherein the first splitting parameter is used to perform data splitting on the data corresponding to the data object to be converted in the data table to be synchronized, and the second splitting parameter is used to perform data splitting on the data corresponding to the direct synchronization data object in the data table to be synchronized, and the data processing parameters include the first splitting parameter and the second splitting parameter.

4. The method according to claim 3, characterized in that: The step of determining the data objects to be converted and the data objects to be directly synchronized in the data table to be synchronized based on the data table to be synchronized, the target data table and the synchronization configuration includes: Obtaining second table field information of the target data table; Determine at least one first data object to be synchronized in the to-be-synchronized data table in the synchronization configuration and at least one second data object to be synchronized in the target data table; Determine a first data type of each of the first data objects based on first table field information of the data table to be synchronized; Determine the second data type of each of the second data objects based on the second table field information; Determine, from at least one of the first data objects, a first data object whose first data type is different from the second data type of the corresponding second data object configured in the synchronization configuration, to obtain the object to be converted; From at least one of the first data objects, an object configured in the synchronization configuration other than the object to be converted is determined to obtain the direct synchronization data object.

5. The method according to claim 3, characterized in that: The determining the first splitting parameter and the second splitting parameter based on the primary key information includes: Determine the parallelism of task execution; Based on the primary key information and the task execution parallelism, the first splitting parameter and the second splitting parameter are determined.

6. The method according to claim 3, characterized in that The step of processing the data in the to-be-synchronized data table based on the data processing parameters and the synchronization configuration to obtain a target synchronization data set includes: Splitting the data corresponding to the data object to be converted from the to-be-synchronized data table according to the first splitting parameter to obtain at least one first data set; Performing conversion processing on each of the first data sets according to the conversion processing configuration configured in the synchronization configuration to obtain a corresponding second data set; wherein the target synchronization data set includes the second data set corresponding to each of the first data sets; caching the corresponding second data set; Splitting the data corresponding to the direct synchronization data object from the to-be-synchronized data table according to the second splitting parameter to obtain at least one third data set; Each of the third data sets is cached; wherein the target synchronization data set includes at least one of the third data sets.

7. The method according to claim 6, characterized in that The synchronizing the target synchronization data set to the target data table includes: Determine the parallelism of task execution; According to the task execution parallelism, each of the second data set and the third data set is synchronized in parallel with the target data table.

8. A data synchronization device, characterized in that: The device comprises: a first determining unit, a second determining unit, a processing unit and a synchronization unit; wherein: The first determining unit is used to determine the primary key information of the data table to be synchronized in the source database; The first determining unit is further used to determine a target data table in a target database for synchronously storing the data in the data table to be synchronized; The second determining unit is used to determine the data processing parameters based on the to-be-synchronized data table, the target data table, the primary key information and the synchronization configuration; The processing unit is used to process the data in the to-be-synchronized data table based on the data processing parameters and the synchronization configuration to obtain a target synchronization data set; The synchronization unit is used to synchronize the target synchronization data set to the target data table.

9. A data synchronization device, characterized in that: The device comprises at least: a memory, a processor and a communication bus; wherein: The memory is used to store executable instructions; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the data synchronization program stored in the memory to implement the steps of the data synchronization method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a data synchronization program, which, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the data synchronization method according to any one of claims 1 to 7.

Images