Whole database synchronization method and electronic equipment

Abstract

The invention discloses a whole database synchronization method and electronic equipment. The method comprises the following steps: constructing a data connection task; in response to a task starting operation of a user, obtaining a source parameter of a source database and a target parameter of a target database corresponding to the data connection task; according to connection information in the source parameters, a source database is connected through a source connector, and a source data structure of each data table in the source database is obtained; creating a source association table in the source connector according to the source data structure; according to connection information in the target parameter, connecting a target database through a target connector, and creating a target data structure of each data table in the target database; creating a target association table in the target connector according to the target data structure; and if the data change in the source association table is obtained, inserting the changed data into the target association table so as to synchronize the changed data to the target physical library through the target association table. The database synchronization workload is reduced, and the synchronization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and in particular to a method and electronic device for synchronizing an entire database. Background Art

[0002] A database is a container for storing tables and other related structures. The relationship between tables and databases is that a database can contain multiple tables. Tables are objects used to store data in a database. They are a collection of structured data and the basis of the entire database system. Each database is composed of several data tables. That is, data tables are used to store data in a database.

[0003] Traditional data access is single-table access. Therefore, if you want to synchronize the data of the entire database, you usually need to create the same number of tasks as the tables, and at the same time, the same number of database connections as the tables will be occupied. In this way, all database connections are occupied by tasks, resulting in other services being unable to access the database normally. Summary of the invention

[0004] In an exemplary embodiment of the present application, a method and electronic device for synchronizing an entire database are provided to reduce the workload of table creation during database synchronization and improve the efficiency of database synchronization.

[0005] According to a first aspect of an exemplary implementation, a method for synchronizing an entire database is provided, comprising:

[0006] Build data connection tasks;

[0007] In response to a task start operation by a user, a source parameter of a source database and a target parameter of a target database corresponding to the data connection task are obtained;

[0008] According to the connection information in the source parameters, connect to the source database through the source connector and obtain the source data structure of each data table in the source database;

[0009] According to the source data structure, a source association table is created in the source connector; wherein the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database;

[0010] According to the connection information in the target parameters, connect to the target database through the target connector, and create the target data structure of each data table in the target database;

[0011] According to the target data structure, a target association table is created in the target connector; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database;

[0012] If data changes in the source association table are obtained, the changed data is inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0013] According to a second aspect of an exemplary embodiment, there is provided an electronic device, including a task construction unit and a processor:

[0014] Task building blocks are configured to execute:

[0015] Build data connection tasks;

[0016] A processor configured to execute:

[0017] In response to a task start operation by a user, a source parameter of a source database and a target parameter of a target database corresponding to the data connection task are obtained;

[0018] According to the connection information in the source parameters, connect to the source database through the source connector and obtain the source data structure of each data table in the source database;

[0019] According to the source data structure, a source association table is created in the source connector; wherein the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database;

[0020] According to the connection information in the target parameters, connect to the target database through the target connector, and create the target data structure of each data table in the target database;

[0021] According to the target data structure, a target association table is created in the target connector; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database;

[0022] If data changes in the source association table are obtained, the changed data is inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0023] According to a third aspect of an exemplary implementation, a device for synchronizing an entire database is provided, comprising:

[0024] Task building unit, used to: build data connection tasks;

[0025] A parameter acquisition unit, used to: in response to a task start operation by a user, acquire a source parameter of a source database and a target parameter of a target database corresponding to a data connection task;

[0026] A first connection unit is used to: connect to the source database through the source connector according to the connection information in the source parameter and obtain the source data structure of each data table in the source database;

[0027] A source association table creation unit is used to: create a source association table in a source connector according to a source data structure; wherein the source association table represents the relationship between each data table in a source database and each data table in a source physical library corresponding to the source database;

[0028] The second connection unit is used to: connect to the target database through the target connector according to the connection information in the target parameter, and create a target data structure for each data table in the target database;

[0029] The target association table creation unit is used to: create a target association table in the target connector according to the target data structure; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database;

[0030] The synchronization unit is used to: if the data change in the source association table is obtained, insert the changed data into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0031] According to a fourth aspect of the exemplary implementation, a computer storage medium is provided, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the method for synchronizing the entire database as described in the first aspect.

[0032] In order to reduce the number of database connection occupancy and the memory occupied, the embodiment of the present application is designed to occupy only one database connection. In this process, a data access task is first constructed, and then the task is started, and the whole database synchronization process of the database is realized in the task. In the synchronization process, first, according to the connection information in the source parameter, the source database is connected through the source connector and the source data structure of each data table in the source database is obtained; according to the source data structure, a source association table is created in the source connector; the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database. Then, according to the connection information in the target parameter, the target database is connected through the target connector, and the target data structure of each data table in the target database is created; according to the target data structure, a target association table is created in the target connector; the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database. Finally, if the data change in the source association table is obtained, the changed data is inserted into the target association table to synchronize the changed data to the target physical library through the target association table. In this process, the relationship between the virtual table and the physical table is established through the associated library. Only one data access task needs to be created to complete the access of the entire library, which only occupies one database connection, reduces the database pressure, and improves the efficiency of database synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 A schematic diagram exemplarily shows a database synchronization process provided by an embodiment of the present application;

[0035] Figure 2 A schematic diagram exemplarily shows an application scenario of database synchronization provided by an embodiment of the present application;

[0036] Figure 3 A flowchart of a method for synchronizing an entire database provided by an embodiment of the present application is exemplarily shown;

[0037] Figure 4 A flowchart of a method for constructing a data connection task provided by an embodiment of the present application is exemplarily shown;

[0038] Figure 5 A schematic diagram exemplarily shows another database synchronization process provided by an embodiment of the present application;

[0039] Figure 6 A schematic diagram of a business process provided by an embodiment of the present application is exemplified;

[0040] Figure 7 An exemplary diagram of a system architecture for accessing a database of an entire database provided by an embodiment of the present application is shown;

[0041] Figure 8 A flowchart of a complete database synchronization method provided by an embodiment of the present application is exemplified;

[0042] Fig. 9 The following is a schematic diagram showing the structure of a device for synchronizing an entire database provided by an embodiment of the present application;

[0043] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is exemplarily shown. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0045] For ease of understanding, the terms involved in the embodiments of the present application are explained below:

[0046] (1) Physical table: The actual storage table in the data warehouse, including entity table, fact table, dimension table, etc. Physical table stores actual data and is the core part of the data warehouse.

[0047] (2) Logical table: In a data warehouse, a logical table is a logical table derived from multiple entity tables based on business needs. It does not store data but serves as an intermediate result of data query. A logical table can be understood as a virtual table that does not store data but generates results by querying entity tables.

[0048] In the embodiment of the present application, the data tables in the source database and the target database are both logical tables, the data tables in the source physical library and the target physical library are both physical tables, the source physical library corresponds to the source database, and the target physical library corresponds to the target database.

[0049] Traditional data access is single-table access. Therefore, if you want to synchronize the data of the entire database, you usually need to create the same number of tasks as the tables, and at the same time, the same number of database connections as the tables will be occupied. In this way, all database connections are occupied by tasks, resulting in other services being unable to access the database normally.

[0050] To this end, the embodiment of the present application provides a method for synchronizing the entire database. In this method, for a database, a data connection task is constructed to access all data in all data tables in the source database and synchronize them to the target database. Compared with traditional single-table tasks, it can greatly reduce the workload of table construction, improve synchronization efficiency, and save labor costs.

[0051] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0052] refer to Figure 1 , shows a database synchronization process, during which data in a source database is synchronized to a target database through a data transmission interface.

[0053] refer to Figure 2, shows an application scenario of database synchronization, taking n units as an example. Among them, in smart cities, data access is particularly extensive. The data between different units or enterprises are mostly independent. How to unify the data of different units or enterprises to form an original database, and then conduct data governance on the original database to form a theme database, a special database, and a basic database is an important link in smart cities. At present, to collect data from various units or enterprises, it is necessary to extract the front database of each unit or enterprise. This extraction requires the creation of a scheduled task corresponding to the number of tables, resulting in repeated work and cumbersome, which greatly reduces the efficiency of data access. In addition, the extracted data is mostly extracted on a scheduled basis, and the freshness of the data cannot be guaranteed. At present, the data access tools of many enterprises mostly establish data access tasks based on tables as dimensions, and cannot access data for an entire database.

[0054] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.

[0055] Combine the following Figure 2 The application scenario shown is shown in Figure 3 A flowchart of a method for synchronizing an entire database is shown to illustrate the technical solution provided by an embodiment of the present application.

[0056] S301: Building a data connection task.

[0057] S302: In response to a task start operation by a user, a source parameter of a source database and a target parameter of a target database corresponding to the data connection task are obtained.

[0058] S303: According to the connection information in the source parameters, connect to the source database through the source connector and obtain the source data structure of each data table in the source database.

[0059] S304: Creating a source association table in the source connector according to the source data structure.

[0060] The source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database.

[0061] S305: Connect to the target database through the target connector according to the connection information in the target parameters, and create target data structures for each data table in the target database.

[0062] S306: Create a target association table in the target connector according to the target data structure.

[0063] The target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database.

[0064] S307: If data changes in the source association table are obtained, the changed data is inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0065] In order to reduce the number of database connection occupancy and the memory occupied, the embodiment of the present application is designed to occupy only one database connection. In this process, a data access task is first constructed, and then the task is started, and the whole database synchronization process of the database is realized in the task. In the synchronization process, first, according to the connection information in the source parameter, the source database is connected through the source connector and the source data structure of each data table in the source database is obtained; according to the source data structure, a source association table is created in the source connector; the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database. Then, according to the connection information in the target parameter, the target database is connected through the target connector, and the target data structure of each data table in the target database is created; according to the target data structure, a target association table is created in the target connector; the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database. Finally, if the data change in the source association table is obtained, the changed data is inserted into the target association table to synchronize the changed data to the target physical library through the target association table. In this process, the relationship between the virtual table and the physical table is established through the associated library. Only one data access task needs to be created to complete the access of the entire library, which only occupies one database connection, reduces the database pressure, and improves the efficiency of database synchronization.

[0066] The process of synchronizing the entire database in the embodiment of the present application can be divided into two stages. The first stage is the process of building the data connection task, and the second stage is the process of synchronizing the database.

[0067] The first two stages: the process of constructing the data connection task in step S301.

[0068] Figure 4 A flow chart of a method for constructing a data connection task provided in an embodiment of the present application includes at least the following steps:

[0069] S401: In response to a task construction request from a user, configure source parameters of at least one source database and target parameters of at least one target database.

[0070] The electronic device may provide an operation page or portal to the user, and when a task construction request from the user is detected, source parameters of at least one source database and target parameters of at least one target database may be configured.

[0071] Exemplarily, the source parameters include source connection information, source database name, source database type, source user name and source password; the target parameters include target connection information, target database name, target database type, target user name and target password. The source connection information includes the internet protocol (IP) address and port of the source database, and the target connection information includes the IP address and port of the target database. For database types with unstructured storage such as HBASE, KAFKA, etc., the data parsing format needs to be configured, and two types of JSON and string are supported. The database type supports relational databases such as MYSQL, ORACLE, POSTGRESQL, DORIS, CLICKHOUSE, and large data type databases such as HIVE, HBASE, IMPALA, ELASTICSEARCH, KAFKA, etc.

[0072] In the actual application process, you can also configure the task running parameters. The task running parameters include the type of task running, which is periodic or real-time, that is, whether the task is a periodic task or a real-time task. Among them, periodic tasks are submitted through batch processing. Periodic tasks need to configure the running frequency of periodic tasks, and real-time tasks are submitted through stream processing. Periodic tasks and real-time tasks each correspond to different task running engines.

[0073] S402: In response to the user's operation on the task construction page, determining the source database and target database selected by the user.

[0074] The task construction page may display partial parameters of at least one source database and at least one target database for selection by the user, and the partial parameters may be respective database types, etc. The source database selected by the user is any one of the at least one pre-configured source databases, and the target database selected by the user is any one of the at least one pre-configured target databases.

[0075] During this process, you can also configure parameters such as whether to create a table in the target database and whether to perform special identification based on the source database.

[0076] S403: Building a data connection task according to the source database and the target database.

[0077] In the process of building a data connection task, based on the source database and the target database and according to the task running type, different task running engines are selected to build the data connection task.

[0078] In this way, after the data access task is constructed, the corresponding task running engine is executed, and the information of the data access task can be queried, and the task parameters can be obtained and the task can be executed.

[0079] The second stage: the database synchronization process of steps S302-S307.

[0080] In S302, the execution engine of the task is determined according to the execution type of the task. For example, if the execution type is a real-time task, the execution engine is A, and if the execution type is a periodic task, the execution engine is B. In response to the task start operation of the user, the execution engine of the task is connected to the data to obtain the source parameters of the source database and the target parameters of the target database. The types of the source parameters and the target parameters can be found in the above embodiments, which will not be described here.

[0081] In S303, the correspondence between the source database type and the connector is preset. Therefore, according to the database type in the source parameter, the connector corresponding to the source database type can be determined as source connector Connector1. According to the source IP and source port in the source parameter, the source database is connected through source connector Connector1 and the source data structure of each data table in the source database is obtained. The source data structure includes information such as the table name, the field name in the table, the field comment, the field type and the field length, and the primary key of the table.

[0082] This involves S304: creating a source association table in the source connector according to the source data structure.

[0083] Since the data table in the source database is a virtual table, in order to establish a connection between the virtual table and the real table (physical table), an associated table is used as a link.

[0084] Exemplarily, a source association table may be created in the source connector Connector1 according to the source data structure, and the source association table indicates the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database.

[0085] After obtaining the source data structure, apply the correspondence between each data table and field in the source database to splice the table creation statement. The source data structure of each data table here includes the correspondence between the data table and the field. If it is detected that the user chooses to add a table prefix or chooses to give a special name to the table when creating a task, it will also be spliced ​​into the table creation statement. When the splicing is completed, the table creation statement can be run to create the target data structure of each data table in the target database.

[0086] Involving S305: connecting to the target database through the target connector according to the connection information in the target parameters, and creating respective target data structures of the respective data tables in the target database.

[0087] Exemplarily, after the splicing is completed, the target database is connected through the target connector Connector3 according to the connection information (target IP and target port) in the target parameters of the target database, and the target data structure of each data table in the target database is created.

[0088] This involves S306: creating a target association table in the target connector according to the target data structure.

[0089] The target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database.

[0090] This involves S307: if data changes in the source association table are obtained, the changed data is inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0091] The source association table and the target association table are used as the link between the logical table and the physical table. The changes of the data table of the source database are dynamically obtained according to the data flow, and the addition, update and deletion actions of the data table in the source database can be obtained. With this design, when the data in the source association table changes, the changed data can be inserted into the target association table, so that the changed data can be synchronized to the target physical database through the target association table, thereby realizing the whole database synchronization from the source database to the target database.

[0092] In an optional example, the table creation statement can be a structured query language (SQL) statement, which concatenates the SQL statement for querying the data table in the source database and the SQL statement for inserting into the target database. The SQL statement is executed in the associated table instead of concatenating the target database.

[0093] Figure 5 A schematic diagram of a database synchronization process provided in an embodiment of the present application, through Figure 5 It can be seen that the whole database synchronization process is realized by relying on the source association table and the target association table.

[0094] The embodiments of the present application can realize data synchronization of the entire database of multiple relational databases, and can perceive data changes in real time to ensure the freshness of the data.

[0095] Figure 6A schematic diagram of a business process provided by an embodiment of the present application, wherein the task information may include information such as database parameters related to the data access task.

[0096] Figure 7 A system architecture diagram of a database access for a whole database provided in an embodiment of the present application. Among them, the data type of database 1 is Mysql, the data type of database 2 is oracle, and the data type of database 3 is postgressql. Among them, the task information may include information such as database parameters related to the data access task.

[0097] Combination Figure 6 and Figure 7 The data synchronization process in the embodiment of the present application is a whole database synchronization, so it only occupies one database connection, which can greatly reduce the number of database connections and memory usage. It is helpful to uniformly manage, govern and share data between different systems of different units or enterprises, breaking down data silos.

[0098] In order to make the technical solution of the embodiment of the present application more perfect, a complete flowchart is used below for explanation: Figure 8 A flowchart of a complete method for synchronizing an entire database provided in an embodiment of the present application.

[0099] S800: In response to a task construction request from a user, configure source parameters of at least one source database and target parameters of at least one target database.

[0100] The implementation method of this step may refer to step S401 and will not be described in detail here.

[0101] S801: In response to the user's operation on the task construction page, determining the source database and target database selected by the user.

[0102] The implementation method of this step may refer to step S402 and will not be described in detail here.

[0103] S802: Constructing a data connection task according to the source database and the target database.

[0104] The implementation method of this step may refer to step S403 and will not be described in detail here.

[0105] S803: In response to the task start operation of the user, the source parameters of the source database and the target parameters of the target database are obtained through the running engine of the data connection task.

[0106] The implementation method of this step may refer to step S302 and will not be described in detail here.

[0107] S804: According to the connection information in the source parameters, connect to the source database through the source connector and obtain the source data structure of each data table in the source database.

[0108] The implementation method of this step may refer to step S303 and will not be described in detail here.

[0109] S805: Create a source association table in the source connector according to the source data structure.

[0110] The implementation method of this step may refer to step S304 and will not be described in detail here.

[0111] S806: Apply the corresponding relationship between each data table and field in the source database to splice the table creation statement.

[0112] The implementation method of this step can refer to the relevant embodiments and will not be described here in detail.

[0113] S807: Run the table creation statement.

[0114] The implementation method of this step can refer to the relevant embodiments and will not be described here in detail.

[0115] S808: Connect to the target database through the target connector according to the connection information in the target parameters, and create target data structures for each data table in the target database.

[0116] The implementation method of this step may refer to step S305 and will not be described in detail here.

[0117] S809: Create a target association table in the target connector according to the target data structure.

[0118] The implementation method of this step may refer to step S306 and will not be described in detail here.

[0119] S810: If data changes in the source association table are obtained, the changed data is inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0120] The implementation method of this step may refer to step S307 and will not be described in detail here.

[0121] Compared with traditional data access, the embodiment of the present application realizes the access of the whole database data. The existing data access in the industry currently creates access tasks based on tables as the dimension, and requires the table of the target database to already exist, resulting in the creation of access tasks. It is necessary to manually create the target table, manually create access tasks equal to the number of tables, and establish field mappings between tables. These steps are huge. In the embodiment of the present application, there is no need to create a target table, only the database name is required, and only one task needs to be created to complete the acquisition of the table structure of the entire database and automatically create the target table. Since there is only one task, it only occupies one database connection, which can greatly reduce the pressure on the database.

[0122] like Fig. 9 As shown, based on the same inventive concept, an embodiment of the present application provides a device for synchronizing an entire database, including a task construction unit 91, a parameter acquisition unit 92, a first connection unit 93, a source association table creation unit 94, a second connection unit 95, a target association table creation unit 96 and a synchronization unit 97.

[0123] The task construction unit 91 is used to: construct a data connection task;

[0124] The parameter acquisition unit 92 is used to: in response to the task start operation of the user, acquire the source parameters of the source database and the target parameters of the target database corresponding to the data connection task;

[0125] The first connection unit 93 is used to: connect to the source database through the source connector according to the connection information in the source parameter and obtain the source data structure of each data table in the source database;

[0126] The source association table creation unit 94 is used to: create a source association table in the source connector according to the source data structure; wherein the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database;

[0127] The second connection unit 95 is used to: connect to the target database through the target connector according to the connection information in the target parameter, and create a target data structure for each data table in the target database;

[0128] The target association table creation unit 96 is used to: create a target association table in the target connector according to the target data structure; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database;

[0129] The synchronization unit 97 is used to: if data changes in the source association table are obtained, insert the changed data into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0130] In an optional embodiment, the device further includes a database selection unit, which is used to determine the source database and the target database selected by the user in response to the user's operation on the task construction page before constructing the data connection task;

[0131] The task construction unit 91 is specifically used for:

[0132] Build a data connection task based on the source database and target database.

[0133] In an optional embodiment, the device further includes a configuration unit, which is used to configure source parameters of at least one source database and target parameters of at least one target database in response to a task construction request of a user before responding to an operation of the user on the task construction page;

[0134] The source database is any one of at least one pre-configured source database, and the target database is any one of at least one pre-configured target database.

[0135] In an optional implementation, the configuration unit is further configured to:

[0136] Configure task running parameters; wherein the task running parameters include the type of the task running, which is periodic or real-time;

[0137] The task construction unit 91 is specifically used for:

[0138] Build a data connection task based on the source database and target database and the task operation type.

[0139] In an optional implementation, the parameter acquisition unit 92:

[0140] The source parameters of the source database and the target parameters of the target database are obtained through the running engine of the data connection task; wherein the running engine of the task is determined according to the running type of the task.

[0141] In an optional implementation, the system further includes a connector determination unit, configured to determine a source connector in the following manner:

[0142] According to the source database type in the source parameter, determine the connector corresponding to the source database type as the source connector.

[0143] In an optional embodiment, it further includes a target data structure creation unit, which is used to apply the correspondence between each data table and the field in the source database to splice the table creation statement before connecting the target database through the target connector according to the connection information in the target parameter and creating the target data structure of each data table in the target database; wherein the source data structure of each data table includes the correspondence between the data table and the field;

[0144] Run the create table statement.

[0145] In an optional implementation, if the data connection task is a periodic task, the configuration unit is further used to configure the running frequency of the periodic task;

[0146] If the data connection task is a real-time task, the method further includes:

[0147] Execute data connection tasks through stream processing.

[0148] In an optional implementation, the source parameters include source connection information, source database name, source database type, source user name, and source password;

[0149] Target parameters include target connection information, target database name, target database type, target user name, and target password.

[0150] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0151] like Fig.10 As shown, based on the same inventive concept, an embodiment of the present application provides an electronic device, including a task construction unit 101 and a processor 102:

[0152] The task construction unit 101 is configured to execute:

[0153] Build data connection tasks;

[0154] The processor 102 is configured to execute:

[0155] In response to a task start operation by a user, a source parameter of a source database and a target parameter of a target database corresponding to the data connection task are obtained;

[0156] According to the connection information in the source parameters, connect to the source database through the source connector and obtain the source data structure of each data table in the source database;

[0157] According to the source data structure, a source association table is created in the source connector; wherein the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database;

[0158] According to the connection information in the target parameters, connect to the target database through the target connector, and create the target data structure of each data table in the target database;

[0159] According to the target data structure, a target association table is created in the target connector; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database;

[0160] If data changes in the source association table are obtained, the changed data is inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

[0161] In an optional implementation, the processor 102 is further configured to: before constructing the data connection task, in response to the user's operation on the task construction page, determine the source database and the target database selected by the user;

[0162] In an optional implementation, the processor 102 is specifically configured to:

[0163] Build a data connection task based on the source database and target database.

[0164] In an optional implementation, before responding to the user's operation on the task building page, the processor 102 is further configured to:

[0165] In response to a task building request from a user, configuring a source parameter of at least one source database and a target parameter of at least one target database;

[0166] The source database is any one of at least one pre-configured source database, and the target database is any one of at least one pre-configured target database.

[0167] In an optional implementation, the processor 102 is further configured to:

[0168] Configure task running parameters; wherein the task running parameters include the type of the task running, which is periodic or real-time;

[0169] Build data connection tasks based on the source database and target database, including:

[0170] Build a data connection task based on the source database and target database and the task operation type.

[0171] In an optional implementation, the processor 102 is specifically configured to:

[0172] The source parameters of the source database and the target parameters of the target database are obtained through the running engine of the data connection task; wherein the running engine of the task is determined according to the running type of the task.

[0173] In an optional implementation, the processor 102 is further configured to determine the source connector in the following manner:

[0174] According to the source database type in the source parameter, determine the connector corresponding to the source database type as the source connector.

[0175] In an optional implementation, before connecting to the target database through the target connector according to the connection information in the target parameter and creating the target data structure of each data table in the target database, the processor 102 is further configured to:

[0176] Apply the correspondence between each data table and the fields in the source database to concatenate table creation statements; wherein the source data structure of each data table includes the correspondence between the data table and the fields;

[0177] Run the create table statement.

[0178] In an optional implementation, if the data connection task is a periodic task, the processor 102 is further configured to:

[0179] Configure the running frequency of periodic tasks;

[0180] If the data connection task is a real-time task, the method further includes:

[0181] Execute data connection tasks through stream processing.

[0182] In an optional implementation, the source parameters include source connection information, source database name, source database type, source user name, and source password;

[0183] Target parameters include target connection information, target database name, target database type, target user name, and target password.

[0184] The embodiment of the present application also provides a computer storage medium, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned database whole-database synchronization method.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for synchronizing an entire database, characterized in that: include: Build data connection tasks; In response to a task start operation by a user, a source parameter of a source database and a target parameter of a target database corresponding to the data connection task are acquired; According to the connection information in the source parameters, connect to the source database through a source connector and obtain the source data structure of each data table in the source database; According to the source data structure, a source association table is created in the source connector; wherein the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database; According to the connection information in the target parameters, the target database is connected through the target connector, and the target data structure of each data table in the target database is created; According to the target data structure, a target association table is created in the target connector; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database; If data changes in the source association table are obtained, the changed data are inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

2. The method according to claim 1, characterized in that Before the data connection task is constructed, the method further includes: In response to the user's operation on the task building page, determining the source database and the target database selected by the user; The task of building a data connection includes: A data connection task is constructed according to the source database and the target database.

3. The method according to claim 2, characterized in that Before responding to the user's operation on the task construction page, the method further includes: In response to a task building request from a user, configuring a source parameter of at least one source database and a target parameter of at least one target database; The source database is any one of at least one pre-configured source database, and the target database is any one of at least one pre-configured target database.

4. The method according to claim 2, characterized in that: The method further comprises: Configure task running parameters; wherein the task running parameters include the type of operation of the task, which is periodic or real-time; The task of constructing a data connection task according to the source database and the target database includes: Based on the source database and the target database, and in accordance with the running type of the task, a data connection task is constructed.

5. The method according to claim 1, characterized in that The obtaining of source parameters of the source database and target parameters of the target database corresponding to the data connection task includes: The source parameters of the source database and the target parameters of the target database are obtained through the running engine of the data connection task; wherein the running engine of the task is determined according to the running type of the task.

6. The method according to claim 1, characterized in that The source connector is determined as follows: According to the source database type in the source parameter, a connector corresponding to the type of the source database is determined as a source connector.

7. The method according to claim 1, characterized in that Before connecting the target database through the target connector according to the connection information in the target parameter and creating the target data structure of each data table in the target database, the method further includes: Apply the correspondence between each data table and the fields in the source database to concatenate table creation statements; wherein the source data structure of each data table includes the correspondence between the data table and the fields; Run the table creation statement.

8. The method according to claim 1, characterized in that If the data connection task is a periodic task, the method further includes: Configure the running frequency of the periodic task; If the data connection task is a real-time task, the method further includes: The data connection task is executed by stream processing.

9. The method according to any one of claims 1 to 8, characterized in that: The source parameters include source connection information, source database name, source database type, source user name and source password; The target parameters include target connection information, target database name, target database type, target user name and target password.

10. An electronic device, characterized in that: Includes task building blocks and processors: The task construction unit is configured to execute: Build data connection tasks; The processor is configured to execute: In response to a task start operation by a user, a source parameter of a source database and a target parameter of a target database corresponding to the data connection task are acquired; According to the connection information in the source parameters, connect to the source database through a source connector and obtain the source data structure of each data table in the source database; According to the source data structure, a source association table is created in the source connector; wherein the source association table represents the relationship between each data table in the source database and each data table in the source physical library corresponding to the source database; According to the connection information in the target parameters, the target database is connected through the target connector, and the target data structure of each data table in the target database is created; According to the target data structure, a target association table is created in the target connector; wherein the target association table represents the relationship between each data table in the target database and each data table in the target physical library corresponding to the target database; If data changes in the source association table are obtained, the changed data are inserted into the target association table, so as to synchronize the changed data to the target physical library through the target association table.

Images