Data synchronization method, device, computer equipment and storage medium

Abstract

The present application relates to the fields of financial technology, big data, and data processing technology, and discloses a data synchronization method, apparatus, computer equipment, and storage medium, the method comprising: obtaining address information of a first target database; obtaining a data information group of the first target database according to the address information, wherein the data information group comprises at least one data information, and the data information is all information of a data table in the first target database; generating a flink‑sql statement set according to the data information group, and generating a target flink cluster based on a preset cluster creation statement; based on the target flink cluster and the flink‑sql statement set, synchronizing and storing the data information of each data table in the first target database to a second target database, thereby improving the intelligence level and synchronization efficiency of data synchronization.

Description

Technical Field

[0001] The present invention relates to the fields of big data, data processing, data synchronization, medical health and financial technology, and in particular to a data synchronization method, device, computer equipment and storage medium. Background Art

[0002] With the continuous development of big data technology, data analysis has become increasingly important. More and more companies, such as banks and hospitals, have medical and financial data that is not only diverse in type and complex in relationship, but also explosively growing. It is necessary to synchronize the data in traditional data storage systems to the data lake for storage and analysis. In order to meet the needs of analytical business, it is necessary to maintain data consistency and improve the efficiency of data updates. For example, in the financial order business, it is necessary to synchronize the financial order data in the traditional data storage system to the data lake in real time for storage and analysis, and update the financial order data in the data lake in real time.

[0003] Traditional data synchronization solutions first perform full query-based synchronization to write all the data in the relational database into the data lake at one time, and then start a resident synchronization program to read and analyze the database logs to write incremental data into the data lake in real time. In this process, professional personnel are required to do a lot of development and configuration work, which not only wastes a lot of time and manpower, but is also limited by the relevant knowledge reserves and professional level of professionals, resulting in low efficiency and intelligence of data synchronization. Summary of the invention

[0004] Based on this, it is necessary to propose a data synchronization method, device, computer equipment and storage medium to address the technical problem that when taking images, the existing technology is easily affected by factors such as light, shooting angle, shooting distance in the shooting environment, resulting in low image quality and low accuracy and reliability of information in the image.

[0005] In a first aspect, a data synchronization method is provided, the method comprising:

[0006] Acquire address information of a first target database;

[0007] According to the address information, acquiring a data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database;

[0008] Generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of the flink-sql statements corresponds to one piece of data information;

[0009] Generate the target Flink cluster based on the preset cluster creation statement;

[0010] Based on the target flink cluster and the flink-sql statement set, data information of each data table in the first target database is synchronously stored in the second target database.

[0011] In a second aspect, a data synchronization device is provided, the device comprising:

[0012] A first information acquisition module, used to acquire address information of a first target database;

[0013] A second information acquisition module, configured to acquire a data information group of the first target database according to the address information, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database;

[0014] A statement set generation module, used to generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of the flink-sql statements corresponds to one piece of data information;

[0015] The cluster creation module generates a target Flink cluster using a preset cluster creation statement.

[0016] A data synchronization module is used to synchronize and store data information of each data table in the first target database to a second target database based on the target flink cluster and the flink-sql statement set.

[0017] According to a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data synchronization method when executing the computer program.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data synchronization method are implemented.

[0019] In the present application, a data synchronization method, apparatus, computer equipment and storage medium are provided, which obtain the address information of a first target database; obtain the data information group of the first target database according to the address information, wherein the data information group includes at least one data information, and the data information is all the information of a data table in the first target database; generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information; generate a target flink cluster based on a preset cluster creation statement; based on the target flink cluster and the flink-sql statement set, synchronize and store the data information of each data table in the first target database to the second target database, in the present application In the request, after automatically identifying the first target database, the data information of each data table in the first target database is obtained, and a flink-sql statement set is automatically generated according to the data information group, and a target flink cluster is generated according to the preset cluster creation statement. Finally, based on the automatically generated target flink cluster and flink-sql statement set, the data information of each data table in the first target database is automatically synchronized to the second target database. Due to the use of an automated data synchronization method, a large number of professionals are not required to perform a large amount of development and configuration work, and are not restricted by the relevant knowledge reserves and professional levels of professionals, which reduces development time and labor costs. It not only improves the intelligence level of data synchronization, but also improves the efficiency of real-time synchronization of online data to the data lake or physical library. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] in:

[0022] Figure 1 An application environment diagram of a data synchronization method in an embodiment;

[0023] Figure 2 A schematic diagram of a flow chart of a data synchronization method in an embodiment;

[0024] Figure 3 is a schematic block diagram of the structure of a data synchronization device in one embodiment;

[0025] Figure 4 is a schematic block diagram of the structure of a computer device in one embodiment;

[0026] Figure 5 The figure is a schematic block diagram of the structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The data synchronization method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client 110 communicates with the server 120 through a network. The server 120 can obtain the address information of the first target database through the client 110; according to the address information, obtain the data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all the information of a data table in the first target database; according to the data information group, generate a flink-sql statement set, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information; based on a preset cluster creation statement, generate a target flink cluster; based on the target flink cluster and the flink-sql statement set, synchronize the data information of each data table in the first target database to the second target database, and display it on the client 110. In this application , after automatically identifying the first target database, the data information of each data table in the first target database is obtained, and a flink-sql statement set is automatically generated according to the data information group, and a target flink cluster is generated according to a preset cluster creation statement. Finally, based on the automatically generated target flink cluster and flink-sql statement set, the data information of each data table in the first target database is automatically synchronized to the second target database. Due to the use of an automated data synchronization method, it is avoided that a large number of professionals are required to perform a large amount of development and configuration work, and it is avoided to be limited by the relevant knowledge reserves and professional level of professionals, which reduces development time and labor costs, not only improves the intelligence of data synchronization, but also improves the efficiency of real-time synchronization of online data to the data lake or physical library. Among them, the client 110 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server 120 can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0029] See also Figure 2 As shown, Figure 2 This is a flow chart of a data synchronization method provided by an embodiment of the present invention. The method can be applied to both a terminal and a server. This embodiment is described by applying to a server as an example. The data synchronization method includes the following steps S100 to S500:

[0030] S100: Acquire address information of a first target database.

[0031] The first target database may be a relational database.

[0032] Among them, the address information can be a server name or a server IP address of a relational database. The address information can be an address received by the server after the user inputs it on the client, or it can be pre-stored in the server and the server obtains the address by querying according to a preset address query rule. For example, the preset address query rule includes multiple addresses and an address query instruction associated with each address. When the server receives the address query instruction, it determines the address corresponding to the address query instruction by querying the address query instruction.

[0033] S200: Acquire a data information group of the first target database according to the address information, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database.

[0034] Among them, all information of a data table in the first target database may include field information, number of fields, total number of data items, name of the data table, metadata database address, and real storage address of the data table. The field information may include field name (name), field address (id), field age (age), and data types (Type) corresponding to the field name (name), field address (id), and field age (age), respectively. The total number of data items is the number of rows in the data table, which may include existing data and incremental data.

[0035] Among them, you can use sql ( The row count query function of the structured query language (SQL) statement queries the total number of data entries in the data table to obtain the total number of data entries in the data table. For example, the SQL statement may be seclectcount(*)from table_1, which queries the total number of data entries in the data table table_1 to obtain the total number of data entries in the data table table_1.

[0036] Among them, the field information and the number of fields in the data table can be queried based on the field query function of the SQL statement to obtain the field information and the number of fields in the data table. For example, the SQL statement can be select * from table_1, that is, the field information and the number of fields in the data table table_1 are queried to obtain the field information and the number of fields in the data table table_1.

[0037] Optionally, the stock data and incremental data in the total amount of data may be determined based on the operation log of the first target database.

[0038] S300: Generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information.

[0039] Among them, flink-sql is a query language of Apache Flink, which allows users to use SQL statements to query streaming data or batch data. Apache Flink is a stream processing framework, and its core is a distributed streaming data flow engine written in Java and Scala, where Java and Scala are computer programming languages.

[0040] Among them, a group of flink-sql statements includes flink-sql statements for reading data and creating tables, flink-sql statements for writing data and insert-type flink-sql statements. Among them, the flink-sql statements for reading data and creating tables are used to define the data information of each data table in the first target database (such as field information, metadata database address, real storage address, etc.), the flink-sql statements for writing data and creating tables are used to define the data information of each data table in the first target database output to the data information of the adapted metadata table in the second target database (such as metadata database address, real storage address, field information, etc.), and the insert-type flink-sql statements define the specific algorithm and window rules for outputting each data table in the first target database to the adapted metadata table in the second target database.

[0041] S400: Generate a target Flink cluster based on the preset cluster creation statement.

[0042] When the preset cluster creation statement is executed, it is used to place the running files of the flink cluster in the preset directory of the Hadoop cluster (such as / user / flink), thereby generating a target flink cluster. The Hadoop cluster is pre-installed on a server, which can be a Linnux type server.

[0043] For example, the preset cluster creation statement may be:

[0044] yarn-session.sh-jm 2g-tm 2g-Dparallelism.default=1-qu root.default-nm lqc_002-d, that is, placing the running files of the flink cluster in the preset directory of the Hadoop cluster (such as / user / flink), thereby generating the target flink cluster. Among them, the Hadoop cluster is a distributed system composed of multiple servers, and multiple servers work together to store and process large-scale data sets.

[0045] S500: Based on the target flink cluster and the flink-sql statement set, the data information of each data table in the first target database is synchronously stored in the second target database.

[0046] The second target database can be a data lake or a physical database.

[0047] Among them, the flink-sql statement set is submitted to the target flink cluster, the data information of each data table in the first target database is synchronously stored in the second target database, and the preset task submission command statement can be a sql statement.

[0048] In this embodiment, by obtaining the address information of the first target database; according to the address information, obtaining the data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all the information of a data table in the first target database; according to the data information group, generating a flink-sql statement set, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information; based on the preset cluster creation statement, generating a target flink cluster; based on the target flink cluster and the flink-sql statement set, the data information of each data table in the first target database is synchronously stored in the second target database. In this application, by After automatic identification of the database, the data information of each data table in the first target database is obtained, and a flink-sql statement set is automatically generated according to the data information group, and a target flink cluster is generated according to the preset cluster creation statement. Finally, based on the automatically generated target flink cluster and flink-sql statement set, the data information of each data table in the first target database is automatically synchronized to the second target database. Due to the use of an automated data synchronization method, it avoids the need for a large number of professionals to perform a large amount of development and configuration work, and avoids being restricted by the relevant knowledge reserves and professional levels of professionals, reducing development time and labor costs, which not only improves the intelligence level of data synchronization, but also improves the efficiency of real-time synchronization of online data to the data lake or physical database.

[0049] In some optional implementations of this embodiment, step S200, obtaining a data information group of the first target database according to the address information, includes the following steps S210 to S240:

[0050] S210: Based on a preset query statement, query the first target database corresponding to the address information to obtain data structure information of each data table in the first target database.

[0051] Among them, the preset query statement can be a select query statement, which is used to retrieve data tables that meet preset parameter conditions (such as the IP, port, user name, password, tables that do not need to be synchronized (empty by default) of the first target database, etc.) from the first target database, thereby obtaining the data structure information of each data table.

[0052] The data structure information includes the structure information of the data table, and the structure information of the data table may include field name, field type, whether NULL value can be stored, whether index has been compiled, whether there is a default value, and other information.

[0053] S220: Based on each piece of data structure information, perform data query on the data table corresponding to the data structure information to obtain the total number of data items and the number of fields in each data table in the first target database.

[0054] Optionally, based on the field query statement of the SQL statement, data query can be performed on each column in the data table corresponding to the data structure information according to the structure information of the data table in the data structure information, so as to obtain the field information and field quantity of each data table in the first target database.

[0055] Exemplarily, the SQL data query statement may be:

[0056] Select column_name from test_user_to, that is, obtain the field information and number of fields in the data table test_user_to from the data table test_user_to.

[0057] S230: The data structure information, the total number of data items and the number of fields of the same data table are used as the data information of the data table.

[0058] S240: All data information is used as a data information group of the first target database.

[0059] In this embodiment, by querying each data table in the first target database, the data structure information, the total number of data items, and the number of fields of each data table to be synchronized in the first target database are determined, which is conducive to the subsequent automatic resource data configuration of the target flink cluster to ensure the reliability of the data synchronization process, improve the intelligence of data synchronization, and improve the efficiency of real-time synchronization of online data to the data lake or physical library.

[0060] In some optional implementations of this embodiment, step S300, generating a flink-sql statement set according to the data information group, includes the following steps S310 to S320:

[0061] S310: Obtain data synchronization configuration information for each data table in the second target database, wherein the data synchronization configuration information includes: account number, password, metadata database address, and data storage address.

[0062] Among them, the account and password are used to log in to the server where the second target data is located, the metadata database address is used to locate the address of the target metadata database when writing the data table in the first target database to the second target database, and the data storage address is used to locate the data storage address in the target metadata database when writing the data table in the first target database to the second target database.

[0063] S320: Generate a flink-sql statement set according to a preset code template, data synchronization configuration information, and data information group.

[0064] Among them, the data synchronization configuration information of each data table and the data information associated with the data synchronization configuration information in the data information group are automatically added to the corresponding mark position in the preset code model in turn to generate a flink-sql statement set.

[0065] Optionally, the preset code template can be based on a template generated in Java language, which is used to generate a set of flink-sql statements corresponding to each data table; and multiple sets of flink-sql statements are used as flink-sql statement sets.

[0066] In this embodiment, a flink-sql statement set is automatically generated based on data synchronization configuration information, data information groups, and preset code templates, which is beneficial to improving the intelligence of data synchronization and the efficiency of real-time synchronization of online data to a data lake or a physical library.

[0067] In some optional implementations of this embodiment, before step S400, that is, before generating a target flink cluster based on a preset cluster creation statement, the data synchronization method further includes the following steps S600 to S610:

[0068] S600: Obtain cluster configuration information, which includes server login configuration information, Flink cluster running files, and running file placement paths.

[0069] The target server where the Hadoop cluster is installed is queried through the server login configuration information (such as server account, password, address, etc.), and the flink cluster running file is placed according to the running file placement path.

[0070] S610: Obtain a preset cluster creation statement according to cluster configuration information.

[0071] Among them, when the flink cluster running file is detected in the directory corresponding to the running file placement path according to the cluster configuration information, the preset cluster creation statement is obtained.

[0072] In this embodiment, cluster configuration information is automatically obtained, and a preset cluster creation statement is obtained according to the cluster configuration information, thereby improving the intelligence of data synchronization, improving the efficiency of real-time synchronization of online data to a data lake or a physical library, avoiding being restricted by the knowledge reserves and professional level of professionals, and lowering the threshold for use.

[0073] In some optional implementations of this embodiment, after step S400, that is, after the target flink cluster is generated based on the preset cluster creation statement, the data synchronization method further includes the following steps S700 to S710:

[0074] S700: Generate target resource configuration parameters of the target Flink cluster according to the total number of data entries and the number of fields in each data table.

[0075] Among them, the target resource configuration parameters can be the parallelism of each data table, that is, the parallelism of the data synchronization task corresponding to a set of flink-sql statements, the number of jobmanager processors, the storage of each jobmanager processor, the number of virtual machines, and the storage of each virtual machine.

[0076] Among them, the jobmanager processors are used to coordinate distributed execution. They are used to schedule data synchronization tasks in the flink cluster, coordinate checkpoints, and coordinate recovery when failure occurs.

[0077] Optionally, the parallelism p of each data table is calculated according to the following formula (1):

[0078] p=t*f / F

[0079] In the formula, p is the degree of parallelism, t is the total number of data in the data table, f is the number of fields in the data table, and F is the parallelism coefficient.

[0080] The parallelism coefficient determines the number of virtual machines working in the target Flink cluster. The memory and parallelism coefficient of each virtual machine can be dynamically set according to the actual application scenario. No specific restrictions are made here. For example, the parallelism coefficient can be 100 million and the memory of the virtual machine can be 6G.

[0081] S710: Update preset resource configuration parameters of the target flink cluster according to the target resource configuration parameters.

[0082] Among them, the number of jobmanager processors in the target flink cluster and the memory of each jobmanager processor can be automatically and dynamically allocated according to the calculated target resource configuration parameters.

[0083] Optionally, the preset resource configuration parameters of the target flink cluster can be updated on the preset configuration template according to the target resource configuration parameters, which is helpful for users to view the process of data synchronization tasks in real time and find data. The preset configuration template is a user interaction management page based on programming, which can be set according to the actual application scenario and is not specifically limited here.

[0084] In the prior art, the resource configuration parameters of the flink cluster are usually estimated based on the experience of professionals, resulting in inaccurate setting of the resource configuration parameters of the flink cluster, thereby reducing the resource utilization of the flink cluster. In this embodiment, the target resource configuration parameters of the target flink cluster are calculated according to the total number of data entries and the number of fields in each data table to update the preset resource configuration parameters of the target flink cluster. The real database of each data table is used to automatically and accurately calculate the required configuration resources for the target flink cluster, thereby improving the resource utilization of the target flink cluster and reducing the hardware cost during data synchronization.

[0085] In some optional implementations of this embodiment, step S500, based on the target flink cluster and the flink-sql statement set, synchronously stores the data information of each data table in the first target database to the second target database, includes the following steps S510 to S520:

[0086] S510: In response to a preset cluster startup command, each group of flink-sql statements in the flink-sql statement set is submitted to the target flink cluster in sequence.

[0087] Among them, the preset cluster startup command is used to start the target flink cluster, and submit each group of flink-sql statements in the flink-sql statement set to the target flink cluster in sequence.

[0088] Exemplarily, the preset cluster startup command may be:

[0089] . / bin / sql-client.sh, that is, start the target flink cluster and submit each group of flink-sql statements in the flink-sql statement set to the target flink cluster in sequence. Each group of flink-sql statements represents a data synchronization task.

[0090] S520: Execute each group of flink-sql statements respectively through the target flink cluster to synchronously store data information of each data table in the first target database to the second target database.

[0091] Among them, by parsing each group of flink-sql statements through the target flink cluster, data information of the data table in the first target database and data information of the adapted metadata table in the second target database corresponding to the group of flink-sql statements can be obtained, so as to synchronize the data information of each data table in the first target database and store it in the second target database.

[0092] In this embodiment, by automatically starting the target flink cluster, each group of flink-sql statements in the flink-sql statement set is submitted to the target flink cluster in sequence, and each group of flink-sql statements is executed by the target flink cluster respectively, so as to synchronize and store the data information of each data table in the first target database to the second target database, which is beneficial to improve the degree of intelligence in the data synchronization process and improve the efficiency of real-time synchronization of online data to the data lake or physical library.

[0093] In some optional implementations of this embodiment, during the execution of step S500, that is, during the execution of the target flink cluster and the flink-sql statement set to synchronously store the target data of the first target database in the second target database, the method further includes the following steps S530 to S550:

[0094] S530: Monitor whether there are any exceptions in the synchronization storage process through a preset exception monitoring mechanism, wherein the synchronization storage process is to synchronously store the target data of the first target database to the second target database based on the target flink cluster and the flink-sql statement set.

[0095] Among them, you can set up an application programming interface in the target flink cluster to generate a preset exception monitoring mechanism, and monitor whether there are exceptions in the synchronization storage process by accessing the application programming interface.

[0096] Among them, the anomaly may be caused by network anomaly, hard disk anomaly, or surge in data volume.

[0097] For example, when a data synchronization task is in the running state, each application programming interface in the target flink cluster is monitored to detect whether the running state of the data synchronization task is normal.

[0098] S540: If an exception is detected in the synchronous storage process, the type of the exception is determined based on a preset set of exception types.

[0099] The preset exception type set may be the types of exceptions that have occurred during the synchronization process and the status information corresponding to the types obtained by analyzing historical experience data.

[0100] The type of the exception is determined by comparing the status information corresponding to the exception existing in the synchronous storage process with the status information in the preset exception type set.

[0101] S550: Process the exception according to the type of the exception.

[0102] According to the type of the exception, it is determined whether the exception can be automatically recovered by the target flink cluster. If the exception cannot be automatically recovered by the target flink cluster, the exception alarm information is generated according to the type of the exception and the status information corresponding to the type of the exception.

[0103] Among them, the abnormal alarm information can be text or voice, and can be sent to the designated client via phone, text message, or email.

[0104] In this embodiment, the target flink cluster monitors the exceptions in the synchronization storage process, automatically recovers the exceptions that can be automatically recovered, and issues alarms for the exceptions that cannot be automatically recovered, which is beneficial to improving the intelligence and reliability of the data synchronization process, and also improves the efficiency of real-time synchronization of online data to the data lake or physical library.

[0105] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0106] In one embodiment, a data synchronization device is provided, which corresponds one-to-one to the data synchronization method in the above embodiment. Figure 3 As shown, the data synchronization device includes a first information acquisition module 10, a second information acquisition module 20, a statement set generation module 30, a cluster creation module 40 and a data synchronization module 50. The functional modules are described in detail as follows:

[0107] The first information acquisition module 10 is used to acquire address information of the first target database.

[0108] The second information acquisition module 20 is used to acquire a data information group of the first target database according to the address information, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database.

[0109] The statement set generation module 30 is used to generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information.

[0110] The cluster creation module 40 generates a target Flink cluster using a preset cluster creation statement.

[0111] The data synchronization module 50 is used to synchronize and store the data information of each data table in the first target database to the second target database based on the target flink cluster and the flink-sql statement set.

[0112] In this embodiment, by obtaining the address information of the first target database; according to the address information, obtaining the data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all the information of a data table in the first target database; according to the data information group, generating a flink-sql statement set, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information; based on the preset cluster creation statement, generating a target flink cluster; based on the target flink cluster and the flink-sql statement set, synchronously storing the data information of each data table in the first target database to the second target database, in this application, after automatically identifying the first target database, the first target database is obtained. The data information of each data table in the first target database is automatically generated, and a flink-sql statement set is automatically generated according to the data information group, and a target flink cluster is generated according to the preset cluster creation statement. Finally, based on the automatically generated target flink cluster and flink-sql statement set, the data information of each data table in the first target database is automatically synchronized to the second target database. Since each of the above steps adopts an automated method to synchronize the data between the first target database and the second target database, it avoids the need for a large number of professionals to perform a large amount of development and configuration work each time data synchronization is performed, and avoids being restricted by the relevant knowledge reserves and professional levels of professionals, reducing development time and labor costs, which not only improves the intelligence level of data synchronization, but also improves the efficiency of real-time synchronization of online data to the data lake or physical library.

[0113] Optionally, the first information acquisition module 10 acquires the data information group of the first target database according to the address information, including:

[0114] The first query submodule is used to query the first target database corresponding to the address information based on a preset query statement to obtain data structure information of each data table in the first target database.

[0115] The second query submodule is used to perform data query on the data table corresponding to the data structure information based on each piece of data structure information, and obtain the total number of data items and the number of fields in each data table in the first target database.

[0116] The first information determination submodule is used to use the data structure information, the total number of data items and the number of fields of the same data table as the data information of the data table.

[0117] The second information determination submodule is used to take all data information as the data information group of the first target database.

[0118] Optionally, the statement set generation module 30 generates a flink-sql statement set according to the data information group, including:

[0119] The synchronization configuration information acquisition submodule is used to obtain the data synchronization configuration information of each data table in the second target database, wherein the data synchronization configuration information includes: account number, password, metadata database address, and data storage address.

[0120] The statement set generation submodule is used to generate a flink-sql statement set based on the preset code template, data synchronization configuration information, and data information group.

[0121] Optionally, before generating a target flink cluster based on a preset cluster creation statement in the cluster creation module 40, the following steps are included:

[0122] The cluster configuration information acquisition module is used to obtain cluster configuration information, including server login configuration information, Flink cluster running files, and running file placement paths.

[0123] The cluster creation statement generation module is used to obtain a preset cluster creation statement according to cluster configuration information.

[0124] Optionally, after the target flink cluster is generated based on the preset cluster creation statement in the cluster creation module 40, the method further includes:

[0125] The resource configuration parameter acquisition module is used to generate the target resource configuration parameters of the target Flink cluster according to the total number of data entries and the number of fields in each data table.

[0126] The resource configuration parameter update module is used to update the preset resource configuration parameters of the target Flink cluster according to the target resource configuration parameters.

[0127] Optionally, the data synchronization module 50 synchronizes and stores data information of each data table in the first target database to the second target database based on the target flink cluster and the flink-sql statement set, including:

[0128] The statement submission submodule is used to submit each group of flink-sql statements in the flink-sql statement set to the target flink cluster in sequence in response to the preset cluster startup command.

[0129] The data synchronization submodule is used to execute each group of flink-sql statements through the target flink cluster to synchronize and store the data information of each data table in the first target database to the second target database.

[0130] Optionally, in the process where the data synchronization module 50 executes the target flink cluster and the flink-sql statement set to synchronously store the target data of the first target database in the second target database, the data synchronization device further includes:

[0131] The monitoring module is used to monitor whether there are any exceptions in the synchronization storage process through a preset exception monitoring mechanism, wherein the synchronization storage process is to synchronize and store the target data of the first target database to the second target database based on the target flink cluster and flink-sql statement set.

[0132] The exception type determination module is used to determine the type of exception based on a preset exception type set if an exception is detected in the synchronous storage process.

[0133] The exception handling module is used to handle exceptions according to their types.

[0134] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data synchronization method.

[0135] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a data synchronization method.

[0136] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0137] Acquire address information of a first target database;

[0138] According to the address information, a data information group of the first target database is obtained, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database;

[0139] Generate a flink-sql statement set according to the data information group, where the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information;

[0140] Generate the target Flink cluster based on the preset cluster creation statement;

[0141] Based on the target flink cluster and flink-sql statement set, the data information of each data table in the first target database is synchronously stored in the second target database.

[0142] In this embodiment, by obtaining the address information of the first target database; according to the address information, obtaining the data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all the information of a data table in the first target database; according to the data information group, generating a flink-sql statement set, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information; based on the preset cluster creation statement, generating a target flink cluster; based on the target flink cluster and the flink-sql statement set, the data information of each data table in the first target database is synchronously stored in the second target database. In this application, by After automatic identification of the database, the data information of each data table in the first target database is obtained, and a flink-sql statement set is automatically generated according to the data information group, and a target flink cluster is generated according to the preset cluster creation statement. Finally, based on the automatically generated target flink cluster and flink-sql statement set, the data information of each data table in the first target database is automatically synchronized to the second target database. Due to the use of an automated data synchronization method, it avoids the need for a large number of professionals to perform a large amount of development and configuration work, and avoids being restricted by the relevant knowledge reserves and professional levels of professionals, reducing development time and labor costs, which not only improves the intelligence level of data synchronization, but also improves the efficiency of real-time synchronization of online data to the data lake or physical database.

[0143] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0144] Acquire address information of a first target database;

[0145] According to the address information, a data information group of the first target database is obtained, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database;

[0146] Generate a flink-sql statement set according to the data information group, where the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information;

[0147] Generate the target Flink cluster based on the preset cluster creation statement;

[0148] Based on the target flink cluster and flink-sql statement set, the data information of each data table in the first target database is synchronously stored in the second target database.

[0149] In this embodiment, by obtaining the address information of the first target database; according to the address information, obtaining the data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all the information of a data table in the first target database; according to the data information group, generating a flink-sql statement set, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of flink-sql statements corresponds to one data information; based on the preset cluster creation statement, generating a target flink cluster; based on the target flink cluster and the flink-sql statement set, the data information of each data table in the first target database is synchronously stored in the second target database. In this application, by After automatic identification of the database, the data information of each data table in the first target database is obtained, and a flink-sql statement set is automatically generated according to the data information group, and a target flink cluster is generated according to the preset cluster creation statement. Finally, based on the automatically generated target flink cluster and flink-sql statement set, the data information of each data table in the first target database is automatically synchronized to the second target database. Due to the use of an automated data synchronization method, it avoids the need for a large number of professionals to perform a large amount of development and configuration work, and avoids being restricted by the relevant knowledge reserves and professional levels of professionals, reducing development time and labor costs, which not only improves the intelligence level of data synchronization, but also improves the efficiency of real-time synchronization of online data to the data lake or physical database.

[0150] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0152] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that: The method comprises: Acquire address information of a first target database; According to the address information, acquiring a data information group of the first target database, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database; Generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of the flink-sql statements corresponds to one piece of data information; Generate the target Flink cluster based on the preset cluster creation statement; Based on the target flink cluster and the flink-sql statement set, data information of each data table in the first target database is synchronously stored in the second target database.

2. The data synchronization method according to claim 1, characterized in that: The step of acquiring the data information group of the first target database according to the address information includes: Based on a preset query statement, query the first target database corresponding to the address information to obtain data structure information of each data table in the first target database; Based on each piece of the data structure information, respectively, a data query is performed on the data table corresponding to the data structure information to obtain the total number of data items and the number of fields in each of the data tables in the first target database; The data structure information, the total number of data items and the number of fields of the same data table are used as the data information of the data table; All the data information are taken as the data information group of the first target database.

3. The data synchronization method according to claim 1, characterized in that: The generating a flink-sql statement set according to the data information group includes: Acquire data synchronization configuration information of each data table in the second target database, wherein the data synchronization configuration information includes: account number, password, metadata database address, and data storage address; The flink-sql statement set is generated according to the preset code template, the data synchronization configuration information and the data information group.

4. The data synchronization method according to claim 1, characterized in that: Before generating the target flink cluster based on the preset cluster creation statement, the method further includes: Get cluster configuration information, including server login configuration information, Flink cluster running files, and running file placement path; The preset cluster creation statement is obtained according to the cluster configuration information.

5. The data synchronization method according to claim 1, characterized in that: After generating the target flink cluster based on the preset cluster creation statement, the method further includes: Generate target resource configuration parameters of the target Flink cluster according to the total number of data items and the number of fields in each of the data tables; According to the target resource configuration parameters, the preset resource configuration parameters of the target flink cluster are updated.

6. The data synchronization method according to claim 1, characterized in that: The step of synchronously storing the data information of each data table in the first target database to the second target database based on the target flink cluster and the flink-sql statement set includes: In response to a preset cluster startup command, submit each group of flink-sql statements in the flink-sql statement set to the target flink cluster in sequence; Each group of the flink-sql statements is executed separately through the target flink cluster to synchronously store the data information of each data table in the first target database to the second target database.

7. The data synchronization method according to claim 1, characterized in that: In the process of executing the target data of the first target database to be synchronously stored in the second target database based on the target flink cluster and the flink-sql statement set, the method further includes: Monitor whether there are any exceptions in the synchronization storage process through a preset exception monitoring mechanism, wherein the synchronization storage process is to synchronously store the target data of the first target database to the second target database based on the target flink cluster and the flink-sql statement set; If an exception is detected in the synchronization storage process, the type of the exception is determined based on a preset set of exception types; The exception is processed according to the type of the exception.

8. A data synchronization device, characterized in that: The device comprises: A first information acquisition module, used to acquire address information of a first target database; A second information acquisition module, configured to acquire a data information group of the first target database according to the address information, wherein the data information group includes at least one data information, and the data information is all information of a data table in the first target database; A statement set generation module, used to generate a flink-sql statement set according to the data information group, wherein the flink-sql set includes at least one group of flink-sql statements, and each group of the flink-sql statements corresponds to one piece of data information; The cluster creation module generates a target Flink cluster using a preset cluster creation statement. A data synchronization module is used to synchronize and store data information of each data table in the first target database to a second target database based on the target flink cluster and the flink-sql statement set.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data synchronization method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 7 are implemented.

Images