A distributed database synchronization method, system, device and medium

Abstract

The application discloses a kind of distributed database synchronization method and system and device and medium, it is related to data synchronization technical field, the method includes the following steps: the first parameter information of configuration distributed database;Configuration database management system second parameter information;Configuration satisfies the third parameter information of distributed data stream processing unit operation;The first parameter information, the second parameter information and the third parameter information are parsed and verified, and parsing verification result is obtained;Based on the parsing verification result, create structured query language and generate real-time warehousing code script;The structured query language and the real-time warehousing code script are uploaded to the distributed data stream processing unit release and execute synchronization task and obtain data synchronization result.The application adopts distributed data stream processing unit to execute data synchronization task, improves synchronization efficiency, significantly reduces distributed database full data synchronization time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data synchronization, in particular to a distributed database synchronization method and system, device and medium. BACKGROUND

[0002] The existing distributed database synchronization method is to use a client collection tool written in Java or other languages to obtain full and incremental data of the distributed database. This method uses a single process task to synchronize data. When the data volume of the data table to be synchronized is particularly large and there is a large amount of data, the time for synchronizing full data is long, and the collection efficiency is low. SUMMARY

[0003] To solve the problem of long time spent in synchronizing data of a distributed database and low collection efficiency, the present application provides a distributed database synchronization method and system, device and medium. The method comprises the following steps:

[0004] Configuring first parameter information of the distributed database;

[0005] Configuring second parameter information of the database management system;

[0006] Configuring third parameter information meeting the operation of the distributed data stream processing unit;

[0007] Analyzing and verifying the first parameter information, the second parameter information and the third parameter information to obtain an analysis and verification result;

[0008] Based on the analysis and verification result, creating a structured query language and generating a real-time warehousing code script;

[0009] Uploading the structured query language and the real-time warehousing code script to the distributed data stream processing unit to publish and execute a synchronization task to obtain a data synchronization result.

[0010] The principle of the present application is that the first parameter information of the distributed database and the second parameter information of the database management system are configured to meet the database management system obtaining full or incremental data from the distributed database, and the third parameter information meeting the operation of the distributed data stream processing unit is configured to define the general parameters of the distributed data stream processing unit for computing and synchronizing data. First, the database management system creates a structured query language and generates a real-time warehousing code script file based on the analysis and verification results, and uploads the structured query language and the real-time warehousing code script to the distributed data stream processing unit for publishing and executing a synchronization task. Then, the structured query language is used to obtain data of the distributed database, and the obtained data is stored in the database management system based on the real-time warehousing code script. Finally, the distributed data stream processing unit starts a multi-thread processing unit to perform synchronous calculation on the obtained data in the database management system to complete data synchronization.

[0011] Preferably, the method configures the first parameter information on a first page, configures the second parameter information on a second page, and configures the third parameter information on a third page. The use of three pages to configure the first parameter information, the second parameter information and the third parameter information respectively facilitates the differentiation of parameter information and makes the configuration interface more concise.

[0012] Preferably, it is judged whether the distributed database starts the change data capture function, and if not, the task is terminated and the user is prompted to start the change data capture function of the distributed database. The change data capture function of the distributed database records the insertion, update and deletion activities of the table, so that detailed information of these changes can be provided in an easy-to-use relational format.

[0013] Preferably, the verification method of the first parameter information, the second parameter information and the third parameter information includes judging whether the first parameter information meets the first preset parameter configuration requirement, and if not, prompting to set the corresponding parameter information; judging whether the second parameter information meets the second preset parameter configuration requirement, and if not, prompting to set the corresponding parameter information; judging whether the third parameter information meets the third preset parameter configuration requirement, and if not, prompting to set the corresponding parameter information. The first preset parameter configuration requirement, the second preset parameter configuration requirement and the third preset parameter configuration requirement stipulate that some parameters must be configured. When the configured parameter information does not meet the preset parameter configuration requirement, the corresponding parameter information needs to be prompted to be set.

[0014] Preferably, the verification manner of the first parameter information, the second parameter information and the third parameter information further comprises: judging whether the type of the first parameter information satisfies a first preset type respectively, and if not, prompting to change the type of the first parameter information; judging whether the type of the second parameter information satisfies a second preset type respectively, and if not, prompting to change the type of the second parameter information; judging whether the third parameter information satisfies a third preset type respectively, and if not, prompting to change the type of the third parameter information. The first preset type, the second preset type and the third preset type stipulate the type of the parameter information to be configured, so as to ensure that the parameter information of the correct type can be configured.

[0015] Preferably, the creating the structured query language specifically comprises creating a stream table statement and generating a structured query statement containing data insertion code; and generating the real-time warehousing code script based on the structured query statement. The stream table statement is a code template for dynamically generating a table creation statement according to the input parameters; the structured query statement containing the data insertion code is used to convert the data acquisition mode of the distributed database into a structured query mode, and the synchronized data is stored into other external systems in dependence on the data insertion code; and the real-time warehousing code script is used to store the acquired data into the database management system, so as to facilitate subsequent data synchronization calculation of the distributed data stream processing unit and the database management system.

[0016] Preferably, the resource end script file and the connection end script file of the database management system are generated based on the structured query statement. The resource end script file is used to provide a port for the database management system to acquire full or incremental data of the distributed database, and the connection end script file is used to provide a port for storing the data after the synchronization calculation into other preset external systems.

[0017] Preferably, the stream table statement and the real-time warehousing code are uploaded to the distributed data stream processing unit to publish and execute a synchronization task, which comprises:

[0018] building a job graph and submitting the job graph to a job manager;

[0019] the job manager analyzes the logical graph structure of the job graph and optimizes to generate a task execution graph;

[0020] based on the task execution graph, the job manager starts a corresponding number of task executors in different task managers, and the task executors execute the data synchronization task and obtain a data synchronization result.

[0021] Wherein, the flow table statement and the real-time warehousing code are uploaded to the distributed data flow processing unit, that is, the data stream of the database management system is uploaded to the client of the distributed data flow processing unit, and then a job graph is constructed based on the related program of the data stream, the job graph is a data structure for expressing a job which is uniquely identified by the data stream engine of the distributed data flow processing unit. The constructed job graph is submitted to a job manager, the job manager is responsible for task scheduling and resource management of data synchronization, the job manager analyzes the logical graph structure of the job graph to generate a task execution graph, the task execution graph is a parallelized version of the job graph, can perform multi-threaded data processing according to configuration parameters, and is also the most core data structure of the scheduling layer of the distributed data flow processing unit. Finally, based on the task execution graph, the job manager starts a corresponding number of task executors in different task managers, the task manager is responsible for specific task execution and resource application and management of the corresponding task on each node, and the task executor is an executor for executing the data synchronization task, and the data synchronization result is obtained by the task executor completing the data synchronization calculation task.

[0022] The application further provides a distributed database synchronization system, the system comprising:

[0023] A first configuration unit configured to configure first parameter information of the distributed database;

[0024] A second configuration unit configured to configure second parameter information of the database management system;

[0025] A third configuration unit configured to configure third parameter information meeting the operation of the distributed data flow processing unit;

[0026] An analysis unit configured to analyze and verify the first parameter information, the second parameter information and the third parameter information to obtain an analysis and verification result;

[0027] A creation unit configured to create a structured query language and generate a real-time warehousing code script based on the analysis and verification result;

[0028] An execution unit configured to upload the structured query language and the real-time warehousing code to the distributed data flow processing unit to publish and execute a synchronization task to obtain a data synchronization result.

[0029] The application further provides a distributed database synchronization device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the distributed database synchronization method when executing the computer program.

[0030] The application further provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the distributed database synchronization method.

[0031] The one or more technical solutions provided by the application have at least the following technical effects or advantages.

[0032] The application adopts a distributed data flow processing unit to perform a data synchronization task, thereby improving synchronization efficiency and significantly reducing the full data synchronization time of a distributed database. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principle of the application, and should not be used to limit the application;

[0034] Figure 1 is a flowchart of the distributed database synchronization method in the application;

[0035] Figure 2 is a composition diagram of the distributed database synchronization system in the application. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned purposes, features and advantages of the application, the application will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that the embodiments of the application and the features in the embodiments can be combined with each other without conflict.

[0037] In the following description, many specific details are set forth in order to provide a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein within the scope of the application, therefore, the scope of protection of the application is not limited by the specific embodiments disclosed below.

[0038] Embodiment one

[0039] Please refer to Figure 1 is a flowchart of the distributed database synchronization method in the application, a distributed database synchronization method includes the following steps:

[0040] The first parameter information of the distributed database is configured;

[0041] The second parameter information of the database management system is configured;

[0042] The third parameter information meeting the operation of the distributed data flow processing unit is configured;

[0043] The first parameter information, the second parameter information and the third parameter information are parsed and verified to obtain a parsing and verification result;

[0044] Based on the analysis and verification result, a structured query language is created and a real-time warehousing code script is generated;

[0045] The structured query language and the real-time warehousing code script are uploaded to the distributed data stream processing unit to publish and execute a synchronization task to obtain a data synchronization result.

[0046] Preferably, the distributed database is a DB2 database, the database management system is a MySQL database management system, and the distributed data stream processing unit is a Flink architecture. The distributed database, the database management system and the distributed data stream processing unit involved in subsequent embodiments of the present application are all based on a DB2 database, a MySQL database management system and a Flink architecture.

[0047] Preferably, the first parameter information of the distributed database includes a database address (such as host: 172.27.17.109), a port (such as port: 50000), schema: TESTEB (organization and structure of the database), an account (such as username: db2inst1) and password information (such as password: 123456). The above data is only one embodiment of the present application, and in actual application, it can be configured as needed, and the present application is not specifically limited. The second parameter information of the database management system includes an address (such as host: 172.27.17.109), a port (such as port: 3306), a database (such as database: mydb), an account (such as username: root) and password information (such as password: 123456). The above data is only one embodiment of the present application, and in actual application, it can be configured as needed, and the present application is not specifically limited. The third parameter information configured to meet the operation of the distributed data stream processing unit includes task parallelism (such as parallelism = 4), checkpoint time interval (such as checkpoint.interval.ms = 1000) and distributed storage address for storing checkpoints (such as checkpoint.location = hdfs: / / node1:8020 / flink / flink-checkpoints). The above data is only one embodiment of the present application, and in actual application, it can be configured as needed, and the present application is not specifically limited.

[0048] The distributed database synchronization method configures the first parameter information on the first page, the second parameter information on the second page, and the third parameter information on the third page. The allocation of these three parameter configuration pages can be done on one, two, or three pages, with three pages being preferred for easier differentiation of parameter information and a more concise configuration interface.

[0049] The process involves determining whether change data capture is enabled in the distributed database. If not, the task is terminated, and the user is prompted to enable change data capture. The command `db2 "get snapshotfor db on db_hadr"|grep -p HADR` is executed in the distributed database to check if change data capture is enabled. If it is not enabled, the user is prompted to enter the command `START HADR` to enable change data capture in the distributed database.

[0050] When the configuration parameter information does not meet the preset parameter information configuration requirements, such as the preset parameter information configuration requirements for a distributed database being address, port, account, and password information, and the configuration parameter information being port, account, and password information, then the user needs to be prompted to configure the address information of the distributed database. The above data is only for illustrative purposes. In actual practice, the preset parameter information configuration requirements can be set as needed, and this invention does not impose specific limitations.

[0051] When the configuration parameter information type does not meet the preset type, such as when configuring the port information of the distributed database, the port number range is 0 to 65535. If a string or other type is used, the parameter type setting is unreasonable. In this case, the user needs to be prompted to modify the type of the distributed database port information. The above data is only for illustrative purposes. In actual process, it can be set as needed. This invention does not impose specific limitations.

[0052] The creating structured query language specifically includes creating a stream table statement and generating a structured query statement containing data insertion code; based on the structured query statement, a real-time warehousing code script is generated. The creating stream table statement specifically includes setting {id INT; name STRING (name string); description STRING (enumeration type string); PRIMARY KEY (id) NOT ENFORCED (not enforced primary key constraint)}; WITH {'connector' = 'jdbc' (through Java database connection); 'url' = 'jdbc:db2: / / 172.27.17.109:50000 / TESTDB' (uniform resource locator including using Java database connection, name db2, address, port and schema); 'username' = 'db2inst1' (account 'db2inst1'); 'password' = '123456' (password 123456); 'database-name' = 'TESTDB' (database name); 'table-name' = 'products' (table name)}; the stream table statement is a code template, and a table creation statement is dynamically generated according to the input parameters. The structured query statement containing the data insertion code is mainly implemented through an insert into statement, and the specific code belongs to the prior art and should be known by the person skilled in the relevant art. The implementation manner of generating the real-time warehousing code script can be specifically referred to the comparison file CN113961570A.

[0053] The resource-side script file and the connection-side script file of the database management system are generated. The resource-side script file is generated specifically as follows: {id INT; name STRING (name string); description STRING (enumeration type string); PRIMARY KEY (id) NOT ENFORCED (not enforced primary key constraint)}; WITH {'connector'='jdbc' (through Java database connection); 'url'='jdbc:db2: / / 172.27.17.109:50000 / TESTDB' (uniform resource locator including using Java database connection, name db2, address, port and database name); 'username'='db2inst1' (account is 'db2inst1'); 'password'='123456' (password is 123456); 'database-name'='TESTDB' (database name); 'table-name'='products' (table name)}. The connection-side script file is generated specifically as follows: {id INT; name STRING (name string); description STRING (enumeration type string); PRIMARY KEY (id) NOT ENFORCED (not enforced primary key constraint)}; WITH {'connector'='jdbc' (through Java database connection); 'url'='jdbc:mysql: / / 172.27.17.109:3306 / mydb' (uniform resource locator including using Java database connection, name mysql, address, port and database name); 'username'='root' (account is 'root'); 'password'='123456' (password is 123456); 'database-name'='mydb' (database name); 'table-name'='products_copy' (table name)}. For the resource-side script file and the connection-side script file, the above implementation manner only provides an embodiment, and the implementation manner of the resource-side script file and the connection-side script file is not limited specifically.

[0054] Wherein, a job graph is constructed, and the job graph is submitted to a job manager; the job manager parses a logical graph structure of the job graph, and optimizes a task execution graph to be generated; based on the task execution graph, the job manager starts a corresponding number of task executors in different task managers, and the task executors execute a data synchronization task and obtain a data synchronization result. First, data sources and programs obtained by a Mysql database management system are uploaded to a client of a Flink architecture, and the client parses an execution plan of the program and optimizes the execution plan of the program after receiving the program. The specific content of the parsing program is to parse each calculation in the program into an operator, and then the operators are combined according to the relationship between the operators to form a job graph composed of the operators. The specific content of the optimized program is to submit the generated job graph to the job manager, and the job manager combines multiple nodes meeting a condition together as a node to form a task execution graph, so that serialization / deserialization / transmission consumption required by data flowing between the nodes can be reduced, and throughput can be improved. The job manager is responsible for scheduling of the entire Flink cluster task and management of resources, allocates corresponding task slots for the submitted data sources according to a use condition of the task slots on the task manager, communicates with the task manager through an Actor System (communication system), obtains execution conditions of the task, and sends the execution conditions to the client through the Actor System (communication system). Meanwhile, in the process of task execution, the job manager triggers a checkpoint operation, and each task manager node receives the checkpoint operation instruction and completes the checkpoint operation. When the task manager receives the task arranged by the job manager, the task manager starts a task executor using a task slot resource, the task executor is a smallest task execution unit in the Flink, a network connection for data access is established, data is received, and data processing and synchronization are started. When all the tasks are completed, the Flink feeds back information of the task execution to the client, and releases the resources in the task manager for use in the next task submission. The data after synchronization can be stored into an external system by using a source end script file and a structured query language for inserting data, and the selection of the external system is not limited in the application.

[0055] Embodiment two

[0056] Please refer to Figure 2 , Figure 2 The distributed database synchronization system provided in the embodiment two of the application includes:

[0057] The first configuration unit is configured to configure first parameter information of the distributed database.

[0058] The second configuration unit is configured to configure second parameter information of the database management system.

[0059] A third configuration unit is configured to configure third parameter information meeting running of the distributed data stream processing unit.

[0060] A parsing unit is configured to parse and verify the first parameter information, the second parameter information and the third parameter information to obtain a parsing verification result.

[0061] A creating unit is configured to create a structured query language and generate a real-time warehousing code script based on the parsing verification result.

[0062] An executing unit is configured to upload the structured query language and the real-time warehousing code to the distributed data stream processing unit to publish and execute a synchronization task to obtain a data synchronization result.

[0063] Embodiment three

[0064] Embodiment three of the present application provides a distributed database synchronization device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the distributed database synchronization method when executing the computer program.

[0065] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0066] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the distributed database synchronization device in the application by running or executing data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, an intelligent memory card, a secure digital card, a flash memory card, at least one magnetic disk storage device, a flash memory device or other volatile solid-state memory device.

[0067] Embodiment four

[0068] The fourth embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the distributed database synchronization method.

[0069] The distributed database synchronization device can be stored in a computer readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment methods, and can also be stored in a computer readable storage medium by a computer program, which can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, point carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0070] The present application has described the basic concept, and it is obvious that the above detailed disclosure is only used as an example and does not constitute a limitation on the present application. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and corrections to the present application. Such modifications, improvements and corrections are suggested in the present application, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present application.

[0071] Meanwhile, specific words are used in the present application to describe the embodiments of the present application. As "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "one alternative embodiment" mentioned in different positions in the present application does not necessarily mean the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be properly combined.

[0072] Moreover, those skilled in the art will appreciate that the various aspects of the disclosure can be illustrated and described in connection with a number of various kinds of systems or circumstances, including any new and useful processes, machines, products, or compositions of matter, or any new and useful improvements thereof, as defined by the plain language of the claims. Accordingly, the various aspects of the disclosure can be implemented in whole or in part any number of hardware, software codes, firmware codes, and / or other programming codes. Such hardware, software codes, firmware codes, and / or other programming codes can be referred to as a "data block", "module", "engine", "unit", "component", or "system". In addition, aspects of the disclosure can be embodied as a computer program product on one or more computer readable media (moving or physical) having computer readable program code embodied therein.

[0073] Computer storage media can include a propagated data signal with the computer program code embodied therein, e.g., in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer storage media can be any media that can be accessed by a computer. By way of example, and not limitation, such computer storage media can comprise RAM, ROM, EEPROM, CD-ROM or any other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer program code in the form of computer-readable program code means, and which can be accessed by a computer. Also, any connection is properly termed a computer storage medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, or twisted pair, as exemplary signal bearing media, then the coaxial cable, fiber optic cable, or twisted pair are computer storage media that transmits the software. Computer storage media can also comprise computer program code temporarily and / or actually loaded into and executed by a computer. Such a computer program product can take many forms, including but not limited to a tangible storage medium (e.g., non-transitory storage medium) like a floppy diskette, a CD-ROM, a zip drive, a USB drive, a magnetic tape, or any other magnetic storage device, a ROM, a RAM, a flash memory, a phase-change memory, or any other non-transitory computer storage medium.

[0074] The computer program code for carrying out the operations of the aspects of the disclosure can be written in any one or more of a variety of programming languages, including an object-oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, and the like, conventional procedural programming languages, such as the "C" programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages. The program code can execute entirely on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic program code can be downloaded from an on-line system source to a user's computer for use in accordance with the present disclosure. The various forms of computer program code or

[0075] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0076] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0077] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0078] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

[0079] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0080] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method for distributed database synchronization, the method comprising: The method comprises the following steps: configuring first parameter information of a distributed database, the first parameter information comprising database address, port, organization and structure of the database, account and password information; configuring second parameter information of a database management system, the second parameter information comprising address, port, database, account and password information; configuring third parameter information meeting the operation of a distributed data stream processing unit, the third parameter information comprising task parallelism, checkpoint time interval and distributed storage address of checkpoint storage; parsing and verifying the first parameter information, the second parameter information and the third parameter information to obtain a parsing and verification result; based on the parsing and verification result, creating a structured query language and generating a real-time warehousing code script; uploading the structured query language and the real-time warehousing code script to the distributed data stream processing unit to publish and execute a synchronization task to obtain a data synchronization result; creating a structured query language specifically comprises creating a stream table statement and generating a structured query statement containing an inserted data code; based on the structured query statement, generating the real-time warehousing code script; based on the structured query statement, generating a resource end script file and a connection end script file of the database management system; uploading the stream table statement and the real-time warehousing code to the distributed data stream processing unit to publish and execute a synchronization task comprises: building a job graph and submitting the job graph to a job manager; the job manager parses the logical graph structure of the job graph and optimizes to generate a task execution graph; based on the task execution graph, the job manager starts a corresponding number of task executors in different task managers, and the task executors execute a data synchronization task and obtain a data synchronization result.

2. The method of claim 1, wherein, The method configures the first parameter information on the first page of a configuration page, configures the second parameter information on the second page of the configuration page, and configures the third parameter information on the third page of the configuration page.

3. The method of claim 1, wherein, determining whether the distributed database is started with a change data capture function, if not, terminating the task and prompting the user to start the change data capture function of the distributed database.

4. The method of claim 1, wherein, The verification method of the first parameter information, the second parameter information and the third parameter information comprises: determining whether the first parameter information meets a first preset parameter configuration requirement, if not, prompting to set the corresponding parameter information; determining whether the second parameter information meets a second preset parameter configuration requirement, if not, prompting to set the corresponding parameter information; determining whether the third parameter information meets a third preset parameter configuration requirement, if not, prompting to set the corresponding parameter information.

5. The method of claim 1, wherein, The verification manner of the first parameter information, the second parameter information and the third parameter information further comprises: judging whether the type of the first parameter information satisfies a first preset type respectively, and if not, prompting to change the type of the first parameter information; judging whether the type of the second parameter information satisfies a second preset type respectively, and if not, prompting to change the type of the second parameter information; judging whether the third parameter information satisfies a third preset type respectively, and if not, prompting to change the type of the third parameter information.

6. A distributed database synchronization system, comprising: The system comprises: A first configuration unit configured to configure first parameter information of a distributed database, the first parameter information comprising database address, port, organization and structure of the database, account and password information; A second configuration unit configured to configure second parameter information of a database management system, the second parameter information comprising address, port, database, account and password information; A third configuration unit configured to configure third parameter information satisfying operation of a distributed data stream processing unit, the third parameter information comprising task parallelism, checkpoint time interval and distributed storage address of checkpoint storage; An analysis unit configured to analyze and verify the first parameter information, the second parameter information and the third parameter information to obtain an analysis and verification result; A creation unit configured to create structured query language and generate real-time warehousing code script based on the analysis and verification result; An execution unit configured to upload the structured query language and the real-time warehousing code to the distributed data stream processing unit to publish and execute a synchronization task to obtain a data synchronization result; Creating structured query language specifically comprises creating a stream table statement and generating a structured query statement containing an inserted data code; and generating the real-time warehousing code script based on the structured query statement; Generating resource end script files and connection end script files of the database management system based on the structured query statement; Uploading the stream table statement and the real-time warehousing code to the distributed data stream processing unit to publish and execute a synchronization task comprises: Building a job graph and submitting the job graph to a job manager; The job manager analyzes the logical graph structure of the job graph to optimize a task execution graph; Based on the task execution graph, the job manager starts a corresponding number of task executors in different task managers, and the task executors execute a data synchronization task to obtain a data synchronization result.

7. A distributed database synchronization apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the computer program to implement the steps of the distributed database synchronization method of any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the distributed database synchronization method of any one of claims 1-5.

Images