A method, system, device and medium for collecting data of GoldenDB database

Abstract

The present invention discloses a data collection method, system, device and medium for a GoldenDB database, and relates to the technical field of database collection. The method includes: creating a new collection cluster, completing the capture and integration of the database binlog log of the GoldenDB database cluster based on the service of the collection cluster, and generating a database log that can be reflowed by a single database node on each collection node of the collection cluster; starting the data distribution platform KAFKA cluster service to receive the real-time change data generated during the collection process of the data fusion platform; starting the service of the data fusion platform, retrieving and parsing the real-time change data in the database log of the collection cluster node, and pushing the real-time change data to the data distribution platform KAFKA cluster for downstream consumption. A method and system for quasi-real-time collection of table-level data of a GoldenDB database that is easy to operate and has a wide range of applications is provided, filling the gap in this subdivision of the GoldenDB database.

Description

Technical Field

[0001] The present invention relates to the technical field of database acquisition, and in particular to a method, system, device and medium for acquiring data of a GoldenDB database. Background Art

[0002] With the orderly advancement of information technology innovation in the financial industry, the GoldenDB database has been widely used in the industry. However, the construction of the GoldenDB database industry ecosystem is slow, and no effective solution has been proposed for table-level data collection.

[0003] This is because the GoldenDB database, as a typical representative of domestic distributed databases, is very different from traditional databases. Traditional databases have only one master database node, while the GoldenDB database has multiple master database nodes, and they are physically isolated from each other. In order to achieve quasi-real-time collection of table-level data in this distributed database, it is necessary to explore a data collection method and system that is adapted to the distributed database environment to fill the gap in this niche area of ​​the GoldenDB database. Summary of the invention

[0004] The present invention provides a data collection method, system, device and medium for a GoldenDB database, so as to solve the technical problem that there is a lack of corresponding collection solutions in the current field of data collection of a GoldenDB database.

[0005] On the one hand, the present invention provides a data collection method for a GoldenDB database, the method comprising: creating a new collection cluster, completing the capture and integration of the database binlog log of the GoldenDB database cluster based on the service of the collection cluster, and generating a database log that can be reflowed by a single database node on each collection node of the collection cluster; starting the data distribution platform KAFKA cluster service to receive the real-time change data generated during the collection process of the data fusion platform; starting the service of the data fusion platform, retrieving and parsing the real-time change data in the database log of the collection cluster node, and pushing the real-time change data to the data distribution platform KAFKA cluster for downstream consumption.

[0006] In one implementation of the present invention, the service of the collection cluster specifically includes: when the data on each database node in the GoldenDB database cluster changes, the event is written into the binlog log in sequence; based on the developed collection cluster one-click deployment script, the collection middleware is installed to build several collection nodes of the collection cluster; the collection function of the collection node is enabled to provide external services; wherein, the collection function is specifically: using the cdc user to enter the command line in each node of the collection cluster; enabling the I / O thread in the collection function, the command is START SLAVE IO_THREAD, the command is used to accept the binlog content sent by the binlog dump thread of each master DN node, and write it into the local relaylog, and each master DN node generates a relaylog; enabling the FETCH thread and COMPOSE thread in the collection function, the command is START SLAVECDC COMPOSE_THREAD.

[0007] In an implementation of the present invention, the method further includes: synchronizing the data collected by the collection node to the maximum position of the existing binlogs in all shards in the GoldenDB database cluster.

[0008] In one implementation of the present invention, the services of the data fusion platform specifically include: creating a new account for the collection cluster in the GoldenDB database cluster and granting permissions; connecting to the computing node, obtaining metadata information and full data of the data table, and displaying the metadata information on the page; calling the cleaning program to uniformly process the collected heterogeneous data source data; converting the data format of the local data distribution platform KAFKA data, using SchemaRegistry to define the data structure, and using the Confluent Avro structure to write to the data distribution platform.

[0009] In an implementation of the present invention, the method further includes: connecting to a collection node of a collection cluster by connecting to a database in Java language, and subscribing to a binlog log of the collection cluster node.

[0010] In one implementation of the present invention, the method further includes: adding a new data source node to configure the IP, port, account and password information of the GoldenDB database cluster and the collection cluster; connecting the computing node of the GoldenDB database cluster with the collection node of the collection cluster by connecting the database via Java language.

[0011] The present invention also provides a data acquisition system for a GoldenDB database, which is applied to the aforementioned data acquisition method for a GoldenDB database. The system includes: a GoldenDB database cluster, a collection cluster, a data fusion platform and a data distribution platform KAFKA cluster. The main architecture of the GoldenDB database cluster includes a management node, a global transaction node, a database node, and a computing node; the collection cluster is deployed on each server and is used to support the reflux of all sharded data for a distributed framework to generate a binlog log that can be refluxed by a single database node; the data fusion platform is used to parse the binlog incremental log of the collection middleware node into real-time data, complete the processing and conversion of the data format through advanced cleaning procedures, and push the real-time produced data to the data distribution platform KAFKA cluster so that downstream application systems can subscribe and consume it.

[0012] In one implementation of the present invention, the management node is used to execute all operation and maintenance operations of the distributed database system; wherein the operation and maintenance operations include system management, system backup and recovery, system metadata management, and data migration; the global transaction node is used to manage the life cycle of distributed transactions and generate a list of all currently executing distributed transactions; the database node is used to store business data and execute distributed sub-transactions; the computing node is used to receive business statements sent by applications, perform syntax analysis on business statements, and perform distributed optimization based on data distribution information, wherein the optimization includes statement rewriting and parallel distribution.

[0013] The present invention also provides a data collection device for a GoldenDB database, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can complete the aforementioned data collection method for a GoldenDB database.

[0014] The present invention also provides a non-volatile computer storage medium for data collection of a GoldenDB database, which stores computer executable instructions. The computer executable instructions are executed by a processor to complete the aforementioned method for data collection of a GoldenDB database.

[0015] The present invention provides a GoldenDB database data collection method, system, device and medium, which have the following beneficial effects:

[0016] (1) Integrate the GoldenDB database cluster and the collection cluster, obtain a series of real-time data technologies based on the incremental and full data of the GoldenDB database log, build a data link oriented to business goals, and quickly execute data tasks on demand;

[0017] (2) Supports real-time data collection, data subscription and distribution at the database table level, and achieves real-time data acquisition with a delay of seconds;

[0018] (3) It can support all application systems that use the GoldenDB database to access the collection business. You only need to configure the corresponding data source node information, link information and task information when the new system is connected. It is simple, convenient and easy to operate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 A flow chart of a data collection method for a GoldenDB database provided by an embodiment of the present invention;

[0021] Figure 2 A composition diagram of a data acquisition system of a GoldenDB database provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a data acquisition device for a GoldenDB database provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] The embodiment of the present invention provides a method, system, device and medium for collecting data of a GoldenDB database. The technical solution proposed in the embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A flow chart of a data collection method for a GoldenDB database provided by an embodiment of the present invention. Figure 1 As shown, the method mainly includes the following steps:

[0026] Step 101: Create a new collection cluster, capture and integrate the database binlog logs of the GoldenDB database cluster based on the services of the collection cluster, and generate a database log that can be reflowed by a single database node on each collection node of the collection cluster.

[0027] Step 102: Start the KAFKA cluster service of the data distribution platform to receive the real-time change data generated during the data fusion platform collection process.

[0028] Step 103: Start the service of the data fusion platform, retrieve and parse the real-time change data in the database log of the acquisition cluster node, and push the real-time change data to the data distribution platform KAFKA cluster for downstream consumption.

[0029] In an embodiment of the present invention, the service of the collection cluster specifically includes: when the data on each database node in the GoldenDB database cluster changes, the event is written into the binlog log in sequence; based on the developed one-click deployment script of the collection cluster, the collection middleware is installed to build several collection nodes of the collection cluster; the collection function of the collection node is enabled to provide external services; wherein, the collection function is specifically: using the cdc user to enter the command line in each node of the collection cluster; enabling the I / O thread in the collection function, the command is START SLAVE IO_THREAD, the command is used to accept the binlog content sent by the binlog dump thread of each master DN node, and write it into the local relaylog, and each master DN node generates a relaylog; enabling the FETCH thread and COMPOSE thread in the collection function, the command is START SLAVE CDCCOMPOSE_THREAD.

[0030] In an embodiment of the present invention, the method further includes: synchronizing the data collected by the collection node to the maximum position of the binlog existing in all shards in the GoldenDB database cluster.

[0031] In an embodiment of the present invention, the services of the data fusion platform specifically include: creating a new account for the collection cluster in the GoldenDB database cluster and granting permissions; connecting to the computing node, obtaining metadata information and full data of the data table, and displaying the metadata information on the page; calling the cleaning program to uniformly process the collected heterogeneous data source data; converting the data format of the local data distribution platform KAFKA data, using the Schema Registry to define the data structure, and using the Confluent Avro structure to write to the data distribution platform.

[0032] In the embodiment of the present invention, the method further includes: connecting to a collection node of a collection cluster by connecting to a database using Java language, and subscribing to a binlog log of the collection cluster node.

[0033] In an embodiment of the present invention, the method also includes: adding a new data source node to configure the IP, port, account and password information of the GoldenDB database cluster and the collection cluster; connecting the computing node of the GoldenDB database cluster and the collection node of the collection cluster by connecting to the database through the Java language.

[0034] The above is a data collection method for a GoldenDB database provided by an embodiment of the present invention. Based on the same inventive concept, an embodiment of the present invention also provides a data collection system for a GoldenDB database. Figure 2 A data acquisition system composition diagram of a GoldenDB database provided by an embodiment of the present invention, such as Figure 2 As shown, the system mainly includes: GoldenDB database cluster, collection cluster, data fusion platform and data distribution platform KAFKA cluster. The main architecture of the GoldenDB database cluster includes management node, global transaction node, database node and computing node; the collection cluster is deployed on each server and is used to support the backflow of all shard data for the distributed framework to generate a binlog log that can be backflowed by a single database node; the data fusion platform is used to parse the binlog incremental log of the collection middleware node into real-time data, complete the processing and conversion of the data format through advanced cleaning procedures, and push the real-time production data to the data distribution platform KAFKA cluster, so that the downstream application system can subscribe and consume it.

[0035] Furthermore, the management node is a node used for management. It carries all operation and maintenance operations of the distributed database system, including cluster high availability and high reliability management, system backup and recovery, system metadata management, data migration and other operations.

[0036] The global transaction node coordinates the computing nodes CN to perform concurrent control of distributed transactions, manages the life cycle of distributed transactions, and is responsible for maintaining a list of all currently executing distributed transactions.

[0037] Data nodes are nodes used to store business data and execute distributed sub-transactions. Usually a cluster contains multiple data nodes, each of which is an independent and autonomous database system. The data of a business will only fall into one data cluster. Business data will be horizontally split and distributed on several data nodes. In order to ensure data availability, multiple data backup nodes will be deployed for each data shard.

[0038] The main function of the computing node is to accept business statements sent by the application, perform grammatical analysis on the business statements, and perform distributed optimization based on the data distribution information, including statement rewriting, parallel distribution, etc., and finally generate a distributed execution plan. According to the execution plan, the statements are sent in parallel to the next layer of data node clusters.

[0039] In the embodiment of the present invention, when the data manipulation language, UPDATE, DELETE and INSERT operations or the data definition language DDL operations, CREATE, ALTER and DROP operations of the table in the GoldenDB database are performed, the metadata (table structure) of the database table and the binlog logs of the corresponding multiple database nodes will change. To realize the quasi-real-time collection of table-level data of the GoldenDB database, it is necessary to perceive the table structure changes and capture and integrate the binlog logs in real time. Therefore, the collection cluster, collection platform and data distribution platform are introduced to build the quasi-real-time collection system of table-level data of the GoldenDB database.

[0040] In the embodiment of the present invention, the collection middleware of the collection cluster is a product developed in C, which can be independently deployed on different servers to build a collection cluster, used for a distributed framework, supports the return of all shard data, and is used to generate a binlog that can be returned to a single DB.

[0041] Specifically, first, the collection middleware function is enabled in the GoldenDB database instance. When the data on each primary DN changes, the event change will be written to the binlog in sequence. When the collection middleware is linked to each primary DN, each primary DN machine will start the binlog dump thread for the collection middleware. When the binlog of each primary DN changes, the binlog dump thread will notify the collection middleware and send the corresponding binlog content to the collection middleware. Secondly, develop a one-click deployment script for the collection cluster. Then, use the deployment script to install the collection middleware to build a collection cluster. Thirdly, enable the collection function of the collection cluster. Then, the collection cluster is started to provide services to the outside world. The collection cluster is synchronized to the maximum position of the existing binlog in all shards of the GoldenDB database.

[0042] Enabling the collection function of the collection cluster includes: 1. Enter the command line as the cdc user on each node of the collection cluster.

[0043] 2. Enable the I / O thread in the collection function. The command is START SLAVE IO_THREAD, which is used to receive the binlog content sent by the binlog dump thread of each master DN node and write it into the local relaylog. Each master DN node generates a relaylog.

[0044] 3. Enable the FETCH thread and COMPOSE thread in the collection function. The command is START SLAVE CDCCOMPOSE_THREAD. Use multi-source replication to receive the binlog of the multi-shard (primary DN) host under the distribution. Assuming there are N shards, start N FETCH threads. Each FETCH thread reads a shard relaylog. Each FETCH thread has a transaction queue TQ. The size of the queue can be set. The initial setting is 32. The FETCH thread reads the transaction into the TQ. If the TQ is full, it waits; start a COMPOSE thread, open the binlog file, and create a complete distributed transaction queue FDTQ. The COMPOSE thread reads the first transaction in the TQ of each FETCH thread and notifies the corresponding FETCH thread. Let it read another one and put it into TQ, parse the CTID (all transaction submission IDs) and GMAP (distributed transactions that record how many shards the current distributed transaction involves) of each transaction, if it is a single-machine transaction, write it directly into the binlog file, and then read a transaction from the corresponding TQ and notify the FETCH thread. If it is a distributed transaction, sort these distributed transactions according to the value of CTID, traverse the sorted transactions, put the transaction into FDTQ, and according to its GMAP information, take out the other parts of the distributed transaction from the TQ of the corresponding FETCH line and put it into FDTQ. After all the files are completed, write the transaction into the binlog file.

[0045] In an embodiment of the present invention, a data fusion platform serves as a unified management platform for database log collection. The platform is seamlessly integrated with the collection cluster. The SOURCE module of the platform parses the binlog incremental log of the collection middleware node into real-time data in real time, and completes the processing and conversion of the data format through the advanced cleaning program. The SINK module of the platform finally pushes the real-time produced data to the KAFKA cluster, so that the downstream application system can subscribe and consume it.

[0046] Specifically, the services performed by the data fusion platform are:

[0047] 1. Create a new account for the collection system in the GoldenDB database cluster and grant the user the static global SELECT privilege for the collection platform to access the database using the Java Database Connectivity (JDBC) method.

[0048] 2. Create a new account for the collection system on each node of the collection cluster, and grant replication client and replication slave permissions to this account so that the collection platform can access the collection cluster using JDBC.

[0049] 3. Add a new data source node on the collection platform and configure the IP, port, account and password information of the GoldenDB database cluster and the collection cluster. Connect the computing node of the GoldenDB database cluster and the collection node of the collection cluster through JDBC to achieve seamless integration between systems. At the same time, it supports filling in multiple computing nodes and multiple collection nodes in the JDBC connection string to prevent data collection from being unavailable after a single node fails.

[0050] 4. Collect information metadata and obtain metadata information of the data table by connecting to the database computing node. The metadata information is displayed on the page so that users can select the synchronized data table and view the synchronized fields.

[0051] 5. Full data acquisition: The collection platform obtains the full data of the synchronized data table by connecting to the database computing node.

[0052] 6. To obtain table structure change information, the collection platform uses JDBC to connect to the collection node of the collection cluster, subscribe to the binlog log of the collection cluster node, and parse the log to obtain table structure change information.

[0053] 7. To obtain incremental data at the table level, the collection platform uses JDBC to connect to the collection node of the collection cluster and subscribe to the binlog logs of the collection cluster node. The collection task configured in the WEB interface of the collection platform management console generates a list of subscription tables. The SOURCE module of the collection platform will only retrieve and parse the logs of the tables involved in the subscription order to obtain incremental change data, and replay the incremental change data to the KAFKA (business repository) of the collection platform.

[0054] 8. The advanced cleaning program mainly realizes the unified processing of data collected from heterogeneous data sources. The KAFKA data destination can combine advanced cleaning with the data mapping advanced cleaning of the Avro data structure configured on the page to realize the KAFKA message landing in the standard Avro format.

[0055] 9. The collected data is written to the downstream. The SINK module of the collection platform calls the advanced cleaning program to convert the local KAFKA data into a data format, uses the Schema Registry to define the data structure, and uses the Confluent Avro structure to write to the data distribution platform. This is a data format example.

[0056] In the embodiment of the present invention, Figure 2As shown in the figure, each collection middleware of the collection cluster captures and integrates the binlog logs of multiple database master nodes of the GoldenDB database cluster through Socket, and generates a database binlog log that can be reflowed by a single database node on each node of the collection cluster; the database quasi-real-time data fusion platform directly connects to the computing node of the GoldenDB database cluster through JDBC to obtain the metadata information of the database, and retrieves and analyzes the database table subscribed to the collection platform in the database log of the collection cluster node through Socket to obtain incremental change data, and generates real-time data and pushes it to the destination KAKKA cluster for downstream consumption.

[0057] The embodiment of the present invention designs a collection middleware based on data change capture and database log integration technology, which can be deployed on an independent server to capture and integrate the data change parts in the database logs of multiple master database nodes, and finally form a globally consistent binlog log. The collection cluster is integrated with the data fusion platform. First, the globally consistent binlog log is parsed in real time and stored in the database through the SOURCE module of the collection platform, and then written to the destination KAFKA cluster according to the cleaning rules through the SINK module of the collection platform, finally realizing the quasi-real-time collection of distributed database table-level data.

[0058] The above is a data acquisition system of a GoldenDB database provided by an embodiment of the present invention. Based on the same inventive concept, an embodiment of the present invention also provides a data acquisition device of a GoldenDB database. Figure 3 A schematic diagram of a data acquisition device for a GoldenDB database provided in an embodiment of the present invention, such as Figure 3 As shown, the device mainly includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor; wherein the memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can complete the aforementioned data collection method of the GoldenDB database.

[0059] In addition, an embodiment of the present invention further provides a non-volatile computer storage medium for data collection of a GoldenDB database, which stores computer executable instructions, and the computer executable instructions are executed by a processor to complete the aforementioned method for data collection of a GoldenDB database.

Claims

1. A data collection method for a GoldenDB database, characterized in that: The method comprises: Create a new collection cluster, capture and integrate the database binlog logs of the GoldenDB database cluster based on the services of the collection cluster, and generate a database log that can be reflowed by a single database node on each collection node of the collection cluster; Start the KAFKA cluster service of the data distribution platform to receive the real-time change data generated during the data fusion platform collection process; Start the service of the data fusion platform, retrieve and parse the real-time change data in the database log of the acquisition cluster node, and push the real-time change data to the data distribution platform KAFKA cluster for downstream consumption; The services of the data fusion platform specifically include: Create a new account for the collection cluster in the GoldenDB database cluster and grant permissions; Connect to the computing node, obtain the metadata information and full data of the data table, and display the metadata information on the page; Call the cleaning program to uniformly process the collected data from heterogeneous data sources; Convert the data format of the local data distribution platform KAFKA data, use Schema Registry to define the data structure, and use the Confluent Avro structure to write to the data distribution platform.

2. The data collection method of a GoldenDB database according to claim 1, characterized in that: The services of the collection cluster specifically include: When the data on each database node in the GoldenDB database cluster changes, the event is written to the binlog log in sequence; Install the collection middleware based on the developed collection cluster one-click deployment script and build several collection nodes of the collection cluster; Enable the collection function of the collection node to provide external services; the collection function is specifically: use the cdc user to enter the command line in each node of the collection cluster; enable the I / O thread in the collection function, the command is START SLAVE IO_THREAD, the command is used to receive the binlog content sent by the binlog dump thread of each master DN node, and write it into the local relaylog, each master DN node generates a relaylog; enable the FETCH thread and COMPOSE thread in the collection function, the command is START SLAVE CDC COMPOSE_THREAD.

3. The data collection method of a GoldenDB database according to claim 2, characterized in that: The method further includes: synchronizing the data collected by the collection node to the maximum position of the binlog existing in all shards in the GoldenDB database cluster.

4. The data collection method of a GoldenDB database according to claim 1, characterized in that: The method further includes: connecting to a collection node of a collection cluster by connecting to a database through Java language, and subscribing to a binlog log of the collection cluster node.

5. The data collection method of a GoldenDB database according to claim 1, characterized in that: The method further comprises: Add a new data source node to configure the IP, port, account, and password information of the GoldenDB database cluster and collection cluster; Use Java language to connect to the database to connect the computing nodes of the GoldenDB database cluster and the collection nodes of the collection cluster.

6. A data collection system for a GoldenDB database, applied to a data collection method for a GoldenDB database as claimed in any one of claims 1 to 5, characterized in that: The system includes: a GoldenDB database cluster, a collection cluster, a data fusion platform and a data distribution platform KAFKA cluster. The main architecture of the GoldenDB database cluster includes a management node, a global transaction node, a database node and a computing node. The collection cluster is deployed on each server and is used to support the backflow of all sharded data for the distributed framework to generate a binlog log that can be backflowed by a single database node. The data fusion platform is used to parse the binlog incremental log of the collection middleware node into real-time data, complete the processing and conversion of the data format through advanced cleaning procedures, and push the real-time produced data to the data distribution platform KAFKA cluster so that the downstream application system can subscribe and consume it.

7. The data collection system of the GoldenDB database according to claim 6, characterized in that: The management node is used to perform all operation and maintenance operations of the distributed database system; wherein the operation and maintenance operations include system management, system backup and recovery, system metadata management, and data migration; the global transaction node is used to manage the life cycle of distributed transactions and generate a list of all currently executing distributed transactions; the database node is used to store business data and execute distributed sub-transactions; the computing node is used to receive business statements sent by applications, perform syntax analysis on business statements, and perform distributed optimization based on data distribution information, and the optimization includes statement rewriting and parallel distribution.

8. A data collection device for GoldenDB database, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can complete the data collection method of the GoldenDB database described in any one of claims 1-5.

9. A non-volatile computer storage medium for data collection of a GoldenDB database, storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the data collection method of the GoldenDB database described in any one of claims 1-5.

Images