Data whole library synchronization method and device based on multiplexer, equipment and medium

Abstract

The invention belongs to the technical field of whole data library synchronization, and discloses a whole data library synchronization method, device and equipment based on a multiplexer and a medium, and the method comprises the following steps: configuring the multiplexer in an Flink distributed computing framework; generating a fragment set comprising all fragments of a plurality of source tables by using a multiplexer, and constructing a fragment distributor; obtaining an execution task graph by using the source data operator, and transmitting the execution task graph to the management node; the computing node requests fragmentation from the management node, the management node responds to the fragmentation request and requests to allocate resource slots to the computing node according to the execution task graph, the fragmentation is allocated to the resource slots in the computing node through the fragmentation distributor for calculation, and the source tables are synchronized to the data lake. By implementing the method and the device, the table task online workload is greatly reduced, the same resource slot can be utilized to execute the fragmentation logic of different tables, and slot computing resources are reused.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing of big data, and in particular to a method, device, equipment and medium for synchronizing an entire data library based on a multiplexer. Background Art

[0002] After years of operation and asset accumulation, enterprises have accumulated tens of thousands of data assets. At the current stage of digital transformation, the first step is data integration, which synchronizes business database data to the data platform. The existing data is synchronized from the business database to the data platform. Currently, the tools for transmitting data include Sqoop, DataX, etc., which have become mature in synchronizing single tables to Hive scenario applications.

[0003] With the rise of data lake technologies such as Hudi, more and more companies are writing business table data in the format of data lake tables to achieve Upsert semantics, just-in-time writing, incremental reading and other features of large-scale tables. Single-table entry into the lake refers to extracting data from the business library table using JDBC technology, and writing it to the Hudi table after partitioning, bucketing, sorting, batch writing and other steps. In the practice of enterprise batch and stream integration, the Flink distributed computing engine is generally used to establish a synchronization link, that is, the data flow direction is: from the business library to Flink, and then to Hudi. Whole-database entry into the lake means that multiple tables of the business library or even the entire library table can be synchronized to Hudi through only one Flink task. Compared with single-table entry into the lake, the whole-database entry into the lake technology can reduce users' repeated operations and save docking time, thereby accelerating data landing and application.

[0004] However, the whole reservoir into the lake process usually has the following problems:

[0005] Data synchronization issues in large existing tables. Using the hash modulus method to process string primary key tables will result in a full table scan, which will put a lot of pressure on the database.

[0006] Problems with data synchronization of multiple source tables. A stock synchronization task requires several connections to pull a table, and multiple tables require multiple or more connections. Partitioned tables require even more connections, and database connection resources are very valuable. An incremental synchronization task processes incremental data by consuming Kafka topics. When data from different tables is in different topics, multiple synchronization tasks need to be used to consume different Kafka topics. When data from different tables is in the same topic, multiple synchronization tasks need to be used to consume the Kafka topic multiple times. After setting the filtering conditions, the target table data is obtained. In the scenario with multiple source tables, the network load is serious.

[0007] Problems in the data processing stage. The Flink engine uses slot resources to process data. Usually, a slot only processes tasks for a specified table. In scenarios with multiple source tables, using too many slots results in a waste of computing resources. Summary of the invention

[0008] The purpose of the embodiments of the present application is to propose a method, device, equipment and medium for synchronizing the entire database based on a multiplexer to solve the problem of synchronizing large table data and multiple source table data.

[0009] In order to solve the above technical problems, the embodiment of the present application provides a multiplexer data whole database synchronization method, which adopts the following technical solution, including:

[0010] Configure the input structure for each source table in the entire database, configure the multiplexer in the Flink distributed computing framework, and add the input structure corresponding to each source table to the multiplexer;

[0011] Generate a shard set using the multiplexer, and construct a shard distributor according to the shard set, wherein the shard set includes all shards of a plurality of source tables;

[0012] constructing a source data operator according to the multiplexer, obtaining an execution task graph using the source data operator, and transmitting the execution task graph to a management node;

[0013] When a computing node requests a shard from the management node, the management node requests the computing node to allocate a resource slot according to the execution task graph, and uses the shard allocator to allocate the shard to multiple resource slots in the computing node for calculation, and synchronizes multiple source tables to the data lake.

[0014] Furthermore, configuring an input structure for each source table in the entire database, configuring a multiplexer in the Flink distributed computing framework, and adding the input structure corresponding to each source table to the multiplexer includes:

[0015] Read the entire database configuration information;

[0016] Obtain the source table information of the entire database through catalog table identification;

[0017] Configuring the multiplexer at the input front end of the Flink distributed computing framework;

[0018] The multiplexer is used to centrally shard multiple source tables input into the Flink distributed computing framework.

[0019] Further, the generating a shard set by using the multiplexer and constructing a shard allocator according to the shard set includes:

[0020] Using the multiplexer to slice the input single source table to generate a sub-slice set;

[0021] Widening the sub-shard set to obtain a large shard set;

[0022] Aggregate the large slice sets of all source tables to obtain the shard sets;

[0023] The shard allocator is constructed using the shard set.

[0024] Further, the obtaining the execution task graph by using the source data operator includes:

[0025] The source data operator obtains sub-shards corresponding to multiple source tables from the shard set, and obtains subtasks using the sub-shards corresponding to the multiple source tables;

[0026] An execution task graph is constructed using the obtained multiple subtasks.

[0027] Further, the using the shard distributor to distribute the shards to multiple resource slots in the computing node for computing and synchronizing multiple source tables to the data lake includes:

[0028] The resource slots are connected to the databases of the source tables corresponding to the sub-shards in the sub-tasks in turn;

[0029] Perform SQL query on the source table to obtain data information of the sub-shard;

[0030] Convert the data information of the sub-shard into a row data structure inside the Flink distributed computing framework, and output a sub-table;

[0031] According to the field name of the source table, the sub-shard data is written to the corresponding data stream, and multiple source tables are synchronized to the data lake.

[0032] Further, the using the shard distributor to distribute the shards to multiple resource slots in the computing node for computing and synchronizing multiple source tables to the data lake includes:

[0033] The resource slot simultaneously and in parallel establishes a connection with the database of the source table corresponding to the sub-shard in the sub-task;

[0034] Perform SQL queries on the source tables in parallel to obtain data information of corresponding sub-shards;

[0035] Convert the data information of each sub-shard into a row data structure inside the Flink distributed computing framework, and output the sub-tables respectively;

[0036] According to the field name of the source table, the sub-shard data is written to the corresponding data stream, and multiple source tables are synchronized to the data lake.

[0037] In order to solve the above problems, the embodiment of the present application further proposes a multiplexer-based data whole library synchronization device, which adopts the multiplexer data whole library synchronization method, including:

[0038] A configuration module is used to configure the input structure for each source table in the entire database, configure a multiplexer in the Flink distributed computing framework, and add the input structure corresponding to each source table to the multiplexer;

[0039] A generation module, configured to generate a shard set using the multiplexer, the shard set including all shards of a plurality of source tables, and construct a shard distributor according to the shard set;

[0040] An acquisition module, configured to construct a source data operator according to the multiplexer, acquire an execution task graph using the source data operator, and transmit the execution task graph to a management node;

[0041] A synchronization module is used for, when a computing node requests a shard from the management node, the management node requests the computing node to allocate a resource slot according to the execution task graph, and uses the shard allocator to allocate the shard to multiple resource slots in the computing node for calculation, and synchronizes multiple source tables to the data lake.

[0042] Furthermore, the synchronization module includes:

[0043] A connection unit, used for the resource slot to establish a connection with the database of the source table corresponding to the sub-shard in the sub-task in sequence;

[0044] A query unit, used to perform an SQL query on the source table to obtain data information of the sub-shard;

[0045] A conversion unit, configured to convert the data information of the sub-shard into a row data structure inside the Flink distributed computing framework, and output a sub-table;

[0046] The synchronization unit is used to write sub-shard data into the corresponding data stream according to the field name of the source table and synchronize multiple source tables to the data lake.

[0047] In order to solve the above problems, an embodiment of the present application also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a multiplexer data library synchronization method.

[0048] In order to solve the above problems, an embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of a multiplexer data library synchronization method are implemented.

[0049] Compared with the existing technology, by configuring a multiplexer at the input front end of the Flink distributed computing framework, and using the multiplexer module to generate a shard set including all shards of multiple source tables, construct a shard distributor and source data operator, one subtask can complete the storage and incremental data synchronization of sub-shard data information of multiple source tables and multiple topics to the lake table, greatly reducing the workload of table task launch. The shard set generated by the multiplexer module uniformly manages and allocates sub-shard resources of different tables, and then the same resource slot can be used to execute the sharding logic of different tables, and slot computing resources are reused. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a multiplexer-based data library synchronization method according to the present application;

[0052] Figure 2 yes Figure 1 A flowchart of a specific implementation of S100;

[0053] Figure 3 yes Figure 1 A flowchart of a specific implementation of S200;

[0054] Figure 4 yes Figure 1 A flowchart of a specific implementation of S300;

[0055] Figure 5 yes Figure 1 Flow chart of a first specific implementation method of S400;

[0056] Figure 6 yes Figure 1 Flow chart of a second specific implementation manner of S400;

[0057] Figure 7 It is a schematic diagram of the framework of a multiplexer-based data library synchronization device of the present application;

[0058] Figure 8 A schematic diagram of the module structure of a multiplexer-based data library synchronization device according to the present application;

[0059] Fig. 9A schematic diagram of the module structure of a computer device according to the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0063] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0064] The present application embodiment provides a data whole database synchronization method based on a multiplexer, such as Figure 1 , Figure 1 A flowchart of a multiplexer-based data whole database synchronization method according to the present application includes:

[0065] S100. The client configures the input structure for each source table in the entire database, configures the multiplexer (Multiplexing Input Format) in the Flink distributed computing framework, and adds the input structure corresponding to each source table to the multiplexer;

[0066] The multiplexer-based data whole database synchronization method in the embodiment of the present application runs on a computer device, and the communication module of the computer device can adopt a wired connection method or a wireless connection method.

[0067] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future.

[0068] In this embodiment, the source table is a data table to be processed by the client. Its input structure (InputFormat) defines the input format of the data and how to divide the data into multiple logical units for subsequent data processing. Its main function is to read data from the source table, divide the input data, usually called input splits, and assign corresponding tasks to the divided input splits. The Flink distributed computing framework is an open source stream processing framework and distributed processing engine, which processes data in the form of streams, assigns tasks to different nodes for execution, and manages the status through the backend.

[0069] In some optional embodiments, such as Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation method of S100 in the embodiment; S100 includes: S110, reading the configuration information of the entire database; S120, obtaining the source table information of the entire database through directory table identification; S130, configuring the multiplexer at the input front end of the Flink distributed computing framework; S140, using the multiplexer to centrally shard multiple source tables input to the Flink distributed computing framework.

[0070] In this embodiment, the source table usually includes field names, data types, and data records. For example, customer information may include data items such as name, age, and address. The source table also includes primary key information, that is, a field that uniquely identifies each record, and some also include index information, etc. The client reads the entire database configuration information, which includes JDBC connection string information, account number and password information, etc. Among them, the JDBC (Java Database Connectivity) connection string information is used to establish a connection string between a Java application and a database, including a database driver name, a database connection URL, a user name and a password, etc.

[0071] Get the source table information of the entire database through the catalog table, set the input structure for each table, and prepare for efficient execution of sharding logic.

[0072] In this embodiment, a multiplexer module is introduced to replace the data stream (DataStream) and the integration tool (Union) mode in the prior art, and the input structure information of each source table is added to the multiplexer module. By configuring the multiplexer at the input end of the model, the input source table information can be processed in parallel at the same time, thereby improving work efficiency and saving resources.

[0073] S200: Generate a shard set using a multiplexer, and construct a shard distributor according to the shard set, where the shard set includes all shards of multiple source tables.

[0074] In some optional embodiments, such as Figure 3 , Figure 3 yes Figure 1 A flowchart of a specific implementation method of S200; S200 includes: S210, using a multiplexer to shard an input single source table to generate a sub-shard set; S220, widening the sub-shard set to obtain a large shard set; S230, summarizing the large shard sets of all source tables to obtain a shard set; S240, constructing a shard distributor using the shard set.

[0075] Flat map processing is used to flatten the input list or set, so as to output a single iterable data structure. In this embodiment, through the flat map processing, multiple single-table shard sets contained in the sub-shard set are constructed to obtain a larger shard set, that is, a large shard set, which contains all shards of multiple tables.

[0076] In this embodiment, when sharding, you can select the time field index as the sharding field and map it to a long integer value type. According to the custom sharding strategy, slice the time range. For example, if the minimum and maximum values ​​of the time field are 3 years apart, and the custom shard size is 1 hour / shard, the total number of shards is 24 / 1*365*3=26280. After adjusting the shard size, the total number of shards will change accordingly.

[0077] When using the multiplexer module to generate a shard set, the size of the single-table shard collection it contains is widened to construct a larger shard set that contains all shards of multiple source tables, and then the shard set is used to construct a shard distributor.

[0078] The sub-shard resources of different tables are uniformly managed and allocated through shard collections, so that the sharding logic of different tables can be executed using the same resource slot, and slot computing resources can be reused.

[0079] S300, constructing a source data operator according to the multiplexer, using the source data operator to obtain an execution task graph, and transmitting the execution task graph to a management node (Job Manager).

[0080] In this embodiment, DataStreamSource is a source data operator responsible for reading data from various external data sources, such as file system network sockets and message queues, providing an initial data set for subsequent data processing and starting the entire streaming data processing task.

[0081] After receiving the task submitted by the client, the management node will schedule the task. The management node will analyze the task graph and determine which computing resources (such as CPU, memory) and which computing nodes (TaskManager) are needed to execute the task.

[0082] In some optional embodiments, such as Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation of step S300; S300 includes:

[0083] S310, the source data operator obtains sub-shards corresponding to multiple source tables from the shard set, and obtains subtasks using the sub-shards corresponding to the multiple source tables;

[0084] S320: Construct an execution task graph using the obtained multiple subtasks.

[0085] like Figure 7 , Figure 7 It is a framework schematic diagram of a data whole database synchronization device based on a multiplexer of the present application; source table A and source table B are source tables of a client database (DB), the multiplexer is configured at the input front end of the Flink distributed computing framework, and the source data operator is constructed by using the multiplexer. The sub-shards obtained from the shard set include A shard 1, A shard 2, A shard 3 and B shard 1, B shard 2, B shard 3. A shard 1, A shard 2, A shard 3 and B shard 1, B shard 2, B shard 3 are respectively combined to form multiple subtasks (Subtask1, Subtask2, Subtask3). The calculation query is executed through the sharding of the resource slot, and a unified table style <field name of the source table, field name of the data row, data row> is output to the corresponding data stream, and the source table A and the source table B are finally synchronized to the data lake (Hudi).

[0086] By using the source data operator of the multiplexer module to obtain sub-shards from the shard set and using sub-tasks to build an execution task graph, only 3 connections are required, and the resource slots in the computing nodes can be reused, greatly saving computing resources.

[0087] A source data operator is constructed through the multiplexer module. The source data operator constructs a shard distributor according to the shard set and submits the task operation diagram to the management node. When applying for computing resources of the computing node, the task only uses a small number of computing node resources to complete the task of synchronizing multiple source tables to the data lake.

[0088] S400. When a computing node requests a shard from a management node, the management node requests the computing node to allocate a resource slot according to the execution task graph, and uses the shard allocator to allocate the shard to multiple resource slots in the computing node for calculation, and synchronizes multiple source tables to the data lake.

[0089] A resource slot is a resource allocation unit. In the distributed computing in this embodiment, a resource slot represents a unit of computing resources, and includes multiple resource slots.

[0090] The computing node is responsible for performing the actual tasks, such as data processing and calculation. Each computing node has one or more resource slots, which are units for executing tasks in parallel. In this application, after the computing node receives the subtask assigned by the management node, the computing node will start to perform data synchronization operations.

[0091] In one embodiment, i.e., the stock phase embodiment, as Figure 5 , Figure 5 yes Figure 1 Flow chart of a first specific implementation of step S400; S400 includes:

[0092] S410. The resource slot establishes connections with the database of the source table corresponding to the sub-shard in the sub-task in turn; S420. Performs SQL query on the source table to obtain the data information of the sub-shard; S430. Converts the data information of the sub-shard into the row data structure inside the Flink distributed computing framework and outputs the sub-table; S440. Writes the sub-shard data into the corresponding data stream according to the field name of the source table, and synchronizes multiple source tables to the data lake.

[0093] By uniformly managing and allocating sub-shard resources of different tables, the same resource slot can be used to execute the sharding logic of different tables, and slot computing resources can be reused; by consuming the same or different Kafka topics through a subtask to process multiple source tables, the network load pressure caused by multiple consumption links can be alleviated.

[0094] In another embodiment, namely the incremental phase embodiment, as Figure 6 , Figure 6 yes Figure 1 Flow chart of a second specific implementation of S400; S400 includes:

[0095] S450. The resource slot simultaneously and in parallel establishes a connection with the database of the source table corresponding to the sub-shard in the sub-task; S460. SQL queries are performed on the source table in parallel to obtain the data information of the corresponding sub-shard; S470. The data information of each sub-shard is converted into a row data structure inside the Flink distributed computing framework, and the sub-tables are output respectively; S480. According to the field name of the source table, the sub-shard data is written into the corresponding data stream, and multiple source tables are synchronized to the data lake.

[0096] The multiplexer module encapsulates partitions of different topics into unified shard resources. The computing node requests the management node to allocate shards and execute the reading logic. After being processed by the deserializer, the unified data format is output as <field name, data row>. Different from the stock phase, due to the unbounded nature of the data source in the incremental phase, each shard in the same subtask is processed in parallel. From the perspective of network topology, the traffic starts from each topic and goes to a fixed location and a fixed number of machines. Compared with multiple synchronous links, the network load pressure is reduced.

[0097] In order to solve the above problems, Figure 8 , Figure 8 According to a module structure diagram of a multiplexer-based data whole database synchronization device of the present application; the embodiment of the present application also proposes a multiplexer-based data whole database synchronization device 500, and the multiplexer-based data whole database synchronization method adopted includes:

[0098] Configuration module 501, used by the client to configure the input structure for each source table in the entire database, configure the multiplexer in the Flink distributed computing framework, and add the input structure corresponding to each source table to the multiplexer;

[0099] A generation module 502 is used to generate a shard set using a multiplexer, where the shard set includes all shards of multiple source tables, and construct a shard distributor according to the shard set;

[0100] An acquisition module 503 is used to construct a source data operator according to the multiplexer, obtain an execution task graph using the source data operator, and transmit the execution task graph to the management node;

[0101] The synchronization module 504 is used to allocate resource slots to the computing node according to the execution task graph when the computing node requests shards from the management node, and use the shard allocator to allocate shards to multiple resource slots in the computing node for calculation, and synchronize multiple source tables to the data lake.

[0102] By uniformly managing and allocating sub-shard resources of different tables, the same resource slot can be used to execute the sharding logic of different tables, and slot computing resources can be reused; by consuming the same or different Kafka topics through a subtask to process multiple source tables, the network load pressure caused by multiple consumption links can be alleviated.

[0103] Furthermore, the synchronization module includes: a connection unit, which is used for the resource slot to establish a connection with the database of the source table corresponding to the sub-shard in the subtask in turn; a query unit, which is used to perform SQL query on the source table to obtain data information of the sub-shard; a conversion unit, which is used to convert the data information of the sub-shard into a row data structure inside the Flink distributed computing framework and output the sub-table; a synchronization unit, which is used to write the sub-shard data into the corresponding data stream according to the field name of the source table, and synchronize multiple source tables to the data lake.

[0104] By uniformly managing and allocating sub-shard resources of different tables, the same resource slot can be used to execute the sharding logic of different tables, and slot computing resources can be reused.

[0105] In order to solve the above problems, an embodiment of the present application also proposes a computer device, which adopts the following technical solution: it includes a memory, a communication module and a processor, the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a multiplexer data library synchronization method.

[0106] The computer device may be a computer, server, workstation or other device, or a mobile phone, tablet, vehicle-mounted mobile terminal or other device with program execution capability. The internal structure diagram of the computer device may be as follows: Fig. 9 As shown, Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. The computer device includes a processor, a memory and a communication module. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, the functions or steps of a method for fine-tuning the above-mentioned pre-trained model are implemented. The communication module of the computer device may include a network interface and / or a wireless communication module, and the computer device may communicate with other devices or service platforms through the communication module. In addition, the computer device may also include a display screen and an input device, etc.

[0107] Among them, the memory is used to store computer programs, the computer programs include program instructions, the processor is configured to call program instructions, and the processor implements the following steps when executing instructions or codes: S100, the client configures the input structure for each source table in the entire library, configures the multiplexer in the Flink distributed computing framework, and adds the input structure corresponding to each source table to the multiplexer; S200, using the multiplexer to generate a shard set, and constructing a shard allocator based on the shard set, the shard set includes all shards of multiple source tables; S300, constructing a source data operator based on the multiplexer, using the source data operator to obtain an execution task graph, and transmitting the execution task graph to the management node; S400, when the computing node requests a shard from the management node, the management node requests the computing node to allocate a resource slot based on the execution task graph, and uses the shard allocator to allocate the shards to multiple resource slots in the computing node for calculation, and synchronizes multiple source tables to the data lake.

[0108] In order to solve the above problems, the embodiment of the present application further proposes a computer-readable storage medium, which adopts the following technical solution:

[0109] The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes an adaptive parameter efficient fine-tuning method.

[0110] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figures 1 to 6 A multiplexer-based data library synchronization method is provided in each step. For details, please refer to the implementation methods provided in the above steps, which will not be repeated here.

[0111] The computer-readable storage medium may be a multiplexer-based data library synchronization device provided in any of the aforementioned embodiments, or an internal storage unit of the terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device.

[0112] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0113] However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0114] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0115] Compared with the existing technology, by configuring the multiplexer (Multiplexing Input Format) module at the input front end of the Flink distributed computing framework, and using the multiplexer module to generate a shard set including all shards of multiple source tables, construct a shard distributor and a source data operator (Data Stream Source), one subtask can complete the storage and incremental data synchronization of sub-shard data information of multiple source tables and multiple topics to the lake table, greatly reducing the workload of table task online. By uniformly managing and allocating sub-shard resources of different tables, the same resource slot can be used to execute the sharding logic of different tables, and slot computing resources can be reused; by consuming the same or different kafkatopics through a subtask to process multiple source tables, the network load pressure caused by multiple consumption links is alleviated.

[0116] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0117] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multiplexer-based data whole database synchronization method, characterized in that it includes: Configure an input structure for each source table in the entire database, configure a multiplexer in the Flink distributed computing framework, and add the input structure to the multiplexer; Generate a shard set using the multiplexer, and construct a shard distributor according to the shard set, wherein the shard set includes all shards of the plurality of source tables; constructing a source data operator according to the multiplexer, obtaining an execution task graph using the source data operator, and transmitting the execution task graph to a management node; When a computing node requests a shard from the management node, the management node requests the computing node to allocate a resource slot according to the execution task graph, and uses the shard allocator to allocate the shard to multiple resource slots in the computing node for calculation, and synchronizes multiple source tables to the data lake.

2. The method according to claim 1, characterized in that The step of configuring an input structure for each source table in the entire database, configuring a multiplexer in the Flink distributed computing framework, and adding the input structure corresponding to each source table to the multiplexer includes: Read the entire database configuration information; Acquire source table information of the entire database through catalog table identification; Configuring the multiplexer at the input front end of the Flink distributed computing framework; The multiplexer is used to centrally shard multiple source tables input into the Flink distributed computing framework.

3. The method according to claim 1, characterized in that The step of generating a shard set by using the multiplexer and constructing a shard distributor according to the shard set includes: Using the multiplexer to slice the input single source table to generate a sub-slice set; Widening the sub-shard set to obtain a large shard set; Aggregate the large slice sets of all source tables to obtain the shard sets; The shard allocator is constructed using the shard set.

4. The method according to claim 3, characterized in that The obtaining the execution task graph by using the source data operator includes: The source data operator obtains sub-shards corresponding to multiple source tables from the shard set, and obtains subtasks using the sub-shards corresponding to the multiple source tables; An execution task graph is constructed using the obtained multiple subtasks.

5. The method according to claim 4, characterized in that The using the shard distributor to distribute the shards to multiple resource slots in the computing node for computing, and synchronizing multiple source tables to the data lake, includes: The resource slots are connected to the databases of the source tables corresponding to the sub-shards in the sub-tasks in turn; Perform SQL query on the source table to obtain data information of the sub-shard; Convert the data information of the sub-shard into a row data structure inside the Flink distributed computing framework, and output a sub-table; According to the field name of the source table, the sub-shard data is written to the corresponding data stream, and multiple source tables are synchronized to the data lake.

6. The method according to claim 4, characterized in that The using the shard distributor to distribute the shards to multiple resource slots in the computing node for computing, and synchronizing multiple source tables to the data lake, includes: The resource slot simultaneously and in parallel establishes a connection with the database of the source table corresponding to the sub-shard in the sub-task; Perform SQL queries on the source tables in parallel to obtain data information of corresponding sub-shards; Convert the data information of each sub-shard into a row data structure inside the Flink distributed computing framework, and output the sub-tables respectively; According to the field name of the source table, the sub-shard data is written to the corresponding data stream, and multiple source tables are synchronized to the data lake.

7. A multiplexer-based data whole database synchronization device, using the multiplexer-based data whole database synchronization method according to any one of claims 1 to 6, characterized in that: include: A configuration module is used to configure the input structure for each source table in the entire database, configure a multiplexer in the Flink distributed computing framework, and add the input structure corresponding to each source table to the multiplexer; A generation module, configured to generate a shard set using the multiplexer, the shard set including all shards of a plurality of source tables, and construct a shard distributor according to the shard set; An acquisition module, configured to construct a source data operator according to the multiplexer, acquire an execution task graph using the source data operator, and transmit the execution task graph to a management node; A synchronization module is used for, when a computing node requests a shard from the management node, the management node requests the computing node to allocate a resource slot according to the execution task graph, and uses the shard allocator to allocate the shard to multiple resource slots in the computing node for calculation, and synchronizes multiple source tables to the data lake.

8. The multiplexer-based data whole database synchronization device according to claim 7, characterized in that: The synchronization module comprises: A connection unit, used for the resource slot to establish connections with the database of the source table corresponding to the sub-shard in the sub-task in sequence; A query unit, used to perform an SQL query on the source table to obtain data information of the sub-shard; A conversion unit, configured to convert the data information of the sub-shard into a row data structure inside the Flink distributed computing framework, and output a sub-table; The synchronization unit is used to write sub-shard data into the corresponding data stream according to the field name of the source table and synchronize multiple source tables to the data lake.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the steps of the multiplexer-based data library synchronization method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the multiplexer-based data library synchronization method as described in any one of claims 1 to 6 are implemented.

Images