Data lake multi-table real-time synchronization method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of data processing, and in particular to a method, device, computer equipment and storage medium for real-time synchronization of multiple tables in a data lake. The method comprises: when receiving an upstream table information change request, blocking the input and output data streams of the data lake; performing data cleaning on the cached data streams, and updating the table information that needs to be changed in the storage space according to the table information change request; after the table information update is completed, restoring the blocked input and output data streams. This allows the data lake to be hot-updated online without restarting the task to support the frequent table information update requirements of the upstream.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, computer equipment and storage medium for real-time synchronization of multiple tables in a data lake. Background Art

[0002] Hudi (data lake) multi-table real-time synchronization is widely used in the industry. Schema (table information) change processing is a very common user scenario in upstream databases and is also a difficult point in the implementation of the data synchronization framework.

[0003] Among them, for Hudi's multi-table synchronization link, frequent schema changes of upstream data tables cannot support dynamic adaptation. Currently, the industry's implementation method is mostly to rebuild the latest schema information by restarting tasks. This means that every change in DDL (data definition language) requires restarting the Flink link to rebuild new schema information, which affects incremental consumption. Summary of the invention

[0004] In a first aspect, an embodiment of the present application proposes a method for real-time synchronization of multiple tables in a data lake, which can enable a data lake to be quickly updated when the data structure is updated, and is characterized by comprising:

[0005] When receiving a table information change request from upstream, the input and output data flow of the data lake is blocked;

[0006] Clean up the cached data stream, and update the table information that needs to be changed in the storage space according to the table information change request;

[0007] After the table information is updated, the blocked input and output data flows are restored.

[0008] Furthermore, before receiving the upstream table information change request, the method further includes:

[0009] The Kafka data source operator listens to the message topic. When the message topic receives table information synchronization data from the upstream, the Kafka data source operator deserializes and constructs the corresponding table information change request according to the message topic, and sends each table information change request to the downstream task node.

[0010] Furthermore, the data cleaning of the cached data stream and updating the table information that needs to be changed in the storage space according to the table information change request include:

[0011] According to the table information change request, determine all tables that need to be updated in the storage space and the storage location of each of the tables;

[0012] Delete the historical data in the cache;

[0013] The storage location of each table that needs to be updated is determined, and an update operation is performed on the table that needs to be updated in the storage location.

[0014] Furthermore, each of the tables is stored in the storage space of the event conversion operator and the storage space of the table in the data lake;

[0015] The updating operation on the table to be updated in the storage location includes:

[0016] The event conversion operator updates the relevant table structure in the storage space according to the table information change request, and the event conversion operator constructs an update data processing object according to the updated table;

[0017] Calling the write executor to execute the operation of deleting the historical data in the cache after receiving the update data processing object, and sending a message of the cleaning completion to the table information change coordinator;

[0018] After receiving the cleaning completion message, the table information change coordinator updates the table information stored in the data lake according to the table information change request.

[0019] Further, the restoring the blocked input and output data flows includes:

[0020] After the table information change coordinator updates the table structure cached in the storage space, it sends update completion information;

[0021] The update completion information is monitored by a message processing operator, and upon receiving the update completion information, the input and output data flows of the data lake are opened.

[0022] Furthermore, when receiving an upstream table information change request, the method further includes:

[0023] The data cleanup message and the table update message are evenly distributed to all downstream devices in a broadcast manner through the pre-partition operator to update the table information of all the downstream devices.

[0024] Furthermore, the even distribution to all downstream task nodes includes:

[0025] The data table is partitioned according to the bucket primary key, multiple information distribution areas are determined, and data cleanup messages and table update messages are evenly distributed to each information partition.

[0026] In a second aspect, the present application also provides a data lake multi-table real-time synchronization device, including:

[0027] The information processing module is used to block the input and output data flows of the data lake when receiving a table information change request from upstream;

[0028] A table information update module is used to clean up the cached data stream data and update the table information that needs to be changed in the storage space according to the table information change request;

[0029] The information processing module is also used to restore the blocked data flow after the table information is updated.

[0030] In a third aspect, the present application provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program runs on the processor, the data lake multi-table real-time synchronization method is executed.

[0031] In a fourth aspect, the present application provides a readable storage medium storing a computer program, which executes the data lake multi-table real-time synchronization method when running on a processor.

[0032] The present invention discloses a data lake multi-table real-time synchronization method, device, computer equipment and storage medium. The method comprises: when receiving an upstream table information change request, blocking the data flow; performing data cleaning, and updating the table that needs to be changed in the storage control according to the table information change request; after the table information is updated, restoring the blocked data flow. The data lake can be hot-updated online without restarting the task to support the frequent table information update requirements of the upstream. And the data lake can update multiple table information at the same time to realize the operation of synchronous update of multiple tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0034] Figure 1 A schematic diagram of a method for real-time synchronization of multiple tables in a data lake according to an embodiment of the present application is shown;

[0035] Figure 2 A data flow diagram of a data lake multi-table real-time synchronization method according to an embodiment of the present application is shown;

[0036] Figure 3A schematic diagram of a data lake multi-table real-time synchronization data flow according to an embodiment of the present application is shown;

[0037] Figure 4 A schematic diagram of the structure of a data lake multi-table real-time synchronization device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0039] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0040] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0041] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0042] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0043] The technical solution of the present application is applied to the table information update scenario in the data lake, especially the synchronous update of multiple tables. The present application performs corresponding synchronous update operations upon receiving an update request, locates the location of the table that needs to be updated, and synchronously updates each table that needs to be updated at the location that needs to be updated, so that the table information in the data lake can be hot-updated without restarting. Through this hot update, frequent online table information update requests can be supported.

[0044] Next, the technical solution of the present application is described with specific embodiments.

[0045] Example 1

[0046] The technical solution of this embodiment targets data structure update events of the hoodie data lake and is mainly applied to the hoodie data lake itself.

[0047] like Figure 1 As shown, the data lake multi-table real-time synchronization method of this embodiment includes:

[0048] Step S100: When receiving an upstream table information change request, the input and output data flows of the data lake are blocked.

[0049] The technical solution of this embodiment is applied to the table update operation of the data lake, and is mainly used for the table update operation of the hoodie data lake. The table information update demand is generally generated by the upstream device sending a table information change request to the data lake. For example, the upstream database table structure changes, thus affecting the downstream data lake to change synchronously. At this time, when the data lake receives the request, it will block the data flow, because the data flow of the input data lake and the output data lake are all processed according to the structural information of each data table currently stored. If the data flow is not blocked and the table information is updated directly, it will cause data confusion, so it is necessary to block the data flow.

[0050] Among them, the data lake will have a lot of interactions with upstream devices. Therefore, in this embodiment, a Kafka data source operator can be set up in the middle to manage messages of various topics. The operator is a message queue. That is, by specifying the consumption of multiple message topics for the Kafka data source operator, and then deserializing the table data update information received according to the message topic, multiple event messages required by the data lake are constructed, and then the event data stream is sent to the data lake. The above-mentioned table information change request is one of the messages.

[0051] The data lake will also have a dedicated message processing operator to process the event data in the data stream. When the message processing operator receives a request to change table information, it will automatically block the data stream first, and then wait for the table information update completion signal. It should be noted that the message processing operator here not only processes table update events, but also performs corresponding processing based on the messages received by the data lake. If the received message does not need to block the data stream, no blocking operation will be performed.

[0052] Among them, the data lake itself will involve multiple downstream task nodes, which can be various downstream devices or downstream databases. It can be understood that the tables managed by the data lake itself will also be distributed to downstream devices. These downstream devices use these tables for data processing. When the table structure in the data lake changes, these downstream task nodes are also required to uniformly process the table information and complete the unified modification of all tables.

[0053] Therefore, optionally, the data cleaning message and the table update message can be evenly distributed to all downstream task nodes in a broadcast manner through the pre-partitioning operator. This allows each downstream task node to efficiently and in parallel process tasks. At the same time, this embodiment involves the update of table information, so the data table can be partitioned according to the primary key of the bucket, the downstream device can be partitioned, and then the data cleaning message and the table update message can be distributed to each partition to ensure uniform message distribution. This ensures that all data tables can be updated in a timely manner. That is, the fields of the data table are bucketed, and the partitions are performed according to the primary key after bucketing to achieve the partitioning operation.

[0054] Among them, the downstream task node is a specific task running node, which can be a running container or a processing device of the downstream database, etc. In general, it is a parallel calculation of multiple task running nodes. Combined with the above distribution method, it makes it possible to modify multiple tables in parallel.

[0055] Step S200, clearing the cached data stream, and updating the table information that needs to be changed in the storage space according to the table information change request;

[0056] In order to update the table data, a data cleanup operation is also performed. The message of the data cleanup operation can be sent by the aforementioned message processing operator to the downstream processing operator, and the table information change request will also be sent to the downstream task node for processing.

[0057] In this embodiment, the update of table information is actually the update of table structure, such as adding data attributes to the table, changing comments, etc. These updates will change the properties of a data table and also affect the data stored in the table.

[0058] For example, a class roster table initially only has names and student numbers, but later on, the scores of each subject need to be added, so the table needs to be expanded, that is, data attributes are added. At this time, the class roster table needs to be updated. Similarly, the student number column may be cancelled, thereby reducing the data attributes, and the table also needs to be updated.

[0059] These specific update requirements will be included in the table information change request, and the corresponding update operations can be performed based on the table information change request.

[0060] Specifically, Figure 2 As shown, the update operation of this embodiment includes:

[0061] Step S210: determining all tables that need to be updated in the storage space and the storage location of each of the tables according to the table information change request.

[0062] In addition to being stored inside and outside the data lake, tables in the data lake are also cached in the memory opened up by some functional modules because they are called. These table structures also need to be updated synchronously. Therefore, before updating, you need to first determine which tables need to be updated and the storage location of these tables that need to be updated, so that they can be updated correctly.

[0063] In this embodiment, the storage locations of the tables include: tables maintained by the event conversion operator and tables stored in the data lake.

[0064] The event conversion operator is an operator used to convert event data objects consumed from Kafka into data objects written into the data lake. The operator needs to maintain the table structure internally to assist in the conversion of the above data objects. It is understandable that external data will not conform to the form of the data table structure in the data lake, so such an operator is needed for preprocessing to form data that can be read in the data lake. Therefore, when a table is updated, the table maintained by the event conversion operator needs to be updated so that the data lake can understand the external data.

[0065] The second is the table stored in the data lake, which can be reconstructed and updated through the table information change coordinator.

[0066] Step S220, deleting the historical data in the cache.

[0067] The cached historical data is the data flow data in the memory, that is, the data that flows into the data lake before the data flow is blocked, but has not yet been written to the data lake. It is just some cached data in a waiting state. This data is cached according to the rules of the old table and is not applicable to the updated table, so it needs to be deleted.

[0068] For the event conversion operator, the memory space allocated to it mainly contains table information that needs to be maintained. This information will be processed when it is updated, so it will not be taken out separately for a cleanup operation. External data or sent data will not be cached inside the event conversion operator, so the event conversion operator does not need to clear historical data.

[0069] As for the data lake, it will cache multiple historical data, so data cleaning is required.

[0070] Step S230, determining the storage location of each table that needs to be updated, and performing an update operation on the table that needs to be updated in the storage location.

[0071] Among them, because the event conversion operator itself processes the messages in the Kafka message queue, the event conversion operator is the first to receive the message. When it receives the message, it will directly start to update the table. After the update is completed, the received message is converted into a new data processing object through the updated table. Through the data processing object, it is sent to the write executor to complete the sending of table update messages and data cleanup messages to the data lake.

[0072] The data processing object includes messages for updating the table and cleaning the data. After receiving the data processing object, the data lake actually receives the messages for updating the table and cleaning the data.

[0073] After the data lake receives the message, it will flush the historical data, which can clear the historical data in the cache.

[0074] After the historical cache data is flushed, the table information change coordinator can perform table update operations. In this way, all table data is updated online without the need for complex operations such as restart or reset.

[0075] Specifically, the table information change coordinator executes the update of the corresponding table information in the data lake.

[0076] like Figure 3 As shown in the figure, it is the transmission order of the entire table information change request. It can be seen that the message is processed by the Kafka message queue and transmitted to the data lake. At this time, it is first received and processed by the event conversion operator. When the event conversion operator finds the table information change request, it will first update it. After updating the table it maintains to the latest version, it will convert the received message into a data object in a new format through the latest table. The data object carries the message information and enters the data lake. Then the table information change coordinator performs the corresponding update operation to update the data lake.

[0077] It should be noted that if the internal system structure of the data lake is upgraded and iterated, other operators may need to maintain table information, so the tables maintained by these other operators will also need to be modified. In this way, the above table information change message will also be transmitted to the corresponding operator, and the corresponding operator will modify it to update all the tables in the memory that are currently running. After the update is completed, the data lake can complete the update of the table information without restarting the task, and it can also ensure that the functions of the data lake can be continuously applied online.

[0078] Step S300: After the table information is updated, the blocked data flow is restored.

[0079] After the table information is updated, you can know that the table structure of the data lake is now in a new version state, and you can now receive and process data streams.

[0080] According to the above steps, the update is finally completed by the table information change coordinator, so the table information change coordinator will also send an update completion message, and then the data flow blockage can be unblocked and the data lake can return to normal use.

[0081] It can be understood that if there are other operators that need to update the table, they need to wait until all the operator dogs have updated the tables they maintain before sending the corresponding update completion message, thus completing the entire update process.

[0082] For a table update event from the upstream, in addition to updating the table information in the data lake, it is also necessary to update the table information stored in multiple downstream devices or the downstream data lake. Therefore, this embodiment will also broadcast messages to ensure that all downstream devices can receive data update messages to ensure that the table update operation can be applied to all downstream devices.

[0083] The data lake multi-table real-time synchronization method of this embodiment determines the location of each table that needs to be updated and performs targeted online updates to achieve simultaneous updates of multiple tables. It also makes it possible for the data lake to update the information of each table stored in itself without restarting the task, thereby achieving seamless hot update operations, reducing the workload required to update table information, and better adapting to frequent table update requests from upstream.

[0084] Example 2

[0085] like Figure 4 As shown, this embodiment also provides a data lake multi-table real-time synchronization device, including:

[0086] The information processing module 10 is used to block the input and output data flows of the data lake when receiving an upstream table information change request;

[0087] The table information update module 20 is used to clean up the cached data stream data and update the table information that needs to be changed in the storage space according to the table information change request;

[0088] The information processing module 10 is also used to restore the blocked data flow after the table information is updated.

[0089] The present application provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is run on the processor, the data lake multi-table real-time synchronization method is executed.

[0090] The present application provides a readable storage medium storing a computer program, which executes the data lake multi-table real-time synchronization method when running on a processor.

[0091] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow chart, and the combination of boxes in the structure diagram and / or the flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] In addition, the functional modules or units in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0093] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0094] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for real-time synchronization of multiple tables in a data lake, characterized in that: include: When receiving a table information change request from upstream, the input and output data flow of the data lake is blocked; Clean up the cached data stream, and update the table information that needs to be changed in the storage space according to the table information change request; After the table information is updated, the blocked input and output data flows are restored; The data cleaning of the cached data stream and updating the table information that needs to be changed in the storage space according to the table information change request include: According to the table information change request, determine all tables that need to be updated in the storage space and the storage location of each of the tables; Delete the historical data in the cache; Determine the storage location of each table that needs to be updated, and perform an update operation on the table that needs to be updated in the storage location; Each of the tables is stored in the storage space of the event conversion operator and the storage space of the table in the data lake; The updating operation on the table to be updated in the storage location includes: The event conversion operator updates the relevant table structure in the storage space according to the table information change request, and the event conversion operator constructs an update data processing object according to the updated table; Calling the write executor to execute the operation of deleting the historical data in the cache after receiving the update data processing object, and sending a message of the cleaning completion to the table information change coordinator; After receiving the cleaning completion message, the table information change coordinator updates the table information stored in the data lake according to the table information change request.

2. The method for real-time synchronization of multiple tables in a data lake according to claim 1 is characterized in that: Before receiving the upstream table information change request, it also includes: The Kafka data source operator listens to the message topic. When the message topic receives table information synchronization data from the upstream, the Kafka data source operator deserializes and constructs the corresponding table information change request according to the message topic, and sends each table information change request to the downstream task node.

3. The method for real-time synchronization of multiple tables in a data lake according to claim 1 is characterized in that: The method of restoring the blocked input and output data streams comprises: After the table information change coordinator updates the table structure cached in the storage space, it sends update completion information; The update completion information is monitored by a message processing operator, and upon receiving the update completion information, the input and output data flows of the data lake are opened.

4. The method for real-time synchronization of multiple tables in a data lake according to claim 1, characterized in that: When receiving an upstream table information change request, the method further includes: The data cleanup message and the table update message are evenly distributed to all downstream devices in a broadcast manner through the pre-partition operator to update the table information of all the downstream devices.

5. The method for real-time synchronization of multiple tables in a data lake according to claim 4 is characterized in that: The uniform distribution to all downstream task nodes includes: The data table is partitioned according to the bucket primary key, a plurality of information distribution areas are determined, and the data cleaning message and the table update message are evenly distributed to each information partition.

6. A data lake multi-table real-time synchronization device, characterized in that: include: The information processing module is used to block the input and output data flows of the data lake when receiving a table information change request from upstream; A table information update module is used to clean up the cached data stream data and update the table information that needs to be changed in the storage space according to the table information change request; The information processing module is also used to restore the blocked data flow after the table information is updated; The data cleaning of the cached data stream and updating the table information that needs to be changed in the storage space according to the table information change request include: According to the table information change request, determine all tables that need to be updated in the storage space and the storage location of each of the tables; Delete the historical data in the cache; Determine the storage location of each table that needs to be updated, and perform an update operation on the table that needs to be updated in the storage location; Each of the tables is stored in the storage space of the event conversion operator and the storage space of the table in the data lake; The updating operation on the table to be updated in the storage location includes: The event conversion operator updates the relevant table structure in the storage space according to the table information change request, and the event conversion operator constructs an update data processing object according to the updated table; Calling the write executor to execute the operation of deleting the historical data in the cache after receiving the update data processing object, and sending a message of the cleaning completion to the table information change coordinator; After receiving the cleaning completion message, the table information change coordinator updates the table information stored in the data lake according to the table information change request.

7. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is run on the processor, the method for real-time synchronization of multiple tables in a data lake is executed as claimed in any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The computer program stores a computer program, which, when running on a processor, executes the data lake multi-table real-time synchronization method according to any one of claims 1 to 5.

Images