Multi-stream merging method and device based on database logs, computer equipment and storage medium

Abstract

The invention relates to a multi-stream merging method and device based on database logs, computer equipment and a storage medium. The method comprises the following steps: executing a plurality of change information acquisition tasks; the change information collection task is used for capturing a target data change event according to the target incremental log file and converting change information of the captured data change event into a data stream; according to a specified merging key, merging the data of the data streams constructed based on the plurality of change information collection tasks to obtain wide table data; and storing the wide table data into a specified storage system. According to the method, the data of the multiple data streams can be merged in real time based on the incremental log file of the database, and the accuracy and consistency of the merged data are ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for merging multiple streams based on database logs. Background Art

[0002] In order to meet business needs, some businesses need to merge data from multiple data sources into a wide table. With the rapid growth of data volume and the increasing demand for real-time data analysis, how to improve data accuracy and consistency when merging multi-source data has become a solution that needs to be solved. Summary of the invention

[0003] In response to the above-mentioned deficiencies or shortcomings, the present application provides a multi-stream merging method, apparatus, computer equipment and storage medium based on database logs. The embodiments of the present application can merge data of multiple data streams in real time based on the incremental log files of the database, and ensure the accuracy and consistency of the merged data.

[0004] The present application provides a multi-stream merging method based on database logs according to a first aspect. In some embodiments, the method includes:

[0005] Execute multiple change information collection tasks; the change information collection task is used to capture target data change events according to the target incremental log file, and convert the change information of the captured data change events into a data stream;

[0006] Merge the data of the data streams constructed based on the multiple change information collection tasks according to the specified merge key to obtain wide table data;

[0007] Store wide table data into the specified storage system.

[0008] In some embodiments, capturing a target data change event according to a target incremental log file includes:

[0009] Determine a data source specified by the task based on the task information;

[0010] Obtaining the incremental log file related to the data source as the target incremental log file;

[0011] The data change events related to the data source are captured according to the target incremental log file as target data change events.

[0012] In some embodiments, capturing a data change event related to the data source as a target data change event according to the target incremental log file includes:

[0013] Parse the target incremental log file and extract data change events related to the data source from the parsed log information.

[0014] In some embodiments, when parsing the target incremental log file, the method further includes:

[0015] Dynamically discard specified fields or specified database operation statements according to preset configuration information.

[0016] In some embodiments, the specified merge key is a key determined according to business requirements; merging the data of the data stream constructed based on the multiple change information collection tasks according to the specified merge key includes:

[0017] Align the data of each data stream according to the timestamp and operation order of the data of each data stream;

[0018] In the storage layer, data from multiple data streams are merged based on the specified merge key.

[0019] In some embodiments, before executing the change information collection task, the method further includes:

[0020] Extract merge-related operation information from the target incremental log file; the operation information includes library information, table information, field names, and content before and after data changes;

[0021] Determine the wide table structure based on the operation information;

[0022] A wide table is constructed in the storage system based on the wide table structure.

[0023] In some embodiments, the method also includes: in the process of merging data from multiple data streams, using a message confirmation mechanism to implement an error handling and recovery mechanism, and when the message is confirmed to be executed successfully and has been confirmed, executing the next message, otherwise executing retry logic until the message is executed successfully.

[0024] According to a second aspect, the present application provides a multi-stream merging device based on a database log. In some embodiments, the device includes:

[0025] A task execution module is used to execute multiple change information collection tasks; the change information collection task is used to capture target data change events according to the target incremental log file, and convert the change information of the captured data change events into a data stream;

[0026] A multi-stream merging module is used to merge the data of the data streams constructed based on the multiple change information collection tasks according to a specified merging key to obtain wide table data;

[0027] The storage module is used to store wide table data into a specified storage system.

[0028] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-stream merging method based on database logs provided in any of the above embodiments are implemented.

[0029] According to a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-stream merging method based on database logs provided in any of the above embodiments are implemented.

[0030] In the above-mentioned embodiment of the present application, relevant change information is collected from incremental log files related to multiple data sources by executing multiple change information collection tasks, and transmitted to the downstream in the form of data streams, where the data of multiple data streams are merged into wide table data and stored in a designated storage system. Through the above-mentioned operations, the data of multiple data streams can be merged in real time based on the incremental log files of the database, and the accuracy and consistency of the merged data can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a flow chart of a multi-stream merging method based on a database log provided by the present application according to one or more embodiments;

[0032] Figure 2 A structural block diagram of a multi-stream merging device based on a database log provided by the present application according to one or more embodiments;

[0033] Figure 3 This is a diagram of the internal structure of a computer device provided by the present application according to one or more embodiments. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0035] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0036] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0037] In view of the shortcomings or defects of the prior art, the present application provides a multi-stream merging method based on database logs, which can merge the data of multiple data streams in real time based on the incremental log files of the database, and ensure the accuracy and consistency of the merged data. Specifically, by executing multiple change information collection tasks, relevant change information is collected from the incremental log files related to multiple data sources, and transmitted to the downstream in the form of data streams, where the data of multiple data streams are merged into wide table data and stored in a designated storage system.

[0038] In some embodiments, the method can be applied to a data synchronization system, which can be a server independent of each database and storage system, and can be implemented by an independent server or a server cluster composed of multiple servers. Figure 1 , Figure 1 The figure shows the steps included in the multi-stream merging method based on database logs provided in this embodiment. The following takes the application of the method to a data synchronization system as an example to describe the steps of the method in detail.

[0039] S110: Execute multiple change information collection tasks, where the change information collection tasks are used to capture target data change events according to target incremental log files, and convert the change information of the captured data change events into data streams.

[0040] Users determine the sources of data streams that need to be merged based on actual business needs. These data streams come from multiple data sources, which are data tables from the same database or different databases.

[0041] The data synchronization system can access these data sources through message queues (such as Kafka) or data stream processing frameworks (such as Storm).

[0042] The user also needs to build multiple change information collection tasks in the data synchronization system according to actual business needs. Different change information collection tasks are used to collect change information of data change events from incremental log files related to different data sources. In some embodiments, the incremental log file refers to the binlog file of the MySQL database. The binlog file is a binary format file that records the SQL (Structured Query Language) statement information of the user to update (add, delete, update) each data table in the database.

[0043] In some embodiments, the operation of capturing target data change events according to a target incremental log file includes: determining a data source specified by the task based on task information; obtaining an incremental log file related to the data source as a target incremental log file; and capturing data change events related to the data source as target data change events according to the target incremental log file.

[0044] When users build a change information collection task, they need to specify a data source, such as a data table in a database. Then, the data synchronization system can extract relevant information about the data source (such as library information and data table name) from the task information, and then use this information to determine the target data source related to this task; then, the incremental log file of the database where the target data source is located is used as the target incremental log file, and the data change events related to the data source are captured from the target incremental log file as target data change events. Among them, the operation of capturing the data change events related to the data source as the target data change events according to the target incremental log file can be to parse the target incremental log file, obtain log information, and then extract the data change events related to the data source from the parsed log information.

[0045] In some embodiments, when parsing the target incremental log file, the data synchronization system may also dynamically discard specified fields or discard specified database operation statements according to preset configuration information.

[0046] The process of merging multiple streams into a wide table based on database logs may involve a large amount of data processing and calculation, so it is necessary to consider performance and efficiency optimization. Among them, the log parsing algorithm can be optimized. Specifically, the specified database operation statements, namely DML (Data Manipulation Language) statements, can be discarded first during log parsing, or the specified fields can be dynamically discarded according to the configuration to reduce the amount of source data, thereby improving the multi-stream merging performance and efficiency of the data synchronization system.

[0047] S120: Merging the data of the data streams constructed based on the multiple change information collection tasks according to a specified merge key to obtain wide table data.

[0048] The specified merge key is a key determined according to business requirements.

[0049] In the process of merging multiple streams into a wide table, in order to deal with problems such as data conflicts, duplications or missing data between different data streams, you can formulate a suitable merging strategy based on factors such as the business meaning, importance, and timestamp of the data.

[0050] In some embodiments, the operation of merging the data of the data streams constructed based on the multiple change information collection tasks according to the specified merge key can be to align the data of each data stream according to the timestamp and operation order of the data of each data stream, and merge the data of multiple data streams based on the specified merge key in the storage layer.

[0051] When merging multi-stream data, it is necessary to ensure the timing and consistency of the data. To this end, this embodiment uses timestamps to align data from different data streams to ensure the correct merging of data. When merging, the data captured by multiple tasks can be merged through JOIN operations to obtain wide table data. When merging, the data from different data sources are associated according to the specified merge key (such as driver ID, order ID, etc., the specific merge key needs to be determined based on actual business needs). In order to ensure that multi-stream data can be aligned in the correct order and time point when merging, the same business can be executed in the same thread of a partition during the multi-stream merging process, and serialization can be used to implement sequential execution of data.

[0052] In addition, the storage layer can provide the ability to associate multiple streams of data, thereby directly merging multiple streams of data at the storage layer to reduce the burden on the computing engine and improve processing efficiency.

[0053] In some embodiments, before merging multiple streams of data, an application is used to preprocess the data of each data stream. The preprocessing includes removing duplicate data, converting data formats, filtering invalid data, etc. This preprocessing can ensure the consistency and integrity of the data.

[0054] S130: Store the wide table data into a specified storage system.

[0055] The generated wide table data will be sent to the specified storage system for storage for subsequent query and analysis. The specified storage system can be a distributed file system or database system such as Alibaba Cloud ADB database and ElasticSearch.

[0056] In some embodiments, before executing the change information collection task, the method also includes: extracting operation information related to the merge from the target incremental log file; the operation information includes library information, table information, field names, and content before and after the data change; determining a wide table structure based on the operation information; and building a wide table in the storage system based on the wide table structure.

[0057] In order to merge multiple streams into a wide table, it is necessary to build a suitable wide table structure to accommodate data from different streams, which may involve field mapping, type conversion, column name naming and other issues. By designing a reasonable wide table structure, data from multiple dimensions can be integrated into one table, which facilitates multi-dimensional association analysis and data mining. Specifically, a wide table usually contains multiple fields and columns, which may come from different data streams. Therefore, the incremental log files related to each data source can be parsed first to extract the operation information related to the merge. This operation information can be used to build the structure and content of the wide table and ensure the accuracy and consistency of the merged data. Among them, the structure and content of the wide table are related to the actual business. For example, the order wide table will be constructed based on some foreign key information on the order, such as user, product, merchant and other keys (fields).

[0058] In some embodiments, the method also includes: in the process of merging data from multiple data streams, using a message confirmation mechanism to implement an error handling and recovery mechanism, and when the message is confirmed to be executed successfully and has been confirmed, executing the next message, otherwise executing retry logic until the message is executed successfully.

[0059] Among them, in the process of merging multi-stream data, a message confirmation mechanism can be used to implement error handling and recovery mechanisms. Only when the execution is successful and confirmed, the message will continue to move down, otherwise it will go to the retry logic until it succeeds. The above operations can ensure the stability and reliability of the data synchronization system.

[0060] It should be noted that, with respect to the various steps included in the multi-stream merging method based on database logs provided in any of the above embodiments, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0061] Based on the same inventive concept, the present application also provides a multi-stream merging device based on database logs. In some embodiments, Figure 2As shown, the multi-stream merging device based on database logs includes the following modules:

[0062] The task execution module 110 is used to execute multiple change information collection tasks; the change information collection task is used to capture the target data change event according to the target incremental log file, and convert the change information of the captured data change event into a data stream;

[0063] A multi-stream merging module 120 is used to merge the data of the data streams constructed based on the multiple change information collection tasks according to a specified merging key to obtain wide table data;

[0064] The storage module 130 is used to store the wide table data into a designated storage system.

[0065] In some embodiments, the task execution module 110 includes:

[0066] A data source determination unit, used to determine a data source specified by a task based on the task information;

[0067] A log file acquisition unit, used to acquire an incremental log file related to the data source as a target incremental log file;

[0068] The change event capture unit is used to capture data change events related to the data source as target data change events according to the target incremental log file.

[0069] In some embodiments, the change event capture unit captures data change events related to the data source as target data change events based on the target incremental log file, including: parsing the target incremental log file, and extracting data change events related to the data source from the parsed log information.

[0070] In some embodiments, the task execution module 110 parses the operation of the target incremental log file, including: dynamically discarding a specified field or discarding a specified database operation statement according to preset configuration information.

[0071] In some embodiments, the specified merge key is a key determined according to business requirements; the multi-stream merging module 120 includes:

[0072] A data alignment unit, used for aligning the data of each data stream according to the timestamp and operation sequence of the data of each data stream;

[0073] The merge unit is used to merge the data of multiple data streams based on the specified merge key in the storage layer.

[0074] In some embodiments, the apparatus further comprises:

[0075] An operation information extraction unit is used to extract operation information related to the merge from the target incremental log file; the operation information includes library information, table information, field names, and content before and after data changes;

[0076] A table structure determination unit, used to determine a wide table structure based on the operation information;

[0077] A wide table construction unit is used to construct a wide table in a storage system based on a wide table structure.

[0078] In some embodiments, the device also includes an error handling and monitoring module, which is used to use a message confirmation mechanism to implement an error handling and recovery mechanism in the process of merging data from multiple data streams. When the message execution is confirmed to be successful and has been confirmed, the next message is executed, otherwise the retry logic is executed until the message execution is successful.

[0079] For the specific definition of the multi-stream merging device based on database logs, please refer to the definition of the multi-stream merging method based on database logs above, which will not be repeated here. Each module in the above-mentioned multi-stream merging device based on database logs can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In some embodiments, the present application provides a computer device, whose internal structure diagram can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-stream merging method based on a database log is implemented.

[0081] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0082] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-stream merging method based on database logs provided in any of the above embodiments can be implemented.

[0083] Specifically, when the processor executes the computer program, the following steps can be implemented:

[0084] Execute multiple change information collection tasks; the change information collection task is used to capture the target data change event according to the target incremental log file, and convert the change information of the captured data change event into a data stream; merge the data of the data stream constructed based on the multiple change information collection tasks according to the specified merge key to obtain wide table data; store the wide table data in the specified storage system.

[0085] In one embodiment, when the processor executes the computer program to capture the target data change event according to the target incremental log file, the processor further implements the following steps:

[0086] Determine the data source specified by the task based on the task information; obtain the incremental log file related to the data source as the target incremental log file; and capture the data change event related to the data source as the target data change event according to the target incremental log file.

[0087] In one embodiment, when the processor executes the computer program to capture the data change event related to the data source as the target data change event according to the target incremental log file, the processor further implements the following steps:

[0088] Parse the target incremental log file and extract data change events related to the data source from the parsed log information.

[0089] In one embodiment, when the processor executes the computer program to parse the target incremental log file, the processor also implements the following steps:

[0090] Dynamically discard specified fields or specified database operation statements according to preset configuration information.

[0091] In one embodiment, the specified merge key is a key determined according to business requirements; when the processor executes the computer program to merge the data of the data stream constructed based on the multiple change information collection tasks according to the specified merge key, the following steps are also implemented:

[0092] The data of each data stream is aligned according to the timestamp and operation order of the data of each data stream; the data of multiple data streams are merged based on the specified merge key in the storage layer.

[0093] In one embodiment, the processor executes the computer program to implement the following steps before executing the change information collection task:

[0094] Extract merge-related operation information from the target incremental log file; the operation information includes library information, table information, field names, and content before and after data changes; determine the wide table structure based on the operation information; and build a wide table in the storage system based on the wide table structure.

[0095] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0096] In the process of merging data from multiple data streams, a message confirmation mechanism is used to implement an error handling and recovery mechanism. When the message is confirmed to be executed successfully and has been confirmed, the next message is executed, otherwise the retry logic is executed until the message is executed successfully.

[0097] In some embodiments, the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-stream merging method based on database logs provided in any of the above embodiments are implemented.

[0098] Specifically, when the computer program is executed by a processor, the following steps can be implemented:

[0099] Execute multiple change information collection tasks; the change information collection task is used to capture the target data change event according to the target incremental log file, and convert the change information of the captured data change event into a data stream; merge the data of the data stream constructed based on the multiple change information collection tasks according to the specified merge key to obtain wide table data; store the wide table data in the specified storage system.

[0100] In one embodiment, the computer program is executed by a processor, and when capturing a target data change event according to a target incremental log file, the computer program further implements the following steps:

[0101] Determine the data source specified by the task based on the task information; obtain the incremental log file related to the data source as the target incremental log file; and capture the data change event related to the data source as the target data change event according to the target incremental log file.

[0102] In one embodiment, the computer program is executed by a processor, and when capturing data change events related to the data source as target data change events according to the target incremental log file, the computer program further implements the following steps:

[0103] Parse the target incremental log file and extract data change events related to the data source from the parsed log information.

[0104] In one embodiment, the computer program is executed by a processor, and when parsing the target incremental log file, the following steps are further implemented:

[0105] Dynamically discard specified fields or specified database operation statements according to preset configuration information.

[0106] In one embodiment, the specified merge key is a key determined according to business requirements; when the computer program is executed by a processor, the computer program further implements the following steps when merging the data of the data stream constructed based on the multiple change information collection tasks according to the specified merge key:

[0107] The data of each data stream is aligned according to the timestamp and operation order of the data of each data stream; the data of multiple data streams are merged based on the specified merge key in the storage layer.

[0108] In one embodiment, the computer program is executed by a processor, and before executing the change information collection task, the following steps are further implemented:

[0109] Extract merge-related operation information from the target incremental log file; the operation information includes library information, table information, field names, and content before and after data changes; determine the wide table structure based on the operation information; and build a wide table in the storage system based on the wide table structure.

[0110] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0111] In the process of merging data from multiple data streams, a message confirmation mechanism is used to implement an error handling and recovery mechanism. When the message is confirmed to be executed successfully and has been confirmed, the next message is executed, otherwise the retry logic is executed until the message is executed successfully.

[0112] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0113] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-stream merging method based on database logs, characterized in that: The method comprises: Execute multiple change information collection tasks; the change information collection tasks are used to capture target data change events according to target incremental log files, and convert the change information of the captured data change events into data streams; Merging the data of the data streams constructed based on the multiple change information collection tasks according to a specified merge key to obtain wide table data; The wide table data is stored in a designated storage system.

2. The method according to claim 1, characterized in that Capture target data change events based on the target incremental log file, including: Determine a data source specified by the task based on the task information; Obtaining the incremental log file related to the data source as the target incremental log file; Data change events related to the data source are captured according to the target incremental log file as target data change events.

3. The method according to claim 2, characterized in that Capturing data change events related to the data source as target data change events according to the target incremental log file includes: The target incremental log file is parsed, and data change events related to the data source are extracted from the parsed log information.

4. The method according to claim 3, characterized in that When parsing the target incremental log file, the method further includes: Dynamically discard specified fields or specified database operation statements according to preset configuration information.

5. The method according to claim 1, characterized in that The specified merge key is a key determined according to business requirements; merging the data of the data streams constructed based on the multiple change information collection tasks according to the specified merge key includes: Align the data of each data stream according to the timestamp and operation order of the data of each data stream; In the storage layer, data from multiple data streams are merged based on the specified merge key.

6. The method according to claim 1, characterized in that Before executing the change information collection task, the method further includes: Extracting operation information related to the merge from the target incremental log file; the operation information includes library information, table information, field names, and content before and after data changes; determining a wide table structure based on the operation information; A wide table is constructed in the storage system based on the wide table structure.

7. The method according to claim 1, characterized in that The method further comprises: In the process of merging data from multiple data streams, a message confirmation mechanism is used to implement an error handling and recovery mechanism. When the message is confirmed to be executed successfully and has been confirmed, the next message is executed, otherwise the retry logic is executed until the message is executed successfully.

8. A multi-stream merging device based on database logs, characterized in that: The device comprises: A task execution module, used to execute multiple change information collection tasks; the change information collection tasks are used to capture target data change events according to target incremental log files, and convert the change information of the captured data change events into data streams; A multi-stream merging module, used to merge the data of the data streams constructed based on the multiple change information collection tasks according to a specified merging key to obtain wide table data; The storage module is used to store the wide table data into a specified storage system.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images