Data processing methods, management systems, terminal devices and storage media for entering the data lake

Abstract

The present application relates to the field of data processing technology, and discloses a data entry processing method, management system, terminal device and storage medium; the method includes collecting business data in a database and monitoring change events; the change event is an event recorded in a data structure change in a data log of the database; the business data and the change event are processed separately to obtain a main message and a change message respectively; the change message is processed by a coprocessor and consumed with the main message and written into a data lake. The present application can improve the synchronization timeliness of data written into the data lake for the entire database.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, management system, terminal device and storage medium for processing data entering a lake. Background Art

[0002] With the rapid development of digital transformation, the volume of data is also growing rapidly. Data lakes have been introduced into big data platforms by many enterprises and companies due to their efficient real-time ingestion, reading efficiency, and better compatibility with traditional big data warehouse technologies.

[0003] A data lake is a collection of large-scale structured and unstructured data. It can accommodate data from various databases / data sources, including log file data, database data, sensor data, etc. Based on this, the data lake natively provides transactional management, version control capabilities, data structure change capabilities, and supports real-time data synchronization.

[0004] Currently, data in databases or data sources are generally synchronized in real time by offline scheduling or writing forms into the data lake:

[0005] (1) Offline scheduling means that tasks are started at certain time intervals and configured on the scheduling system. The synchronization period is generally daily or hourly. The data synchronization period is long and the time delay is high, which cannot meet the application scenarios with high timeliness requirements.

[0006] (2) When writing data into the lake through forms, when facing large databases or databases with many data tables, a separate task needs to be configured for each table, which consumes a lot of resources. In addition, manual maintenance is required when the data structure changes, which has a certain time lag and cannot be processed in time, affecting the real-time nature of the data. Summary of the invention

[0007] In view of this, the embodiments of the present application provide a method, management system, terminal device and storage medium for processing data entering a data lake, which can effectively solve the problem of poor synchronization timeliness when data consumption is written into a data lake.

[0008] In a first aspect, an embodiment of the present application provides a method for processing data entering a lake, including:

[0009] Collect business data in the database and monitor change events; the change events are events recorded in the data structure change in the data log of the database;

[0010] Performing split processing on the business data and the change event to obtain a main message and a change message respectively;

[0011] The change message is processed by the coprocessor and consumed together with the main message and written into the data lake.

[0012] In some embodiments, the business data and the change event are processed separately to obtain a main message and a change message, respectively, including:

[0013] Dividing the business data and the change event to obtain a main data stream and a side output stream respectively;

[0014] Processing the main data stream to obtain the main message;

[0015] The side output stream is processed to obtain the change message.

[0016] In some embodiments, the main data stream includes the business data and the change event; and the side output stream includes the change event.

[0017] In some embodiments, the main message includes the business data and a data message queue topic;

[0018] The processing of the main data stream to obtain the main message includes:

[0019] When there is no change event of the newly added table in the main data stream, the business data is transmitted to the message queue;

[0020] When there is a change event of a newly added table in the main data stream, a data message queue topic is synchronously created according to the change event of the newly added table, and the data message queue topic is transmitted to the message queue.

[0021] In some embodiments, the step of processing the change message through the coprocessor and consuming the main message and writing it into the data lake includes:

[0022] When the main message is consumed and written through the message queue, if the change event is not captured, the main message is directly written to the data lake; if the change event is captured, the consumption of the main data stream is suspended and the cached data is flushed;

[0023] After the cache data is flushed out, continue to wait for the change event when consumption is suspended in the message queue to be matched with the conversion task event; wherein the change message is processed by the coprocessor to obtain the conversion task event;

[0024] After the match is successful, the data flow for consuming the main message is resumed to continue pushing the next main message in the message queue for consumption and writing into the data lake.

[0025] In some embodiments, the change message is processed by a coprocessor, including:

[0026] The coprocessor captures the change message and processes the change message to obtain change operation information;

[0027] After receiving the cache data flushing completion signal, the coprocessor performs a change operation on the data lake according to the change operation information to update the data structure of the data lake;

[0028] After monitoring that the data structure of the data lake is updated, the coprocessor generates the conversion task event according to the change operation information.

[0029] In some embodiments, the business data and the data message queue topic are transmitted to the message queue via a message system.

[0030] In a second aspect, an embodiment of the present application provides a management system for data entry into a lake, including a data collector, a coprocessor, and a writing device;

[0031] The data collector is used to collect business data in the database and monitor change events; the change events are events recorded in the data structure change in the data log of the database;

[0032] The data collector is also used to perform shunt processing on the business data and the change event to obtain a main message and a change message respectively;

[0033] The coprocessor is used to process the change message;

[0034] The writing device is used to consume and write the processing result of the coprocessor and the main message into the data lake.

[0035] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned data entry processing method.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed on a processor, implements the processing method according to the above-mentioned data entry.

[0037] The embodiments of the present application have the following beneficial effects:

[0038] This application not only collects business data from the database but also monitors the data log, and captures change events corresponding to data structure change records from the monitored data log; in order to improve the real-time performance of data entry into the lake, business data and change events are diverted for processing, thereby obtaining main messages and change messages; the change messages are processed by the coprocessor and consumed together with the main messages and written into the data lake, wherein the coprocessor processes the change messages separately, is able to perceive and process change events, thereby realizing automated change tasks and maintenance; compared with the prior art, table structure change processing requires manual intervention and restart to trigger the loading of new data structure information to construct tasks, therefore, this application improves the real-time and efficiency of data entry into the lake through diversion processing and separate processing of change events by the coprocessor. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a process for processing data entering a lake in an embodiment of the present application is shown;

[0041] Figure 2 A schematic diagram showing a specific implementation process of the method for processing data entering the lake in an application scenario in an embodiment of the present application is shown;

[0042] Figure 3 A schematic diagram of the flow chart of the diversion process in an embodiment of the present application is shown;

[0043] Figure 4 A structural diagram of a management system for data entry into a lake in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0045] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] Hereinafter, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0047] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0048] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0049] The real-time performance of the existing data entry process cannot meet the application scenarios with high timeliness requirements. Therefore, in order to better realize real-time data synchronization, this application proposes a data entry processing method, which combines CDC (Change Data Capture) technology to enable change events of databases or data sources to be captured in real time and seamlessly merged into the data lake through stream processing jobs, thereby ensuring the real-time and consistency of the data lake.

[0050] The following describes the method for processing data entering the lake in conjunction with some specific embodiments.

[0051] Figure 1 A flow chart of a method for processing data entering the lake in an embodiment of the present application is shown. Exemplarily, the method for processing data entering the lake includes the following steps:

[0052] S100, collecting business data in the database and monitoring change events; wherein the event recorded in the data log as a data structure change is a change event.

[0053] In some specific embodiments, the business system includes multiple databases, and the business data in each database is stored via multiple data structures.

[0054] In this example, MySQL database can be used, in which any modification to the database at any time will be recorded in the corresponding binglog log; when data is added, deleted, or modified, the creation of database objects will be recorded in the binlog log, and database replication is also synchronized based on the binlog log. Therefore, by monitoring the binlog log, data structure change records can be captured and business data in the database can be collected.

[0055] It can be understood that binglog is a binary log that records business data in the database and all change events that may cause data changes. Change refers to addition, deletion or modification. This application uses the Change Data Capture (CDC) method to capture events recorded in the data structure change in the data log; therefore, the events recorded in the data structure change in the data log are change events.

[0056] In this embodiment, in a CDC technology collection task, the data of the entire database is synchronized to the Hudi data lake, which does not require task configuration and operation and maintenance management for each individual table. For a CDC collection task, the ETL writing process can also be split into one or more tasks to support databases with many data tables.

[0057] S200, performing diversion processing on the business data and change events of the database to obtain a main message and a change message respectively.

[0058] Exemplarily, when a data structure change record is captured, it indicates that the change event needs to be specially processed based on the business data of the database, that is, the database change event is filtered and sent to the side output stream for subsequent separate processing by the coprocessor, thereby improving the real-time performance of data entering the lake.

[0059] In some specific embodiments, the change event may specifically be creating / deleting / modifying a data table, or creating / deleting / modifying rows and columns in a data table, etc. This application does not limit the specific content of the change event.

[0060] In some specific embodiments, Figure 3 As shown, the business data and change events of the above database are processed separately, and the process of obtaining the main message and the change message respectively includes:

[0061] S210, splitting the business data and change events of the database to obtain a main data stream and a side output stream respectively;

[0062] Specifically, Figure 2In the data collection part, the business data and change events of the database are directly collected into the main data stream; at the same time, the change events of the database are filtered to obtain the side output stream; thus, the data information is split into the main data stream and the side output stream. Since the main data stream is directly collected, the main data stream includes change events and business data; and the side output stream is obtained by filtering the change events in the data information, so the side output stream includes change events that need special processing.

[0063] S220, processing the main data stream to obtain a main message;

[0064] Specifically, Figure 2 In the data collection part, the data in the main data stream is collected and judged. If it is judged that there is no new table in the current data, the current business data is collected into the message system before the next data is collected; if it is judged that there is a new table in the current data, the data message queue topic corresponding to the new table is created synchronously, and after the creation is completed, the new table data is written into the corresponding topic to obtain a complete data message queue. That is, the business data corresponding to the change event of no new table and the data message queue topic created synchronously when there is a new table are the main messages generated by the main data stream. The main message transmission can be directly transmitted to the message queue, or it can be transmitted to the message system first, and then the main message is read from the message system and transmitted to the message queue.

[0065] S230, processing the output stream on the opposite side to obtain a change message.

[0066] Specifically, a change message corresponding to a change event (such as addition, deletion / modification) is issued according to a change event in the side output stream. In other words, the processing of the side output stream is to issue a corresponding change message according to the change event. Exemplarily, when the change event is a new table, a change message of the innovation table is issued, and when the change event is a column deletion, a change message of the column deletion is issued.

[0067] S300: After being processed by the coprocessor, the change message is consumed together with the main message and written into the data lake.

[0068] When a data structure change occurs in the database of the upstream business of this application, the coprocessor will control the processing of the change event and send a pause to the writing process of the data stream to the consumer write process, and flush all the data that has been read in the pipeline of the main data stream (that is, in the message queue) to ensure the consistency of the data structure; when the data structure change is successfully processed in the downstream storage system, the relevant information of the change event is synchronized to the corresponding node of the write process, and the coprocessor resumes the data flow in the message queue to complete the processing of this data structure change.

[0069] In an exemplary description, Figure 2As shown in the coprocessing and message queue section, the process of consuming the above change message and writing it into the data lake with the main message corresponding to the main data stream after the coprocessor coprocesses the change message includes:

[0070] S310, when the main message is consumed and written through the message queue, if no change event is captured, the main message consumption is directly written into the data lake;

[0071] Specifically, during the consumption process, the main message of the main data stream is first transmitted to the message queue to achieve demodulation between data collection and consumption writing processes. That is, the Kafka message queue is used as the middleware to decouple the data collection process and the data writing process, achieve compatibility processing of data structure changes, and provide a highly reliable data integration model for the data lake. In addition, the data in the message queue can be used for data replay or multiple data stream consumption, which can reduce the pressure on the upstream database.

[0072] For example, in some embodiments, when determining whether there is a change event such as "create a table, delete a table, or change a table structure", if no change event is captured, the main message is directly written into the data lake.

[0073] S320, the coprocessor captures the change message and processes the change message to obtain change operation information;

[0074] In some embodiments, after the coprocessor captures the change message of the change event, it parses the change message and converts the parsed event information into change operation information of the data structure.

[0075] More specifically, after the coprocessor captures the change message, it parses the change message to obtain event information of adding and / or deleting a data structure, and converts the event information to obtain change operation information that can be easily executed later.

[0076] S330, when the main message is consumed and written through the message queue, if a change event is captured, the data flow of consuming the main message is paused, and the cached data is flushed;

[0077] If the coprocessor captures a change event, it will send the change event to automatically suspend consumption of the main data stream and flush all business data that have been read in the main data stream to ensure the consistency of the data structure. The business data that has been read from the main data stream before suspending consumption is defined as cached data.

[0078] S340, after the cache data is flushed, the coprocessor executes the change operation information to update the data structure of the data lake;

[0079] like Figure 2As shown, if the cache data is not flushed out, the coprocessor will wait until the cache data is flushed out. The coprocessor will then change the data structure of the data lake according to the change operation information. That is, the coprocessor creates, deletes or changes the data structure of the data lake according to the change operation information. In other words, the data structure of the data lake is updated through the execution operation of the coprocessor.

[0080] S350, after the data structure of the monitoring data lake is updated, the coprocessor generates a conversion task event according to the change operation information, and sends completion information of the change operation information;

[0081] In some embodiments, during the process of changing the table structure of the data lake, the coprocessor keeps monitoring to ensure that the data structure of the data lake is completely changed.

[0082] In some embodiments, after the data lake completes the update of the data structure, the coprocessor generates a conversion task event based on the change event to ensure that the updated structure data can be written into the data lake by ETL. Exemplarily, the conversion task event is to create / delete / change the ETL task in order to match the change event in the ETL writing process; therefore, the conversion task event is used to match the change event when consumption is suspended in the message queue; it is a related event of the writing process for subsequent matching processing. After generating the conversion task event, the coprocessor notifies that the change message according to the change event has been processed. The coprocessor of this step automates the task structure changes of the writing process and notifies the writing process that the related events are completed for subsequent matching.

[0083] S360, after the cache data is flushed out, continue to wait for the change event when consumption is suspended in the message queue to match the conversion task event generated by the coprocessor;

[0084] In some embodiments, after the cache data is flushed, it starts waiting for matching, and when the completion information of the change operation sent by the coprocessor is received, the change event consumed in the message queue is matched with the conversion task event generated by the coprocessor.

[0085] S370: After the match is successful, the data flow of the main message consumption is restored to continue pushing the next main message in the message queue for consumption and writing into the data lake.

[0086] In some embodiments, after a successful match, the main message in the message queue is automatically restored for consumption to ensure that the data corresponding to the subsequent main message continues to be consumed and written into the Hudi data lake.

[0087] It can be understood that in the existing structural change processing, since the data structure information captured in the structural change processing task is loaded when the task is started and built, it cannot be updated during the task execution, so manual intervention is required to restart and trigger the loading of new table structure information to build the task; this structural change method is not very friendly to task maintenance and operation quality, and it is easy to cause abnormal restart loading or restart failure. The dynamic perception method of the coprocessor designed in this application can support the coprocessor to perceive the change event and perform data lake structure change and write process task structure change when the upstream database structure changes, and trigger the processing and distribution of related events; thereby realizing automated change tasks and operation and maintenance.

[0088] In addition, in the data integration scenario where data is synchronized to the data lake, the present application implements real-time synchronization, adaptive change of data structure, and synchronization link design by library and multi-table synchronization. Compared with the common single-table synchronization into the lake in the industry, it greatly reduces the synchronization tasks to be maintained, reduces the resources consumed by computing and synchronization, and reduces the pressure on synchronous reading of upstream databases. Compared with the scenarios in the industry where it is impossible to dynamically change data structures and dynamically add and delete tables, the present application uses a coprocessor to coordinate with other processing nodes of the data synchronization link to process different upstream data structure changes in different modes without restarting or manual intervention, thereby ensuring the stability and timeliness of the real-time synchronization link.

[0089] Figure 4 A schematic diagram of a data entry management system of an embodiment of the present application is shown. Exemplarily, the data entry management system 100 includes a data collector 110, a coprocessor 120 and a writing device 130;

[0090] The data collector is used to collect business data of the database and monitor the change events of the database; the change events are events recorded in the data structure change in the data log of the database.

[0091] The data collector is also used to perform diversion processing on business data and change events to obtain main messages and change messages respectively.

[0092] The coprocessor is used to process the change message. The specific process of the processing can be found in the description of the above embodiment, which will not be repeated here.

[0093] The writing device is used to consume and write the processor result of the co-processing and the main message into the data lake.

[0094] In some embodiments, the process of the writing device performing consumption writing into the data lake includes:

[0095] When the main message enters the writing device after passing through the message queue, if no change event is captured during consumption writing, the main message is directly written to the data lake; if a change event is captured, the data flow of consuming the main message is paused and the cached data is flushed;

[0096] After the writing device waits for the cache data to be flushed out, it continues to wait for the change event when consumption is suspended in the message queue to match and process the conversion task event;

[0097] After the match is successful, the writing device resumes consuming the data flow of the main message, and continues to push the next main message in the message queue for consumption and writing into the data lake.

[0098] In some embodiments, the process of the coprocessor processing the change message includes:

[0099] The coprocessor captures the change message and processes the change message to obtain the change operation information;

[0100] After receiving the cache data output completion information, the coprocessor performs a change operation on the data lake according to the change operation information to update the data structure of the data lake;

[0101] After the coprocessor monitors the data structure of the data lake and completes the update, it generates a conversion task event based on the change operation information and sends down completion information of the change operation; wherein the conversion task event is used to match the change event when consumption is suspended in the message queue.

[0102] It can be understood that the management system of this embodiment corresponds to the data entry method of the above embodiment, and the optional options in the above embodiment are also applicable to this embodiment, so they will not be repeated here.

[0103] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned data entry processing method or the functions of each module in the above-mentioned data entry management system.

[0104] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0105] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0106] The present application also provides a computer-readable storage medium storing a computer program, which, when executed on a processor, implements the above-mentioned data entry processing method. For example, the computer-readable storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0107] In several embodiments provided in the present application, it should be understood that the disclosed method can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of the program segment or code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0109] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0110] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for processing data entering a lake, characterized in that: include: Collect business data in the database and monitor change events; The change event is an event recorded in the data log of the database as a data structure change; Performing split processing on the business data and the change event to obtain a main message and a change message respectively; The performing split processing on the business data and the change event to obtain a main message and a change message respectively includes: The business data and the change event are split to obtain a main data stream and a side output stream respectively; the main data stream includes the business data and the change event; the side output stream includes the change event; Processing the main data stream to obtain the main message; Processing the side output stream to obtain the change message; The change message is processed by the coprocessor and then consumed together with the main message and written into the data lake; The step of processing the change message through the coprocessor and consuming the main message and writing it into the data lake includes: When the main message is consumed and written through the message queue, if the change event is not captured, the main message is directly written to the data lake; if the change event is captured, the data flow consuming the main message is paused and the cached data is flushed; After the cache data is flushed out, continue to wait for the change event when consumption is suspended in the message queue to be matched with the conversion task event; wherein the change message is processed by the coprocessor to obtain the conversion task event; the conversion task event includes creating, deleting and / or changing ETL tasks; After the match is successful, the data flow of consuming the main message is resumed to continue pushing the next main message in the message queue for consumption and writing into the data lake; The change message is processed by the coprocessor, including: The coprocessor captures the change message and processes the change message to obtain change operation information; After receiving the cache data flushing completion signal, the coprocessor performs a change operation on the data lake according to the change operation information to update the data structure of the data lake; After monitoring that the data structure of the data lake is updated, the coprocessor generates the conversion task event according to the change operation information.

2. The method for processing data entering a lake according to claim 1, characterized in that: The main message includes the business data and the data message queue topic; The processing of the main data stream to obtain the main message includes: When there is no change event of the newly added table in the main data stream, the business data is transmitted to the message queue; When there is a change event of a newly added table in the main data stream, a data message queue topic is synchronously created according to the change event of the newly added table, and the data message queue topic is transmitted to the message queue.

3. The method for processing data entering a lake according to claim 2, characterized in that: The business data and the data message queue topic are transmitted to the message queue via a message system.

4. A data lake management system, characterized in that: It includes a data collector, a coprocessor and a writing device; The data collector is used to collect business data in the database and monitor change events; the change events are events recorded in the data structure change in the data log of the database; The data collector is also used to perform shunt processing on the business data and the change event to obtain a main message and a change message respectively; The performing split processing on the business data and the change event to obtain a main message and a change message respectively includes: The business data and the change event are split to obtain a main data stream and a side output stream respectively; the main data stream includes the business data and the change event; the side output stream includes the change event; Processing the main data stream to obtain the main message; Processing the side output stream to obtain the change message; The writing device is used to consume and write the processing result of the coprocessor and the main message into the data lake; when the main message is consumed and written through the message queue, if the change event is not captured, the main message is directly written into the data lake; if the change event is captured, the data flow of consuming the main message is suspended, and the cached data is flushed; After the cache data is flushed out, continue to wait for the change event when consumption is suspended in the message queue to be matched with the conversion task event; wherein the change message is processed by the coprocessor to obtain the conversion task event; After the match is successful, the data flow of consuming the main message is resumed to continue pushing the next main message in the message queue for consumption and writing into the data lake; The coprocessor is used to process the change message, including: The coprocessor captures the change message and processes the change message to obtain change operation information; After receiving the cache data flushing completion signal, the coprocessor performs a change operation on the data lake according to the change operation information to update the data structure of the data lake; After monitoring the data structure of the data lake to complete the update, the coprocessor generates the conversion task event according to the change operation information; the conversion task event includes creating, deleting and / or changing ETL tasks.

5. A terminal device, characterized in that: The terminal device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the data entry processing method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed on a processor, implements the data entry processing method according to any one of claims 1 to 3.

Images