Data transmission method and system
By merging streaming and batch data transfers within a single operating system process and utilizing a component coordinator to control the execution order, Flink solves the problems of data duplication and failure during data transfer, achieving efficient and low-cost data transfer.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING DIPU TECH CO LTD
- Filing Date
- 2023-02-06
- Publication Date
- 2026-04-14
AI Technical Summary
In existing technologies, Flink cannot effectively connect streaming and batch data transmission, resulting in duplicate new data, failures in data modification and deletion, and high deployment costs.
Data is processed by streaming read components and batch read components respectively. The execution order is controlled by the component coordinator, and streaming and batch transmission are achieved in a single operating system process to ensure data order and no duplication.
It simplifies task configuration and operation and maintenance costs, reduces deployment costs, and enables seamless integration of streaming and batch data transmission, avoiding data duplication and operation failures.
Smart Images
Figure CN116150208B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data transmission technology, specifically to a data transmission method and system. Background Technology
[0002] Existing data transmission methods typically only support streaming or batch data transmission. If only streaming transmission is supported, although incremental data can be transmitted in real time, it is difficult to migrate and transmit existing data. If only batch data transmission is supported, the task stops once the existing data transmission is completed, and incremental data cannot be transmitted in real time.
[0003] Flink combines the advantages of streaming and batch processing, allowing for simultaneous data transmission of both types. However, due to its distributed architecture, Flink has high deployment costs and consumes significant resources. Furthermore, because Flink performs both streaming and batch processing simultaneously, newly added data in the source database during transmission will be queried by both streaming and batch processes. If both are sent downstream to be written to the target database, data duplication may occur. For modification and deletion operations in the source database during transmission, there is a certain probability that streaming reads will capture these operations before batch reads. When the streaming process sends the data modification and deletion operations to the downstream target database first, the data has not yet been transmitted to the target database by the batch process, causing the data modification and deletion to fail.
[0004] Among related technologies, although Flink can perform both streaming and batch data transmission simultaneously, it cannot achieve good data integration between streaming and batch transmission. Summary of the Invention
[0005] To at least partially overcome the problems in related technologies where streaming and batch data transmission cannot achieve good data continuity, leading to duplicate new data and failures in data modification and deletion, this application provides a data transmission method and system.
[0006] According to a first aspect of the embodiments of this application, a data transmission method is provided, comprising the following steps:
[0007] Start the streaming read component to connect to the source database and obtain the current offset of the source database;
[0008] The batch read component is activated to transfer a batch of data from the source database to the target database; the batch data is determined based on the current offset.
[0009] After the batch data transfer is completed, the batch read component stops operating;
[0010] Start the streaming read component to transfer incremental data from the source database to the target database.
[0011] Furthermore, transferring batch data from the source database to the target database includes the following steps:
[0012] Generate a data snapshot in the source database;
[0013] Data is read in batches from the data snapshot and then transferred to the target database.
[0014] Furthermore, the read data is transferred to the target database, including the following steps:
[0015] The read data is encapsulated according to a unified standard;
[0016] The packaged data is sent to a preset data queue, and then the next batch of data is read, packaged, and sent until all batches of data have been processed.
[0017] Transfer the data in the data queue to the target database.
[0018] Further, the data in the data queue is transferred to the target database, including the following steps:
[0019] Take a batch of data from the data queue and insert it into the target database;
[0020] Then process the next batch of data until all batches of data in the data queue have been inserted.
[0021] Furthermore, the batch read component stops operating, including the following steps:
[0022] The batch read component sends a message to the component coordinator, notifying that all batch data has been transferred.
[0023] Then the batch read component begins to self-destruct, releasing database connections and thread resources before stopping operation.
[0024] Furthermore, the incremental data is transferred from the source database to the target database, including the following steps:
[0025] The incremental data is determined based on the current offset, and one incremental data point is collected each time.
[0026] Incremental data is collected from the source database, and the collected incremental data is encapsulated and sent to the data queue.
[0027] Transfer incremental data from the data queue to the target database.
[0028] Furthermore, the collected incremental data is encapsulated and sent to the data queue, including the following steps:
[0029] The collected incremental data is encapsulated according to a unified standard, with one incremental data entry encapsulated at a time.
[0030] The encapsulated incremental data is sent to the data queue, and then the next incremental data is collected, encapsulated, and sent.
[0031] The streaming read component continues to run unless a stop command is received.
[0032] Furthermore, the incremental data in the data queue is transferred to the target database, including the following steps:
[0033] Retrieve one incremental data entry from the data queue;
[0034] Synchronize operations on the target database based on the type of incremental data operation.
[0035] Process all incremental data in the data queue sequentially.
[0036] Furthermore, the incremental data includes operation data such as adding, modifying, and / or deleting.
[0037] According to a second aspect of the embodiments of this application, a data transmission system is provided, including: a streaming read component, a batch read component, a component coordinator, and a write component;
[0038] The component coordinator is used to start the streaming read component so that the streaming read component connects to the source database and obtains the current offset of the source database;
[0039] The component coordinator is also used to initiate the batch read component, so that the batch read component and the write component transfer batch data from the source database to the target database; the batch data is data determined based on the current offset;
[0040] After the batch data transfer is completed, the batch read component stops operating;
[0041] The component coordinator is also used to initiate the streaming read component so that the streaming read component and the write component transfer incremental data from the source database to the target database.
[0042] The technical solutions provided by the embodiments of this application have the following beneficial effects:
[0043] The technical solution of this application merges streaming and batch tasks into one, avoiding the trouble of configuring multiple data transmission tasks and simplifying task configuration operations and maintenance costs. This solution achieves integrated streaming and batch processing in a single process, which is more lightweight than a distributed architecture and can save deployment costs. At the same time, this solution can ensure the orderliness of data and can perfectly connect the two transmission methods, so that data transmission is not duplicated, lost, or fails.
[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0046] Figure 1 This is a flowchart illustrating a data transmission method according to an embodiment of the present invention.
[0047] Figure 2 This is a block diagram illustrating a data transmission system according to an embodiment of the present invention. Detailed Implementation
[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.
[0049] In existing technologies, streaming and batch data transmission cannot achieve good data integration, leading to duplicate new data, and failures in data modification and deletion. This method aims to propose a new integrated streaming and batch transmission mode that can simultaneously support the transmission of streaming and batch data within a single operating system process, and perfectly integrate the two transmission modes to ensure that data is not duplicated, not lost, and that operations do not fail.
[0050] Figure 1 This is a flowchart illustrating a data transmission method according to an exemplary embodiment. The method may include the following steps:
[0051] Step S1: Start the streaming read component to connect to the source database and obtain the current offset of the source database;
[0052] Step S2: Start the batch read component to transfer batch data from the source database to the target database; the batch data is determined based on the current offset.
[0053] Step S3: After the batch data transfer is completed, the batch read component stops running;
[0054] Step S4: Start the streaming read component to transfer incremental data from the source database to the target database.
[0055] The technical solution of this application merges streaming and batch tasks into one, avoiding the trouble of configuring multiple data transmission tasks and simplifying task configuration operations and maintenance costs. This solution achieves integrated streaming and batch processing in a single process, which is more lightweight than a distributed architecture and can save deployment costs. At the same time, this solution can ensure the orderliness of data and can perfectly connect the two transmission methods, so that data transmission is not duplicated, lost, or fails.
[0056] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0058] In some embodiments, transferring batch data from a source database to a target database includes the following steps:
[0059] Generate a data snapshot in the source database;
[0060] Data is read in batches from the data snapshot and then transferred to the target database.
[0061] In some embodiments, transferring the read data to the target database includes the following steps:
[0062] The read data is encapsulated according to a unified standard;
[0063] The packaged data is sent to a preset data queue, and then the next batch of data is read, packaged, and sent until all batches of data have been processed.
[0064] Transfer the data in the data queue to the target database.
[0065] In some embodiments, transferring data from a data queue to a target database includes the following steps:
[0066] Take a batch of data from the data queue and insert it into the target database;
[0067] Then process the next batch of data until all batches of data in the data queue have been inserted.
[0068] In some embodiments, the batch read component stops operating, including the following steps:
[0069] The batch read component sends a message to the component coordinator, notifying that all batch data has been transferred.
[0070] Then the batch read component begins to self-destruct, releasing database connections and thread resources before stopping operation.
[0071] In some embodiments, transferring incremental data from the source database to the target database includes the following steps:
[0072] The incremental data is determined based on the current offset, and one incremental data point is collected each time.
[0073] Incremental data is collected from the source database, and the collected incremental data is encapsulated and sent to the data queue.
[0074] Transfer incremental data from the data queue to the target database.
[0075] In some embodiments, the collected incremental data is encapsulated and sent to the data queue, including the following steps:
[0076] The collected incremental data is encapsulated according to a unified standard, with one incremental data entry encapsulated at a time.
[0077] The encapsulated incremental data is sent to the data queue, and then the next incremental data is collected, encapsulated, and sent.
[0078] The streaming read component continues to run unless a stop command is received.
[0079] In some embodiments, transferring incremental data from a data queue to a target database includes the following steps:
[0080] Retrieve one incremental data entry from the data queue;
[0081] Synchronize operations on the target database based on the type of incremental data operation.
[0082] Process all incremental data in the data queue sequentially.
[0083] In some embodiments, the incremental data includes operation data for adding, modifying, and / or deleting data.
[0084] like Figure 2 As shown, the implementation principle of this method is to split the streaming read and batch read of the source database during the data transmission process into two different read components, and the component coordinator controls the order of execution of these two components. Combined with a writing component that integrates streaming and batch, each module is executed by one or more separate threads and can run asynchronously, completing the data transmission of streaming and batch within a single operating system process.
[0085] The key technical aspect of this method is the component coordinator, which controls the execution order of streaming and batch read components, as well as the data queue of the write component, processing batch and streaming data sequentially according to a first-in, first-out (FIFO) rule. Furthermore, the batch read component needs to generate a data snapshot in the source database before reading data to prevent data modifications to the source database during batch data transfer from causing conflicts between streaming and batch data.
[0086] The stream-batch integrated write component needs to maintain a data queue internally to receive data sent from upstream components. The write component needs to process the data in the queue in an orderly manner according to the first-in-first-out rule.
[0087] like Figure 2 As shown, the system of this application includes four modules: a component coordinator, a batch read component, a streaming read component, and a stream-batch integrated write component. The execution order of each module is as follows:
[0088] The component coordinator starts first, and it first notifies the streaming read component to begin initialization.
[0089] The streaming read component begins executing initialization logic. In addition to connecting to the database, the core task is to obtain the current CDC (Change Data Capture) offset (or checkpoint, position, etc.) of the source database and save the offset.
[0090] The component coordinator starts and runs the batch read component (the batch read component will also be initialized before it starts running, but it is just a routine operation such as connecting to the database, and there is nothing special about it, so it will be omitted and will not be described in detail).
[0091] The batch read component starts reading batch data from the source database. Before reading the data, a data snapshot needs to be generated in the source database. Then, the data in the snapshot is read in batches. Data operations in the source database during data transmission will not affect the data in the snapshot. Then, the data is encapsulated according to a unified standard.
[0092] The batch read component sends the encapsulated data to the data queue of the stream-batch write component, and then continues the next round of data reading, encapsulation and sending, repeating until all data has been read and sent.
[0093] The batch write component retrieves a batch of data from the batch read component from the data queue, inserts it into the target database, and then processes the next batch of data until all the batch data in the queue has been inserted.
[0094] The batch read component notifies the component coordinator that all batch data has been read and sent, and then begins to self-destruct, releasing database connections, threads and other resources before the batch read component stops running.
[0095] The component coordinator begins running the streaming read component;
[0096] After the streaming read component starts running, it first collects data addition, modification, and deletion operations from the offset saved in step (2), and then encapsulates the data according to a unified standard. Unlike batch data, streaming data usually contains only one record at a time;
[0097] The streaming read component sends the encapsulated data to the data queue of the streaming batch write component, and then continues the next round of data reading, encapsulation, and sending. Since streaming data only has a beginning, under normal circumstances, unless external intervention stops the task, the streaming read component will continue to run and will not end.
[0098] The stream-batch write component retrieves a single data entry from the streaming read component and, depending on the data operation type, synchronously operates on the target database before processing the next data entry. Because the streaming data source continuously sends data to the data queue, the stream-batch write component will also run continuously without stopping.
[0099] The above is the complete process of the entire stream-batch integrated data transmission. Although steps (5) and (6) and steps (10) and (11) have a specific order, they can be executed asynchronously to improve data transmission efficiency through read-write separation.
[0100] By adopting the technical solution of this application, users can reduce the configuration of multiple data transmission tasks, merging streaming and batch tasks into one, simplifying task configuration operations and reducing operation and maintenance costs. Since this method achieves integrated streaming and batch processing within a single process, it is more lightweight than a distributed architecture, saving deployment costs. Furthermore, the data ordering guaranteed by this method perfectly integrates the two transmission methods, ensuring that data transmission is not duplicated, lost, or fails.
[0101] The technical solution presented in this application is limited to a single operating system process, making it more lightweight compared to distributed systems. However, it's easy to understand that the technical solution presented in this application can also be implemented using a distributed cluster of nodes.
[0102] Figure 2 This is a block diagram illustrating a data transmission system according to an exemplary embodiment. (Refer to...) Figure 2 The system includes: a streaming read component, a batch read component, a component coordinator, and a write component.
[0103] The component coordinator is used to start the streaming read component so that the streaming read component connects to the source database and obtains the current offset of the source database.
[0104] The component coordinator is also used to initiate the batch read component so that the batch read component and the write component transfer batch data from the source database to the target database; the batch data is data determined based on the current offset.
[0105] After the batch data transfer is completed, the batch read component stops operating.
[0106] The component coordinator is also used to initiate the streaming read component so that the streaming read component and the write component transfer incremental data from the source database to the target database.
[0107] Regarding the system in the above embodiments, the specific steps for each module to perform operations have been described in detail in the embodiments related to the method, and will not be elaborated further here. Each module in the above data transmission system can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0108] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0109] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.
[0110] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.
[0111] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0112] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0114] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0116] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A data transmission method, characterized in that, Includes the following steps: Start the streaming read component to connect to the source database, obtain the current offset of the source database, and save the current offset; The batch read component is activated to transfer a batch of data from the source database to the target database; the batch data is determined based on the current offset. After the batch data transfer is completed, the batch read component stops operating; Restart the streaming read component to transfer incremental data from the source database to the target database, starting from the saved current offset.
2. The method according to claim 1, characterized in that, Transferring batch data from the source database to the target database involves the following steps: Generate a data snapshot in the source database; Data is read in batches from the data snapshot and then transferred to the target database.
3. The method according to claim 2, characterized in that, Transferring the read data to the target database includes the following steps: The read data is encapsulated according to a unified standard; The packaged data is sent to a preset data queue, and then the next batch of data is read, packaged, and sent until all batches of data have been processed. Transfer the data in the data queue to the target database.
4. The method according to claim 3, characterized in that, Transferring data from the data queue to the target database includes the following steps: Take a batch of data from the data queue and insert it into the target database; Then process the next batch of data until all batches of data in the data queue have been inserted.
5. The method according to any one of claims 1-4, characterized in that, The batch read component stops operating, including the following steps: The batch read component sends a message to the component coordinator, notifying that all batch data has been transferred. Then the batch read component begins to self-destruct, releasing database connections and thread resources before stopping operation.
6. The method according to any one of claims 1-4, characterized in that, Transferring incremental data from the source database to the target database involves the following steps: The incremental data is determined based on the current offset, and one incremental data point is collected each time. Incremental data is collected from the source database, and the collected incremental data is encapsulated and sent to the data queue. Transfer incremental data from the data queue to the target database.
7. The method according to claim 6, characterized in that, The collected incremental data is encapsulated and sent to the data queue, including the following steps: The collected incremental data is encapsulated according to a unified standard, with one incremental data entry encapsulated each time. The encapsulated incremental data is sent to the data queue, and then the next incremental data is collected, encapsulated, and sent. The streaming read component continues to run unless a stop command is received.
8. The method according to claim 6, characterized in that, Transferring incremental data from the data queue to the target database includes the following steps: Retrieve one incremental data entry from the data queue; Synchronize operations on the target database based on the type of incremental data operation. Process all incremental data in the data queue sequentially.
9. The method according to claim 8, characterized in that, The incremental data includes data from additions, modifications, and / or deletions.
10. A data transmission system, characterized in that, include: Streaming read component, batch read component, component coordinator, and write component; The component coordinator is used to start the streaming read component so that the streaming read component connects to the source database, obtains the current offset of the source database, and saves the current offset. The component coordinator is also used to initiate the batch read component, so that the batch read component and the write component transfer batch data from the source database to the target database; the batch data is data determined based on the current offset; After the batch data transfer is completed, the batch read component stops operating; The component coordinator is also used to restart the streaming read component so that the streaming read component and the write component transfer incremental data from the source database to the target database, starting from the saved current offset.