Data integration task breakpoint continuous running method based on check point mechanism
By introducing a checkpoint mechanism in the data integration task, the task breakpoint continues to run, the problem of resource waste and time cost increase caused by task interruption in the existing technology is solved, and the stability and reliability of data integration tasks are improved.
Patent Information
- Application Number
- CN202411036293.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
AI Technical Summary
Existing data integration technologies need to be re-execute from scratch when tasks are interrupted, resulting in waste of computing resources and increased time costs.
The data integration task breakpoint running method based on the checkpoint mechanism is adopted. By setting checkpoints during the task execution and resuming execution from the nearest checkpoint when interrupted, the task breakpoint running is achieved.
It effectively improves the stability and reliability of data integration tasks, reduces the repeated extraction and calculation of data during task recovery, and avoids the waste of resources and time costs of re-executing tasks from scratch.
Smart Images

Figure CN118963950A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data integration task execution methods, and in particular to a data integration task breakpoint resumption method and device based on a checkpoint mechanism. Background Art
[0002] In existing data integration technologies, when a data integration task encounters a failure or interruption during execution (for example, a task interruption caused by system failure, network interruption or other unpredictable factors), it is usually necessary to re-execute the entire task from the beginning, which results in a large waste of computing resources and increased time costs. Summary of the invention
[0003] In order to overcome the above-mentioned defects of the existing methods, the present invention proposes a new data integration task breakpoint resumption method based on the checkpoint mechanism. By introducing the checkpoint mechanism, the method can set checkpoints during task execution and resume execution from the most recent checkpoint when the task is interrupted, thereby realizing task breakpoint resumption and achieving the purpose of improving the stability and reliability of data integration tasks.
[0004] Explanation of terms
[0005] Data integration: Data integration refers to the process of combining data from different sources, formats or properties into a unified data view or repository. This process usually involves steps such as data extraction, transformation and loading (ETL) to achieve data integration, sharing and consistency. Data integration is the basis for building enterprise-level data warehouses, implementing business intelligence (BI) and big data analysis applications.
[0006] Checkpoint: In data processing, system recovery or computing tasks, a checkpoint is a mechanism used to save the state information of a program or task at a specific moment so that it can be restored to that state and continue execution after a failure or pause. Checkpoint technology is widely used in database management, distributed computing, operating systems and applications.
[0007] Breakpoint resume: Also known as breakpoint resume or breakpoint recovery, it is a function that can save the current execution state and resume execution from that state at a later time when a specific condition (such as user pause, system error, or resource limitation) is encountered during program execution. This mechanism is particularly important in long-running tasks, complex calculations, or interactive applications that require frequent interruptions.
[0008] Task scheduling: Task scheduling system is a commonly used tool in the computer field to manage and execute various tasks. It can help developers to effectively arrange and execute tasks and improve the efficiency and performance of the system.
[0009] Specifically, the present invention provides a data integration task breakpoint continuation method based on a checkpoint mechanism, such as Figure 6 As shown, the method comprises the following steps:
[0010] S1. Pre-set a checkpoint strategy in the data integration module, set a checkpoint during task execution through the checkpoint strategy, and store the set checkpoint information in a storage medium;
[0011] S2. Create an integration task in the data integration module and configure the parameters and data extraction range of the integration task. When the integration task needs to be executed, the task scheduling module sends a trigger message to the data integration module, and the data integration module triggers the integration task;
[0012] S3. Parse the configuration parameters of the integration task, assemble the query SQL statement of the source table according to the source table and the data extraction range, and then segment the assembled query SQL statement according to the parallel parameters, split the integration task into multiple subtasks, and then distribute the split subtasks to the executor of the data integration module for execution;
[0013] S4. When the subtask starts to execute, it first checks and determines the checkpoint strategy, then generates checkpoint data according to the strategy, and sends the checkpoint data to Redis for storage; if the checkpoint strategy is not configured, the task extraction logic is directly executed, and the execution is finally completed or abnormally terminated;
[0014] S5. When an exception occurs during the execution of a subtask, causing the task to be interrupted, the integrated task reads the checkpoint data stored in Redis that is closest to the task interruption point, and then rebuilds and distributes the subtask based on the checkpoint data (to achieve task interruption recovery).
[0015] Furthermore, the checkpoint strategy described in step S1 of the data integration task breakpoint resumption method based on the checkpoint mechanism of the present invention includes the following three types:
[0016] Specify data step strategy: The integration task uses the specified step as a batch to extract or calculate data in batches. When the operation on the previous batch of data is successful, a checkpoint data of the batch of data is generated and sent to Redis for storage;
[0017] Specify the time interval strategy: The integration task records the time of data extraction. When the specified time is executed, a checkpoint data will be generated for the data successfully operated within the time period and sent to Redis for storage, and then the execution time will be calculated again;
[0018] Data step and time interval hybrid strategy: The integration task generates a checkpoint data according to the specified data step and time interval according to the corresponding strategy, whichever arrives first, and sends it to Redis for storage.
[0019] Furthermore, the parameters of the integration task described in step S2 of the data integration task breakpoint resumption method based on the checkpoint mechanism of the present invention include but are not limited to: source table, target table, parallel parameters, checkpoint strategy, and timing parameters;
[0020] The data extraction range includes a time type extraction range and a numerical type extraction range.
[0021] Furthermore, in the method for breakpoint resumption of data integration tasks based on a checkpoint mechanism of the present invention, the integration task is split into multiple subtasks in step S3. Except for the different data extraction ranges, the other configuration parameters of all subtasks are completely the same.
[0022] Furthermore, the checkpoint data generated in step S4 of the data integration task breakpoint resumption method based on the checkpoint mechanism of the present invention includes: the executor IP, the subtask data extraction range, the data range that has been successfully extracted, and the data range that is being extracted.
[0023] Furthermore, in the data integration task breakpoint resumption method based on the checkpoint mechanism of the present invention, the checkpoint data sent to Redis in step S4 is stored in the order of the size of the timestamp, and the system regularly cleans up the checkpoint data of the tasks that have been successfully restored or executed.
[0024] Furthermore, the data integration task breakpoint resumption method based on the checkpoint mechanism of the present invention comprises step S5:
[0025] S51. When an exception occurs during the execution of a subtask, causing the task to be interrupted, the integrated task reads the checkpoint data stored in Redis that is closest to the task interruption point;
[0026] S52. Reassemble the extraction SQL statement for building the integration task by checking the data extraction range recorded in the data point data and the data range that has been successfully extracted;
[0027] S53. Extract the SQL statements from the reassembled integration task and distribute them to the executor of the specified data integration module for execution, so as to realize task interruption recovery.
[0028] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned data integration task breakpoint resumption method based on the checkpoint mechanism are implemented.
[0029] On the other hand, the present invention also provides a data integration task breakpoint continuation device based on a checkpoint mechanism. When the device is running, the steps of the data integration task breakpoint continuation method based on the checkpoint mechanism are implemented. The device includes: a task scheduling module, a data integration module, a relational database, a data warehouse and a Redis cache module, wherein:
[0030] The task scheduling module is responsible for scheduling and triggering integrated tasks;
[0031] The data integration module is responsible for executing integration tasks and producing checkpoint data;
[0032] The Redis cache module is responsible for storing the checkpoint data of the task for query when the task is restored;
[0033] Relational databases and data warehouses are responsible for providing carriers for data extraction and writing in data integration tasks.
[0034] Furthermore, the relational databases described in the data integration task breakpoint resumption device based on the checkpoint mechanism of the present invention include but are not limited to: mysql, oracle, postgreSql.
[0035] In summary, the data integration task breakpoint resumption method based on the checkpoint mechanism of the present invention has the following advantages:
[0036] (1) The method of the present invention realizes the breakpoint resumption of data integration tasks by introducing a Redis-based checkpoint mechanism.
[0037] (2) The method of the present invention provides three checkpoint strategies, namely, specified data step size, specified time interval, and mixed mode, which can effectively reduce the amount of data repeatedly extracted and calculated during task recovery.
[0038] (3) Through the method of the present invention, when a task is interrupted during execution, execution can be resumed from the most recent checkpoint, avoiding the waste of resources and increased time cost of re-executing the entire task from the beginning.
[0039] (4) The method of the present invention ensures the reliability and availability of the checkpoint information by regularly updating the checkpoint information and storing it in a reliable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0041] Figure 1Schematic diagram of the system architecture of the device of the present invention.
[0042] Figure 2 Schematic diagram of the logical architecture of the method of the present invention.
[0043] Figure 3 Schematic diagram of key data flow of the method of the present invention.
[0044] Figure 4 It is an execution flow chart of the implementation process of the method of the present invention.
[0045] Figure 5 The present invention is an interrupt recovery flow chart in the implementation process of the method of the present invention.
[0046] Figure 6 The figure is a flow chart of the overall implementation of the method of the present invention. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The present invention can also be implemented or applied through other different specific implementation methods. The details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.
[0048] At the same time, it should be understood that the protection scope of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the protection scope of the present invention.
[0049] Embodiment: A method for resuming data integration tasks based on a checkpoint mechanism
[0050] (I) The system architecture of the device of the present invention is as follows Figure 1 shown.
[0051] The overall system architecture of the device of the present invention can be divided into three layers: task scheduling, data integration, and data persistence. It consists of five modules: task scheduling, data integration, relational database, data warehouse, and Redis cache. In the architecture diagram of this embodiment, only three types of relational databases are enumerated: mysql, oracle, and postgreSql. In fact, they are not limited to these three types. The device and method are also applicable to other types of relational databases. The functions of each module in the device of the present invention are as follows:
[0052] (1) The task scheduling module is responsible for scheduling and triggering integrated tasks;
[0053] (2) The data integration module is responsible for executing integration tasks and producing checkpoint data;
[0054] (3) The Redis cache module is responsible for storing the task's checkpoint data for query when the task is restored;
[0055] (4) Relational databases and data warehouses are responsible for providing a carrier for data extraction and writing in data integration tasks.
[0056] (II) The logical structure of the method of the present invention is as follows Figure 2 As shown, including:
[0057] (1) Task creation phase: First, the integration task is created in the data integration module, and the source table, target table, data extraction range, parallel parameters, checkpoint strategy, timing parameters and other parameters are configured for the integration task. Then the integration task is hosted in the task scheduling module and waits to be called at the specified time. When the integration task is ready to be executed, the task scheduling module sends a message to the data integration module to trigger the execution of the integration task. After the data integration module receives the task triggering message, the integration task enters the parameter parsing phase.
[0058] (2) Parameter parsing stage: First, parse the configuration parameters of the integration task, assemble the query SQL statement of the source table according to the source table and the data extraction range (the extraction range supports two types: time and value); then, segment the assembled SQL statement according to the parallel parameter. Taking the numerical extraction range as an example, assuming that the task needs to extract data with primary keys between 1 and 100,000, and the parallel parameter is 10, then the extraction range will be segmented with a step size of 10,000, splitting the task into 10 subtasks, and then distributing the split subtasks to the data integration executor for execution, and the integration task enters the execution stage.
[0059] (3) Task execution phase: When executing a subtask, it will first determine whether a checkpoint strategy is configured (checkpoint strategies include three groups of strategies: specified data step size, specified time interval, and mixed data step size and time interval. The execution logic is shown in the implementation process section). If there is no checkpoint strategy, the task extraction logic is directly executed; if there is a checkpoint strategy, the checkpoint data is generated according to the specified strategy and sent to Redis for storage. If the above-mentioned exceptions do not occur during the task execution, the task will end normally; if an exception occurs, the task will enter the interrupt recovery phase.
[0060] (4) During the task interruption and recovery phase, the integrated task will read the most recent checkpoint data stored in Redis, and then perform task interruption recovery based on the relevant data in the checkpoint (see the Implementation Process section for the interruption and recovery process).
[0061] (5) The checkpoint data stored in Redis is stored in order of timestamp size, and the system periodically cleans up the checkpoint data of tasks that have been successfully restored or executed.
[0062] (III) The key data flow of the method of the present invention is as follows Figure 3 As shown, including:
[0063] When creating an integration task, key parameters such as the task's data extraction range, parallel parameters, and checkpoint strategy are configured. When a task is triggered, the overall parameter information of the task is first sent to Redis storage, and then the task is split according to the data extraction range and parallel parameters; the resulting subtasks have the same parameters except for the extraction range. When the subtask is executed, checkpoint data is generated according to the checkpoint strategy and sent to Redis storage. The checkpoint data includes the executor IP, subtask data extraction range, data range that has been successfully extracted, and data range that is being extracted. When the task is sent abnormally for interruption recovery operations, the most recent checkpoint information in Redis is read to rebuild and distribute the subtasks.
[0064] (IV) Implementation process of the method of the present invention
[0065] The integration task involved in the present invention, regardless of whether the parallel parameters are configured, is ultimately executed in the form of subtasks. The implementation process is as follows: Figure 4 , Figure 5 As shown, it is divided into execution process and interrupt recovery process:
[0066] Execution process: Figure 4 As shown, when the subtask starts to execute, the business logic of data integration will start directly, and it will determine whether a checkpoint strategy is configured. If there is no checkpoint strategy, the task will execute normally and eventually complete or terminate abnormally; if a checkpoint strategy is configured, it will determine which strategy it is, and execute different checkpoint data production logics according to the checkpoint strategy. The checkpoint strategies used in the method of the present invention include the following three types:
[0067] (1) Specify data step size strategy: The data integration task will extract or calculate data in batches with a specified step size. When the operation of the previous batch of data is successful, the checkpoint data of the batch of data is generated and sent to Redis for storage. Assuming the specified step size is 1000, a checkpoint data will be generated for every 1000 data extracted.
[0068] (2) Specify the time interval strategy: The data integration task will record the time of data extraction. When the specified time is executed, a checkpoint data will be generated for the data successfully operated within the time period and sent to Redis for storage, and then the execution time will be calculated again. Assuming the specified time interval is 60 seconds, a checkpoint data will be generated every 60 seconds.
[0069] (3) Hybrid strategy of data step and time interval: The data integration task will generate a checkpoint data according to the corresponding strategy based on the specified data step and time interval. Whichever arrives first will be sent to Redis for storage. Assuming the specified step is 10,000 and the time interval is 10 seconds, a checkpoint data will be generated when the operation of 10,000 data is completed first or the time interval of 10 seconds is reached first.
[0070] Interrupt recovery process: Figure 5 As shown in the figure, when a subtask is abnormally interrupted, the interruption recovery operation is performed. First, it is determined whether the current task has a checkpoint strategy. If not, the recovery fails. If a checkpoint strategy is configured, the most recent checkpoint data of the task is read from Redis. The extraction SQL statement of the integration task is reassembled through the data extraction range recorded in the checkpoint and the data range that has been successfully extracted (for example, the extraction range of the subtask is 10000-20000, and 10000-15000 is successfully extracted, then the extraction range of the reassembled interruption recovery task is 15000-20000). Then it is distributed to the executor of the data integration module with the specified IP for execution.
[0071] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical contents disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments of equivalent changes. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A data integration task breakpoint resumption method based on a checkpoint mechanism, characterized in that: The method comprises: S1. Pre-set a checkpoint strategy in the data integration module, set a checkpoint during task execution through the checkpoint strategy, and store the set checkpoint information in a storage medium; S2. Create an integration task in the data integration module and configure the parameters and data extraction range of the integration task. When the integration task needs to be executed, the task scheduling module sends a trigger message to the data integration module, and the data integration module triggers the integration task; S3. Parse the configuration parameters of the integration task, assemble the query SQL statement of the source table according to the source table and the data extraction range, and then segment the assembled query SQL statement according to the parallel parameters, split the integration task into multiple subtasks, and then distribute the split subtasks to the executor of the data integration module for execution; S4. When the subtask starts to execute, it first checks and determines the checkpoint strategy, then generates checkpoint data according to the strategy, and sends the checkpoint data to Redis for storage; S5. When an exception occurs during the execution of a subtask, causing the task to be interrupted, the integrated task reads the checkpoint data stored in Redis that is closest to the task interruption point, and then rebuilds and distributes the subtask based on the checkpoint data.
2. The data integration task breakpoint resumption method based on the checkpoint mechanism according to claim 1 is characterized in that: The checkpoint strategies described in step S1 include the following three types: Specify data step strategy: The integration task uses the specified step as a batch to extract or calculate data in batches. When the operation on the previous batch of data is successful, a checkpoint data of the batch of data is generated and sent to Redis for storage; Specify the time interval strategy: The integration task records the time of data extraction. When the specified time is executed, a checkpoint data will be generated for the data successfully operated within the time period and sent to Redis for storage, and then the execution time will be calculated again; Data step and time interval hybrid strategy: The integration task generates a checkpoint data according to the specified data step and time interval according to the corresponding strategy, whichever arrives first, and sends it to Redis for storage.
3. The data integration task breakpoint resumption method based on the checkpoint mechanism according to claim 1 is characterized in that: The parameters of the integration task described in step S2 include: source table, target table, parallel parameters, checkpoint strategy, and timing parameters; The data extraction range includes a time type extraction range and a numerical type extraction range.
4. The data integration task breakpoint resumption method based on the checkpoint mechanism according to claim 1 is characterized in that: In step S3, the integration task is split into multiple subtasks. Except for the different data extraction scopes, the other configuration parameters of all subtasks are exactly the same.
5. The data integration task breakpoint resumption method based on the checkpoint mechanism according to claim 1 is characterized in that: The checkpoint data generated in step S4 includes: the executor IP, the subtask data extraction range, the data range that has been successfully extracted, and the data range that is being extracted.
6. The data integration task breakpoint resumption method based on the checkpoint mechanism according to claim 1 is characterized in that: The checkpoint data sent to Redis in step S4 is stored in the order of the size of the timestamp, and the system regularly cleans up the checkpoint data of the tasks that have been successfully restored or executed.
7. The data integration task breakpoint resumption method based on the checkpoint mechanism according to claim 1 is characterized in that: Step S5 includes: S51. When an exception occurs during the execution of a subtask, causing the task to be interrupted, the integrated task reads the checkpoint data stored in Redis that is closest to the task interruption point; S52. Reassemble the extraction SQL statement for building the integration task by checking the data extraction range recorded in the data point data and the data range that has been successfully extracted; S53. Extract the SQL statements from the reassembled integration task and distribute them to the executor of the specified data integration module for execution, so as to realize task interruption recovery.
8. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the data integration task breakpoint resumption method based on a checkpoint mechanism as described in any one of claims 1 to 7.
9. A data integration task breakpoint resumption device based on a checkpoint mechanism, characterized in that: When the device is running, the steps of the data integration task breakpoint resumption method based on the checkpoint mechanism according to any one of claims 1 to 7 are implemented, and the device includes: a task scheduling module, a data integration module, a relational database, a data warehouse and a Redis cache module, wherein: The task scheduling module is responsible for scheduling and triggering integrated tasks; The data integration module is responsible for executing integration tasks and producing checkpoint data; The Redis cache module is responsible for storing the checkpoint data of the task for query when the task is restored; Relational databases and data warehouses are responsible for providing carriers for data extraction and writing in data integration tasks.
10. The data integration task breakpoint resumption device based on the checkpoint mechanism according to claim 9, characterized in that: The relational databases include: mysql, oracle, postgreSql.
Citation Information
Cited By
A method and system for ensuring data consistency in an ETL process
CN122733976A
