A method and system for resuming and recovering from a fault in a time-series data conversion pipeline

Abstract

The application discloses a breakpoint continuation and fault recovery method of a time sequence data conversion channel system, which comprises the following steps: an embedded key value storage engine is used as a breakpoint database, and data synchronization points are recorded in three levels of granularity, namely, tasks, jobs and subtasks; a persistent queue component is arranged before a data writing end; after the data writing end successfully writes data into a target database, a confirmation mechanism is used to feed back writing results to an upstream, and after a successful confirmation is received, point information of corresponding subtasks in the breakpoint database is updated, so that atomicity of data landing and point updating is ensured; when a system process is restarted, last successful points of each subtask are read from the breakpoint database, unprocessed data is read from a last consumption position of the persistent queue, or a data source is notified to resend data from the recorded point, so that automatic breakpoint continuation is realized; when a network link interruption is detected, an exponential backoff retry strategy is used to automatically reestablish a connection.

Description

Technical Field

[0001] This invention relates to the field of distributed data synchronization and streaming data processing technology, and in particular to a method and system for implementing automatic resume after process restart, automatic recovery after network interruption, state persistence and idempotency guarantee in a data transformation channel (ETL) system for time-series databases. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), industrial automation, smart energy, and other fields, the generation speed and scale of time-series data continue to grow. Enterprises typically need to synchronize time-series data from various heterogeneous data sources, such as OPC UA / DA, MQTT, Kafka, relational databases (MySQL / PostgreSQL / Oracle / MSSQL), and file systems (CSV / Parquet), to a dedicated time-series database for storage and analysis in real time.

[0003] Existing general-purpose data synchronization and ETL systems (such as Apache DataX, Spark Streaming, Flink, etc.) have the following shortcomings when handling such scenarios:

[0004] State loss problem: After a process restarts, the synchronization state in memory is lost, requiring synchronization to start from scratch or relying on manual specification of the recovery point, which is inefficient and prone to errors.

[0005] Network interruption vulnerability: Network interruptions can cause synchronization tasks to fail directly, lacking automatic reconnection and resume capabilities, requiring manual intervention for recovery.

[0006] Data consistency risk: If the system crashes after some data is successfully written, the range of data that has been written cannot be accurately determined during recovery, resulting in data duplication or omission.

[0007] Lack of unified state coordination: Different data source plugins implement state management independently, lacking a unified offset management interface, which increases system complexity and maintenance costs.

[0008] Large-scale retransmission is costly: after a transmission interruption, there is a lack of intermediate buffers, and all data needs to be retrieved from the data source again, resulting in huge network and computing overhead.

[0009] Insufficient adaptation to time-series scenarios: General ETL systems lack specific optimizations for the characteristics of time-series data (such as the inherent order of timestamps, parallel writing across multiple tables and devices, etc.).

[0010] Therefore, the industry urgently needs a data synchronization system specifically designed for time-series data scenarios that supports automatic breakpoint resume and fault recovery. Summary of the Invention

[0011] To address the aforementioned deficiencies in the prior art, this invention provides a method and system for resuming interrupted transmissions and recovering from faults in a time-series data conversion channel system.

[0012] According to a first aspect of the present invention, a method for resuming interrupted transmission and recovering from faults in a timing data conversion channel system includes the following steps:

[0013] Step S1 of building a hierarchical data processing pipeline specifically involves establishing a streaming processing pipeline consisting of three levels: data source, data transformation, and data writing. Data is transmitted between the three levels of data source, data transformation, and data writing via an inter-process communication (IPC) protocol based on the Apache Arrow columnar memory format.

[0014] Step S2 of establishing a multi-granularity breakpoint persistence mechanism specifically involves using an embedded key-value storage engine as the breakpoint database, automatically recording data synchronization points according to three levels of granularity: task, job, and subtask. The information of the data synchronization points is persistently stored on the local disk in the form of key-value pairs, where the key is the subtask identifier and the value is the synchronization point at which the subtask was last successfully submitted.

[0015] Step S3 of implementing write-ahead persistent queue buffer specifically involves setting up a persistent queue component before the data writing end. The persistent queue component uses a write-ahead log mechanism to write the received data to the disk file in CRC check frame format, thereby achieving safe buffering of data from memory to disk.

[0016] Step S4 of the two-stage site submission is as follows: after the data writing end successfully writes the data to the target database, it sends the writing result back to the upstream through the confirmation mechanism. After receiving the successful confirmation, it updates the site information of the corresponding subtask in the breakpoint database to ensure the atomicity of data landing and site update.

[0017] The automatic recovery step S5 when the process restarts is as follows: when the system process restarts, the last successful position of each subtask is read from the breakpoint database, the unprocessed data is continued to be read from the last consumption position of the persistent queue, or the data source is notified to resend the data from the recorded position to realize automatic breakpoint resume.

[0018] Step S6, which involves reconnecting when a link is interrupted, specifically involves automatically re-establishing the connection using an exponential backoff retry strategy when a network link interruption is detected, and responding with a cancellation token during the retry period to support graceful shutdown.

[0019] Preferably, in step S2, the breakpoint database supports open operations with retries. When the underlying storage engine fails to open due to concurrent access or file locking, it will retry up to 5 times, with an interval of 1 second between each attempt.

[0020] Preferably, in step S3, the persistent queue frame format includes three parts: payload length, CRC32 checksum, and payload data. The checksum is verified during reading to detect file corruption and to support periodic cleanup of consumed log files.

[0021] Preferably, in step S4, the structured confirmation is encoded using Arrow IPC stream format, which includes three fields: return code, message, and context, and can accurately reflect the writing result and error details of each piece of data.

[0022] Preferably, in step S6, the formula for calculating the backoff time of the exponential backoff retry strategy is: retry_interval = min(initial_backoff × 2^(retries-1), max_backoff), and the cancellation token is listened for during each backoff waiting period. If a cancellation signal is received, the retry is stopped immediately and the process returns.

[0023] Where retry_interval is the backoff time, initial_backoff is the initial backoff time, retries is the number of retries, max_backoff is the maximum backoff time, and min() is the minimum value function.

[0024] According to a second aspect of the present invention, a breakpoint resume and fault recovery system for a timing data conversion channel system includes:

[0025] The breakpoint management module is implemented using an embedded key-value storage engine. It provides a three-level granularity interface for reading and writing breakpoints by task / job / subtask, and supports single setting, batch setting, query, deletion and full export operations. It supports both synchronous and asynchronous access modes.

[0026] The persistent queue module is implemented using a file-based write-ahead log queue. It consists of independent readers and writers. The writer writes data frames to disk according to a configurable block size and synchronization interval and periodically executes the file synchronization write command fsync. The reader reads data frames from disk in batches and supports timeout waiting and cancellation operations.

[0027] The IPC (Inter-Process Communication) module implements inter-process data transmission based on the Apache Arrow IPC streaming protocol. It includes an IPC stream reader and an acknowledgment writer, supports efficient zero-copy data exchange in Arrow RecordBatch format, and a remote data transmission channel based on gRPC / Arrow Flight.

[0028] The retry connection module includes a channel factory abstract interface and retry configuration, implementing an exponential backoff reconnection mechanism linked to the cancellation token, supporting both limited-number retries and unlimited retries modes;

[0029] The health monitoring module implements a finite state machine with ten states: initialization, ready, idle, active, waiting, busy, occasional error, source error, write error, and fatal error. It automatically determines the health status of the task through indicators such as error rate and queue depth within a sliding window.

[0030] The task orchestration module enables routing and orchestration of various data sources to the target based on the data source identification protocol. It injects breakpoint information into the data source connection configuration through DSN parameters and supports failover strategies to split a single task into multiple subtasks for parallel execution.

[0031] Preferably, after receiving a cancellation signal, the writer of the persistent queue module consumes the remaining messages in the channel before performing the final disk synchronization to ensure that no data is lost when the process is closed.

[0032] Preferably, the IPC data frame header in the IPC communication module includes an 8-byte acknowledgment status field, an 8-byte query identifier, and an 8-byte batch counter. The first byte of the acknowledgment status field is encoded as follows: 0x00 indicates a request, 0x01 indicates that the data has been received, 0x02 indicates that the data has been processed, 0x03 indicates that the data has been discarded, 0xFE indicates that the stream has ended, and 0xFF indicates that the data has been decoded.

[0033] Preferably, the breakpoint resume and fault recovery system of the time-series data conversion channel system of the present invention further includes an indicator acquisition module, which provides real-time statistical indicators for the number of messages read, the number of messages written, the number of received acknowledgments, and the number of sent batches of the persistent queue.

[0034] Preferably, the heterogeneous data sources supported by the task orchestration module include OPC UA / DA, MQTT, Kafka, Pulsar, InfluxDB, OpenTSDB, MySQL, PostgreSQL, Oracle, MSSQL, MongoDB, CSV, Parquet, ORC, and SparkplugB protocols. Each data source plugin independently manages its own breakpoint and recovery logic and accesses the breakpoint database through a unified task option interface.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Zero-intervention breakpoint resume: Through multi-granularity breakpoint automatic persistence and two-stage site commit, the process can automatically recover to the last successful site within milliseconds after restarting, without the need for manual specification of the recovery site.

[0037] Zero data loss guarantee: The write-ahead persistent queue writes data to disk before it enters the write pipeline. Combined with the graceful shutdown mechanism upon exit (consuming the remaining messages and performing the final fsync), it ensures that no data is lost in any abnormal scenario.

[0038] Highly available connection recovery: The deep integration of the exponential backoff reconnection mechanism and cancellation tokens enables rapid and automatic connection recovery during network outages and immediate response to shutdown commands when the system is shut down, balancing availability and controllability.

[0039] Unified pluggable extensibility: Through the abstract Trait interface and unified task orchestration framework, new data sources or targets only need to implement the standard interface to automatically obtain the ability to resume interrupted downloads and recover from failures, reducing development costs.

[0040] Precise observability: The ten-state health state machine, structured indicator collection, and hierarchical confirmation mechanism provide maintenance personnel with complete means of task operation visualization and fault location.

[0041] Native time-series data adaptation: Given the ordered nature of timestamps in time-series data, positions can be naturally represented by timestamps. Combined with a subtask splitting strategy for parallel writing of multiple tables, the scenario characteristics of time-series data are fully utilized to improve synchronization efficiency. Attached Figure Description

[0042] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0043] Figure 2 This is a diagram illustrating the breakpoint database storage structure and recovery process of the present invention;

[0044] Figure 3 This is a persistent queue frame format and read / write timing diagram of the present invention;

[0045] Figure 4 This is a comparison diagram of the three confirmation modes of IPC communication in this invention;

[0046] Figure 5 This is the timing diagram of the exponential backoff reconnection of the present invention;

[0047] Figure 6 This is the state machine transition diagram for health monitoring in this invention;

[0048] Figure 7 This is a flowchart illustrating the end-to-end fault recovery process of this invention.

[0049] Figure 8 This is a schematic diagram of the breakpoint resume and fault recovery system of the time-series database synchronization system of the present invention. Detailed Implementation

[0050] See Figure 8 The present invention provides a breakpoint resume and fault recovery system for a timing data conversion channel (ETL) system, comprising:

[0051] The breakpoint management module is implemented using an embedded key-value storage engine. It provides a three-level granularity read / write interface for breakpoints, including task, job, and subtask. It supports single-entry setting, batch setting, querying, deletion, and full export operations, and supports both synchronous and asynchronous access modes.

[0052] The persistent queue module is implemented using a file-based write-ahead log queue. It consists of independent readers and writers. The writer writes data frames to disk according to a configurable block size and synchronization interval and performs fsync periodically. The reader reads data frames from disk in batches and supports timeout waiting and cancellation operations.

[0053] The IPC communication module implements inter-process data transmission based on the Apache Arrow IPC streaming protocol. It includes an IPC stream reader (IpcReader) and an acknowledgment writer (AckWriter), supports efficient zero-copy data exchange in Arrow RecordBatch format, and a remote data transmission channel based on gRPC / Arrow Flight.

[0054] The retry connection module includes the ChannelFactory abstract interface and RetryConfig, which implements an exponential backoff reconnection mechanism linked to the cancellation token, supporting both limited-number retries and unlimited retries modes.

[0055] The health monitoring module implements a finite state machine with ten states: Initial, Ready, Idle, Active, Pending, Busy, Bounce, SourceError, SinkError, and Fatal. It automatically determines the health status of tasks by using indicators such as error rate and queue depth within a sliding window.

[0056] The task orchestration module, based on the Data Source Identifier (DSN) protocol, enables routing and orchestration of various data sources to the target. It injects breakpoint information into the data source connection configuration through DSN parameters and supports failover strategies to split a single task into multiple subtasks for parallel execution.

[0057] After receiving a cancellation signal, the writer of the persistent queue module of the present invention first consumes the remaining messages in the channel and then performs the final disk synchronization (sync_data) to ensure that no data is lost when the process is closed.

[0058] The IPC data frame header in the IPC communication module of the present invention includes an 8-byte acknowledgment status field, an 8-byte query identifier (QID), and an 8-byte batch counter. The first byte of the acknowledgment status field is encoded as follows: 0x00 indicates a request, 0x01 indicates that it has been received, 0x02 indicates that it has been processed, 0x03 indicates that it has been discarded, 0xFE indicates that the stream has ended, and 0xFF indicates a decoding error.

[0059] The breakpoint resume and fault recovery system of the time-series data conversion channel system of the present invention also includes an indicator acquisition module, which provides real-time statistical indicators for the number of messages read, messages written, received acknowledgments, and sent batches of the persistent queue.

[0060] The task orchestration module of this invention supports heterogeneous data sources including OPC UA / DA, MQTT, Kafka, Pulsar, InfluxDB, OpenTSDB, MySQL, PostgreSQL, Oracle, MSSQL, MongoDB, CSV, Parquet, ORC, and SparkplugB protocols. Each data source plugin independently manages its own breakpoint and recovery logic and accesses the breakpoint database through a unified task options (TaskOpts) interface.

[0061] See Figures 1-7 The present invention discloses a method for breakpoint resumption and fault recovery in a time-series data conversion channel (ETL) system, applicable to scenarios of synchronizing data from heterogeneous data sources to a time-series database, comprising the following steps:

[0062] Step S1 of constructing a hierarchical data processing pipeline specifically involves establishing a streaming processing pipeline consisting of three levels: data source (Source), data transformation (Transform), and data writing (Sink). Data is transmitted between each level through an inter-process communication (IPC) protocol based on the Apache Arrow columnar memory format. The IPC protocol supports multiple acknowledgment modes, including three modes: none, code acknowledgment, and structured rich acknowledgment (Lush).

[0063] Step S2 of establishing a multi-granularity breakpoint persistence mechanism specifically involves using an embedded key-value storage engine as the breakpoint database (BreakpointDb), automatically recording data synchronization points at three levels of granularity: Task, Job, and SubTask. The point information is persistently stored on the local disk in the form of key-value pairs, where the key is the subtask identifier and the value is the synchronization point at which the subtask was last successfully submitted.

[0064] Step S3 of implementing write-ahead persistent queue buffering specifically involves setting up a persistent queue (PersistQueue) component before the data writing end (Sink). This component uses a write-ahead log (WAL) mechanism to write the received data to the disk file in CRC check frame format, supports file rotation and garbage collection (Vacuum), and achieves safe buffering of data from memory to disk.

[0065] Step S4 of the two-stage site submission is as follows: after the data writing end successfully writes the data to the target database, it sends the writing result back to the upstream through the acknowledgment mechanism (ACK). After receiving the successful acknowledgment, it updates the site information of the corresponding subtask in the breakpoint database to ensure the atomicity of data landing and site update.

[0066] The automatic recovery step S5 when the process restarts is as follows: when the system process restarts, the last successful position of each subtask is read from the breakpoint database, the unprocessed data is continued to be read from the last consumption position of the persistent queue, or the data source is notified to resend the data from the recorded position to realize automatic breakpoint resume.

[0067] Step S6 of intelligent reconnection when the link is interrupted specifically involves automatically re-establishing the connection using an exponential backoff retry strategy when a network link interruption is detected. The strategy includes a configurable initial backoff time, a maximum backoff time, and a maximum number of retries, and responds with a cancellation token during the waiting period for retry to support graceful shutdown.

[0068] In step S2 above, the breakpoint database supports open operations with retries. When the underlying storage engine fails to open due to concurrent access or file locking, it will retry up to 5 times, with an interval of 1 second between each attempt.

[0069] In step S3 above, the persistent queue frame format includes three parts: payload length, CRC32 checksum, and payload data. The checksum is verified during reading to detect file corruption, and periodic cleanup (Vacuum) of consumed log files is supported.

[0070] In step S4 above, the structured acknowledgment (Lush ACK) is encoded in Arrow IPC stream format and includes three fields: code, message, and context, which can accurately reflect the writing result and error details of each piece of data.

[0071] In step S6 above, the backoff time calculation formula for the exponential backoff retry strategy is: retry_interval = min(initial_backoff × 2^(retries-1), max_backoff). During each backoff waiting period, the cancellation token is listened for simultaneously. If a cancellation signal is received, the retry is stopped immediately and the process returns.

[0072] The technical details of the above-described technical solutions of the present invention will be described below.

[0073] 1. System Overall Architecture: The overall system architecture of this invention is as follows: Figure 1 As shown, a three-tier pipeline design of Source → Transform → Sink is adopted: data flows in from a heterogeneous data source (Source), undergoes optional data transformation (Transform), and is then written to the target time-series database (Sink). Running in parallel are the offset cache (OffsetsCache) and the persistent cache (PersistCache), which work together to maintain synchronization state. The underlying persistent storage is organized in a three-tier architecture of memory → disk → distributed storage.

[0074] 2. Breakpoint Database (BreakpointDb): The breakpoint database is the core component for point persistence in this system, and its implementation is as follows:

[0075] Storage engine selection: The SLED embedded key-value storage engine is adopted, which has the following characteristics:

[0076] It does not require a separate process to run; it is embedded in the application process.

[0077] Supports atomic read / write and batch operations;

[0078] Based on B+ tree indexes, it offers excellent read and write performance;

[0079] Supports concurrent access and thread safety.

[0080] Directory organization structure:

[0081]

[0082] Where DATA_DIR is the system data directory, task_id is the task identifier, and job_id is the job identifier. Each task-job combination has its own independent breakpoint database instance.

[0083] Key interface design:

[0084]

[0085] Fault-tolerant opening mechanism: The database opening operation supports automatic retry. When the underlying sled engine fails to open due to file locking or concurrency conflicts, the system will retry up to 5 times at 1-second intervals, which enhances the robustness in a multi-process environment.

[0086] 3. Persistent Queue (see PersistQueue) Figure 3 The persistent queue is a key component for ensuring data is not lost in the event of power failure. Each data frame consists of a payload length (4 bytes), a CRC32 checksum (4 bytes), and the actual payload (variable length). The CRC32 checksum is used to detect file corruption during reading.

[0087] Read / write separation architecture:

[0088] Writer: Receives upstream data and writes it in batches according to a configurable block size (chunk_size). It then flushes the data to disk using fsync, executed via a timer (default 3-second interval). Upon receiving a cancellation signal, it first consumes the remaining messages in the channel before performing the final sync_data operation to ensure data integrity.

[0089] Reader: Reads data frames from disk files in batches. It supports the read_util interface to implement batch reads with timeouts (waiting for at least min_batch_size data or reaching the timeout period), and periodically performs a vacuum operation to clean up consumed log files.

[0090] Breakpoint recovery (see) Figure 2 The read position of the persistent queue is recorded and serialized using EntryPosition and saved to the breakpoint database (key: persist_queue_breakpoint). Upon restart, the last read position is read from the breakpoint database, and consumption continues from that position using the ReadFrom::LastPosition strategy. If no breakpoint is found, the ReadFrom::Earliest strategy is used to start from the earliest position.

[0091] 4. IPC Communication and Acknowledgment Mechanism

[0092] This invention provides inter-process communication (IPC) based on the Apache Arrow IPC streaming protocol, achieving efficient zero-copy data transmission and a multi-level acknowledgment mechanism.

[0093] Data frame metadata structure (MessageMetadata): in sequence, it consists of confirmation status (8 bytes), query identifier (8 bytes), and batch count (8 bytes).

[0094] The meaning of the first byte (ack[0]) in the acknowledgment status field is as follows:

[0095]

[0096] Three confirmation modes, see Figure 4 :

[0097] None mode: No acknowledgment; the write end does not wait for feedback from the target database. Suitable for scenarios with extremely high performance requirements and where a small amount of data loss is acceptable.

[0098] Code mode: Simple integer return code confirmation, using 4-byte little-endian integer encoding. Suitable for scenarios where only success / failure needs to be known.

[0099] Lush Mode: Structured rich confirmation, encoded using Arrow IPC stream format, contains three fields: suitable for production environments requiring precise error diagnosis and localization.

[0100] ○code (Int32): Return code, 0 indicates success;

[0101] ○Message (Binary): Error message description;

[0102] ○context (Binary): Error context information.

[0103] Remote transmission channel: For cross-process and cross-machine data transmission, the system uses the gRPC / Arrow Flight protocol to achieve remote data exchange and supports connection keep-alive mechanisms such as TLS encryption, HTTP / 2 Keep-Alive, and TCP Keep-Alive.

[0104] 5. Indexed backoff intelligent reconnection mechanism, see [link / reference] Figure 5 .

[0105] When the connection to the remote agent or target database is interrupted, the system adopts the following reconnection strategy: Retry configuration (RetryConfig):

[0106]

[0107] Backoff algorithm:

[0108]

[0109] Each time a retry fails, the backoff time doubles until the maximum backoff limit is reached.

[0110] In conjunction with the cancellation token: During each backoff wait period, the system simultaneously listens for the CancellationToken. If a cancellation signal (such as a system shutdown command) is received during the wait, retries are immediately stopped and None is returned, achieving graceful shutdown.

[0111] Two retry modes:

[0112] • Limited retries (retry_forever = false): Returns an error after max_retries;

[0113] • Infinite retry (retry_forever = true): Retry continuously until success or a cancellation signal is received.

[0114] ChannelFactory Trait: Abstracts the connection establishment process into a Trait interface, supporting dependency injection and mock replacement in unit tests, thus improving the testability of the system.

[0115] 6. Health monitoring status machine

[0116] See Figure 6 This invention implements a finite state machine with 10 states to accurately describe the operational health status of a task. The finite state machine implements ten states: Initial, Ready, Idle, Active, Pending, Busy, Bounce, SourceError, SinkError, and Fatal.

[0117] Sliding window monitoring: Health metrics within the window are recorded using a circular buffer through a configurable health check window (default 60 seconds) and check interval (default 5 seconds). A state transition is triggered when the error rate exceeds a threshold or the queue depth exceeds the upper limit.

[0118] 7. Plug-in task orchestration

[0119] The system encapsulates task execution parameters through a unified TaskOpts structure:

[0120]

[0121] DSN Protocol Routing: The system automatically routes to the corresponding data synchronization implementation based on the DSN driver identifiers (such as mqtt: / / , opcua: / / , kafka: / / , taos: / / , etc.) of the source and destination ends.

[0122] Breakpoint injection: Breakpoint information is injected into the source connection configuration via the DSN's params parameter (append_breakpoints_in_dsn method), enabling the data source to start sending data from the specified point.

[0123] Failover: The system supports splitting a single data synchronization task into multiple subtasks based on the characteristics of the data source (such as splitting OPC by device group, or MQTT by topic). Each subtask is executed independently, and the failure of one subtask does not affect the normal operation of other subtasks.

[0124] 8. End-to-end recovery process

[0125] For the complete recovery process after system process restart or link interruption, please refer to [link / reference]. Figure 7 The recovery process includes: process startup / link recovery → reading TaskOpts configuration → opening BreakpointDb path → reading persistent queue position key → rebuilding persistent queue and starting reader / writer from LastPosition → establishing IPC / Flight connection with exponential backoff and automatic reconnection → notifying data source to resume transmission from the breakpoint → restoring data stream channel.

[0126] In summary, the core technologies of this invention include:

[0127] Automatic persistence of multi-granularity sites: Automatically records and persists synchronization sites at three levels of granularity: Task, Job, and SubTask / Table, without manual intervention.

[0128] Write-ahead persistent queue buffer: Set up a persistent queue based on the write-ahead log (WAL) mechanism before writing to ensure that the received data is safely written to disk first, and then sent to the target database asynchronously, reducing the risk of data loss.

[0129] Two-stage site commit protocol: After successful data writing, the result is first fed back through an acknowledgment mechanism, and then the site information is atomically updated to ensure that the site always reflects the actual writing progress.

[0130] Exponential backoff intelligent reconnection: When the link is interrupted, an exponential backoff retry strategy with an upper limit is used to automatically rebuild the connection, and it supports graceful shutdown in conjunction with cancellation tokens.

[0131] Plug-in Sink Abstraction and Unified State Management: Through a unified Trait abstraction interface, any data source plugin can seamlessly access breakpoint persistence and recovery capabilities.

[0132] Structured health monitoring: Through a ten-state finite state machine and sliding window index statistics, the health status of the task can be accurately determined and observed.

[0133] The present invention has the following beneficial effects:

[0134] Zero-intervention breakpoint resume: Through multi-granularity breakpoint automatic persistence and two-stage site commit, the process can automatically recover to the last successful site within milliseconds after restarting, without the need for manual specification of the recovery site.

[0135] Zero data loss guarantee: The write-ahead persistent queue writes data to disk before it enters the write pipeline. Combined with the graceful shutdown mechanism upon exit (consuming the remaining messages and performing the final fsync), it ensures that no data is lost in any abnormal scenario.

[0136] Highly available connection recovery: The deep integration of the exponential backoff reconnection mechanism and cancellation tokens enables rapid and automatic connection recovery during network outages and immediate response to shutdown commands when the system is shut down, balancing availability and controllability.

[0137] Unified pluggable extensibility: Through the abstract Trait interface and unified task orchestration framework, new data sources or targets only need to implement the standard interface to automatically obtain the ability to resume interrupted downloads and recover from failures, reducing development costs.

[0138] Precise observability: The ten-state health state machine, structured indicator collection, and hierarchical confirmation mechanism provide maintenance personnel with complete means of task operation visualization and fault location.

[0139] Native time-series data adaptation: Given the ordered nature of timestamps in time-series data, positions can be naturally represented by timestamps. Combined with a subtask splitting strategy for parallel writing of multiple tables, the scenario characteristics of time-series data are fully utilized to improve synchronization efficiency.

Claims

1. A method for breakpoint resumption and fault recovery in a time-series data conversion channel system, characterized in that, Includes the following steps: Step S1 of building a hierarchical data processing pipeline specifically involves establishing a streaming processing pipeline consisting of three levels: data source, data transformation, and data writing. Step S2 of establishing a multi-granularity breakpoint persistence mechanism specifically involves using an embedded key-value storage engine as the breakpoint database, automatically recording data synchronization points according to three levels of granularity: task, job, and subtask. The information of the data synchronization points is persistently stored on the local disk in the form of key-value pairs, where the key is the subtask identifier and the value is the synchronization point at which the subtask was last successfully submitted. Step S3 of implementing write-ahead persistent queue buffer specifically involves setting up a persistent queue component before the data writing end. The persistent queue component uses a write-ahead log mechanism to write the received data to the disk file in CRC check frame format, thereby achieving safe buffering of data from memory to disk. Step S4 of the two-stage site submission is as follows: after the data writing end successfully writes the data to the target database, it sends the writing result back to the upstream through the confirmation mechanism. After receiving the successful confirmation, it updates the site information of the corresponding subtask in the breakpoint database to ensure the atomicity of data landing and site update. Step S5, which automatically resumes when the process restarts, specifically involves determining whether a breakpoint exists when the system process restarts. If it does, the system reads the last successful position of each subtask from the breakpoint database and continues reading unprocessed data from the last consumed position of the persistent queue. Otherwise, the system notifies the data source to resend data from the recorded position, thus achieving automatic breakpoint resumption. Step S6, which involves reconnecting when a link is interrupted, specifically involves automatically re-establishing the connection using an exponential backoff retry strategy when a network link interruption is detected, and responding with a cancellation token during the retry period to support graceful shutdown.

2. The method according to claim 1, characterized in that, In step S2, the breakpoint database supports open operations with retries. When the underlying storage engine fails to open due to concurrent access or file locking, it will retry up to 5 times, with an interval of 1 second between each attempt.

3. The method according to claim 1, characterized in that, In step S3, the persistent queue frame format includes three parts: payload length, CRC32 checksum, and payload data. The checksum is verified during reading to detect file corruption and to support periodic cleanup of consumed log files.

4. The method according to claim 1, characterized in that, In step S4, the structured confirmation uses ArrowIPC stream format encoding and includes three fields: return code, message, and context, which can accurately reflect the writing result and error details of each piece of data.

5. The method according to claim 1, characterized in that, In step S6, the formula for calculating the backoff time of the exponential backoff retry strategy is: retry_interval = min(initial_backoff × 2^(retries-1),max_backoff), and during each backoff waiting period, the cancellation token is listened for simultaneously. If a cancellation signal is received, the retry is stopped immediately and the process returns. Where retry_interval is the backoff time, initial_backoff is the initial backoff time, retries is the number of retries, max_backoff is the maximum backoff time, and min() is the minimum value function.

6. A breakpoint resume and fault recovery system for a time-series data conversion channel system, characterized in that, include: The breakpoint management module is implemented using an embedded key-value storage engine. It provides a three-level granularity interface for reading and writing breakpoints by task / job / subtask, and supports single setting, batch setting, query, deletion and full export operations. It supports both synchronous and asynchronous access modes. The persistent queue module is implemented using a file-based write-ahead log queue. It consists of independent readers and writers. The writer writes data frames to disk according to a configurable block size and synchronization interval and periodically executes the file synchronization write command fsync. The reader reads data frames from disk in batches and supports timeout waiting and cancellation operations. The IPC communication module is based on the Apache Arrow IPC streaming protocol to realize inter-process data transmission. It includes an IPC stream reader and an acknowledgment writer, supports efficient zero-copy data exchange in Arrow RecordBatch format, and a remote data transmission channel based on gRPC / Arrow Flight. The retry connection module includes a channel factory abstract interface and retry configuration, implementing an exponential backoff reconnection mechanism linked to the cancellation token, supporting both limited-number retries and unlimited retries modes; The health monitoring module implements a finite state machine with ten states: initialization, ready, idle, active, waiting, busy, occasional error, source error, write error, and fatal error. It automatically determines the health status of the task by using the error rate and queue depth indicators within a sliding window. The task orchestration module enables routing and orchestration of various data sources to the target based on the data source identification protocol. It injects breakpoint information into the data source connection configuration through DSN parameters and supports failover strategies to split a single task into multiple subtasks for parallel execution.

7. The system according to claim 6, characterized in that, Upon receiving a cancellation signal, the writer of the persistent queue module first consumes the remaining messages in the channel before performing the final disk synchronization, ensuring that no data is lost when the process is closed.

8. The system according to claim 6, characterized in that, The IPC data frame header in the IPC communication module includes an 8-byte acknowledgment status field, an 8-byte query identifier, and an 8-byte batch counter. The first byte of the acknowledgment status field is encoded as follows: 0x00 indicates a request, 0x01 indicates that the data has been received, 0x02 indicates that the data has been processed, 0x03 indicates that the data has been discarded, 0xFE indicates that the stream has ended, and 0xFF indicates that the data has been decoded.

9. The system according to claim 6, characterized in that... It also includes an indicator collection module, which provides real-time statistical indicators for the number of messages read, messages written, acknowledgments received, and batches sent in the persistent queue.

10. The system according to claim 6, characterized in that, The task orchestration module supports heterogeneous data sources including OPC UA / DA, MQTT, Kafka, Pulsar, InfluxDB, OpenTSDB, MySQL, PostgreSQL, Oracle, MSSQL, MongoDB, CSV, Parquet, ORC, and SparkplugB protocols. Each data source plugin independently manages its own breakpoint and recovery logic and accesses the breakpoint database through a unified task option interface.

Images