FTP unstructured file transmission breakpoint continuation implementation method and device based on DataX

By integrating a progress tracking module and an independent process monitor into DataX Writer, combined with an FTP resume scheduler, the lack of resume capability in the DataX framework for FTP unstructured file transfers is resolved. This enables precise recording of transfer progress and manually controllable resume, improving the robustness and efficiency of data synchronization.

CN121262202BActive Publication Date: 2026-06-26CHINA ELECTRONICS CLOUD DIGITAL INTELLIGENCE TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511383775.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-09-26
Publication Date
2026-06-26
Estimated Expiration
2045-09-26

AI Technical Summary

Technical Problem

The DataX framework lacks fine-grained breakpoint resumption capabilities in FTP unstructured file transfer scenarios, resulting in low resource utilization, an uncontrollable automatic recovery mechanism that increases the risk of data inconsistency, opaque transfer status, and an inability to accurately record transfer progress, which affects the robustness and efficiency of data synchronization.

Method used

By adopting a progress recording-active saving-manually triggered recovery mechanism, and by integrating a progress tracking module into DataX Writer, combined with an independent process monitor and FTP resume scheduler, the system enables real-time recording of transmission progress and manually controllable resume operations, ensuring the integrity and controllability of data transmission.

Benefits of technology

It enables accurate recording of the transfer progress of unstructured files, reduces resource waste, improves the robustness and efficiency of data synchronization, reduces operational complexity, and enhances user controllability and manageability of the transfer process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121262202B_ABST
    Figure CN121262202B_ABST
Patent Text Reader

Abstract

The application relates to an FTP unstructured file transmission breakpoint continuation implementation method and device based on DataX. The method comprises the following steps: integrating a progress tracking module into a DataX Writer, creating a transmission progress metadata file (TPMF) containing information such as the number of uploaded bytes and the state for each file and dynamically updating; deploying an independent process monitor to monitor task interruption and trigger progress persistence to the TPFM; a user selects a file through an independent FTP continuation scheduler, reads the TPFM and the server-side target file size to determine the breakpoint, calls an FTP REST command to continue from the breakpoint, updates the state after completion, and the like. The method realizes accurate progress recording, active saving in abnormal interruption and user-controllable continuation, avoids redundant retransmission, improves transmission efficiency and resource utilization, enhances data synchronization robustness and reduces operation and maintenance complexity. The method has small invasiveness to the DataX core, is strong in universality and is suitable for high-reliability synchronization scenes of large-scale unstructured files.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of database synchronization and data migration technology, and in particular to a method, apparatus, computer-readable storage medium, and electronic device for resuming interrupted FTP unstructured file transfers based on DataX. Background Technology

[0002] In the era of big data, data synchronization tools, as core components of cross-system data interaction, directly impact the continuity of data-driven business through their performance and reliability. DataX, an open-source framework widely used in enterprise-level data synchronization, supports efficient data migration between various heterogeneous data sources. However, when handling large-scale FTP unstructured file transfer scenarios, it still faces a key technical bottleneck—the lack of built-in fine-grained breakpoint resumption capabilities. This issue has become a major obstacle restricting its application in scenarios with high stability requirements.

[0003] Currently, for the transfer of unstructured files (such as logs, images, audio / video files, and binary files) via FTP, existing solutions generally rely on two modes: first, a primitive mechanism based on simple retransmission, where the transfer completely restarts after a task is interrupted; second, some extended solutions attempt to introduce automatic resume functionality. However, these solutions have revealed significant limitations in practical applications:

[0004] 1. DataX native framework lacks checkpointing capabilities.

[0005] DataX's task execution architecture prioritizes atomicity, with each task instance running independently and lacking cross-task state persistence and tracking mechanisms. Its metadata management focuses solely on task configuration and overall execution results, neglecting to design a checkpoint recording module for file-level transfer processes (such as single-file transfer progress and successfully transferred byte offsets). Therefore, if a task is interrupted, the framework cannot identify completed transfers and can only achieve synchronization through a full retransmission, resulting in low resource utilization.

[0006] 2. The uncontrollability and risks of automatic recovery mechanisms

[0007] While some improved solutions achieve automatic resume, their essence is a system-level fault response strategy—immediately triggering a restart and attempting recovery based on residual files after a task fails. This model has significant drawbacks in a production environment: on the one hand, unstable factors such as network fluctuations and temporary storage service failures may lead to repeated triggering of invalid resumes, exacerbating the consumption of network bandwidth and computing resources; on the other hand, users cannot intervene in the recovery process, making it difficult to selectively recover based on business needs (such as skipping confirmed corrupted files and prioritizing the recovery of critical files), and it is also impossible to manually verify the rationality of the recovery operation, increasing the risk of data inconsistency.

[0008] 3. Insufficient transparency of transmission status and lack of user awareness.

[0009] Automatic recovery processes typically operate as a "black box," meaning users cannot obtain crucial information in real time, such as the transfer progress (byte offset) of specific files when the task was interrupted, the list of files successfully resumed, and the success rate of the current recovery phase. This lack of transparency makes it difficult for operations and maintenance personnel to pinpoint the root cause of the failure and assess the completeness and effectiveness of the resume operation, thus reducing the manageability and reliability of data synchronization.

[0010] 4. Technical challenges posed by the characteristics of unstructured documents

[0011] Unlike unstructured files, structured data (such as database tables) can be precisely located at breakpoints using identifiers such as row numbers and transaction IDs, enabling incremental resumption of transmissions. Unstructured files, however, rely on the continuity of byte streams, and their content lacks logical fragmentation markers. Existing solutions that only record file-level completion status (e.g., "File A successfully transferred") cannot handle large file transfer interruption scenarios (e.g., a 10GB file being interrupted when transferred down to 5GB). Attempting to record byte offsets requires addressing issues such as offset synchronization and file lock contention during multi-threaded concurrent transmissions, resulting in high technical complexity and the potential introduction of new consistency risks.

[0012] 5. Systemic impacts of resource waste and inefficiency

[0013] Frequent full retransmissions or invalid automatic resumes directly lead to a double waste of time and network resources. For synchronization tasks of TB-level unstructured file clusters, retransmission after a single interruption may take hours or even days, seriously affecting the timeliness of business data; at the same time, redundant network traffic may squeeze the bandwidth resources of other critical services, reduce the overall operating efficiency of the data platform, and ultimately affect user satisfaction with data synchronization services.

[0014] In summary, the existing DataX framework urgently needs a method to support file-level granular state tracking, provide user-controllable resume strategies, enhance the transparency of the transmission process, and adapt to the characteristics of unstructured file byte streams in FTP unstructured file transfer scenarios, so as to improve the robustness, efficiency and manageability of data synchronization. Summary of the Invention

[0015] To address the aforementioned technical bottlenecks in existing technologies, this invention proposes a novel method and apparatus for resuming interrupted FTP unstructured file transfers based on DataX. This invention aims to solve the technical problems of DataX's lack of a checkpoint function and the fact that existing interrupted transfer mechanisms are mostly automatic recovery, leading to difficulties in accurately recording transfer progress, effectively handling abnormal interruptions, and lacking manually controllable resumption operations during FTP unstructured file transfers, resulting in low data transmission efficiency and compromised reliability.

[0016] To address the aforementioned technical issues, this invention develops a DataX-based method for resuming interrupted FTP unstructured file transfers and for manually triggered recovery. The core of this method lies in constructing a linkage mechanism of "progress recording - active saving - manual trigger recovery" to ensure the integrity and controllability of data transmission.

[0017] To achieve the above objectives, the present invention devises the following technical strategies:

[0018] 1. Employ a complete lifecycle management system: "Progress recording - proactive saving - manual triggering recovery".

[0019] (1) Accurate progress recording: By integrating a progress tracking module, DataX Writer can record the transmission progress (byte offset) in real time and accurately, which solves the fundamental problem that DataX does not natively support progress recording.

[0020] (2) Active error saving: An independent process monitor and progress tracking module are introduced to ensure that the latest transmission progress is actively saved in a timely manner when the DataX task is interrupted in any way (normal termination, manual termination, abnormal exit). This mechanism is more efficient and reliable than passively waiting for the system to recover automatically.

[0021] (3) Flexible manual resume: As one of the core innovations of this invention, the traditional automatic recovery mode is abandoned, and an independent "FTP resume scheduler" is designed, which allows users to select the files to be recovered on demand and in batches, and manually trigger the resume operation. This design greatly enhances the user's controllability over the data synchronization process and avoids invalid automatic retries caused by network fluctuations and other reasons.

[0022] 2. Separate progress tracking and saving mechanism: The triggering and execution of progress tracking (real-time recording) and progress saving (persistence) are reasonably separated. DataX Writer is responsible for efficiently recording and transmitting progress, while the process monitor triggers saving at critical moments (when interrupted). This design optimizes performance and increases reliability.

[0023] 3. "Status-Aware" Breakpoint Determination: Before resuming a download, the FTP resume scheduler not only reads the locally recorded UploadedSize, but also optionally queries the server-side file size using the FTP SIZE command. This "status-aware" comparison can more accurately determine whether a resume is needed and from which point to resume, avoiding problems caused by inaccurate TPMF records or accidental modifications to server-side files.

[0024] 4. Minimize DataX core invasiveness: This solution is mainly achieved by enhancing the DataX Writer plugin or building external proxy services, independent process monitors and schedulers. It involves very little modification to the core DataX code, is easy to deploy and maintain, and can better support different versions of DataX.

[0025] 5. Enhanced robustness of unstructured file transfer: For unstructured files, this invention relies on the underlying byte offset (REST command) without parsing the file content, thus possessing high versatility and robustness, ensuring the continuity of transmission of large files, binary files, etc.

[0026] The overall design architecture of this invention is as follows: Figure 1 As shown, its core functions and processes are described below:

[0027] 1. Transfer progress recording and dynamic updates (DataX Writer integration)

[0028] (1) Transmission Progress Metadata File (TPMF):

[0029] Create a separate metadata file for each file to be transmitted (e.g., using JSON format in this embodiment of the invention).

[0030] TPMF should include at least: SourceFilePath (source file path), TargetFTPInfo (including server address, port, username, password, target directory, and target file name), TotalFileSize (total file size), UploadedSize (number of bytes successfully uploaded, initially 0), Status (e.g., PENDING, TRANSFERRING, INTERRUPTED, COMPLETED, FAILED), and LastUpdateTime (last update time).

[0031] TPMF can be stored on the local file system, shared network directory, or database on the DataX running node.

[0032] (2) Progress tracking module:

[0033] This module communicates with DataX Writer through a separate background process.

[0034] When DataX Writer starts transferring a file, it looks up or creates the corresponding TPMF and reads its UploadedSize.

[0035] After each successful write to a file chunk, DataX Writer adds the number of bytes written to the current UploadedSize.

[0036] Dynamic updates: DataX Writer will update the latest UploadedSize to the progress tracking module at controlled time intervals (e.g., after transferring 1MB of data, or after completing an internal Read / Write buffer process). The progress tracking module is responsible for temporarily caching this update for later persistence.

[0037] 2. Active save mechanism in case of anomalies

[0038] (1) Process monitor: Deploy an external process monitor (e.g., a standalone daemon, configured using the supervisor tool in this embodiment of the invention). This monitor is responsible for monitoring the running status of the DataX main process in real time.

[0039] (2) Save the trigger signal:

[0040] Manual termination: When a user manually terminates a DataX task via command line (such as the kill command) or DataX's web UI, the process monitor will capture the task's termination signal.

[0041] Abnormal Interruption: When the DataX process exits abnormally due to crashes, network interruptions, or other reasons, the process monitor will also detect the termination of the process.

[0042] Progress saving: Upon receiving a "terminate" signal, the process monitor sends a "save progress" instruction to the progress tracking module (e.g., via an HTTP request).

[0043] The progress tracking module's response: Upon receiving an instruction, the progress tracking module will immediately persist its cached latest UploadedSize and current file status (e.g., updated to "INTERRUPTED") to the corresponding TPMF file.

[0044] Saving upon normal termination: Before the DataX Writer's lifecycle ends (for example, when it has finished processing all files and is ready to close), it will proactively notify the progress tracking module to persist the final UploadedSize and status (set to "COMPLETED" if the transfer was successful; otherwise, handle as appropriate).

[0045] 3. Manually triggered resume operation

[0046] (1) FTP resume scheduler: Set up a separate application that provides a visual interactive interface. This tool is responsible for receiving users' resume requests.

[0047] (2) User interaction: Users select one or more tasks that need to resume transmission through the FTP resume scheduler and trigger the “resume” operation.

[0048] (3) Resumption logic:

[0049] Query TPMF: The FTP resume scheduler reads the TPMF of the selected file to obtain UploadedSize and TotalFileSize.

[0050] Optional server-side verification: To improve robustness, the scheduler can use the FTP SIZE command to query the actual size of the target file on the FTP server.

[0051] (4) Breakpoint detection:

[0052] If the UploadedSize recorded in TPMF is equal to TotalFileSize, or the target file size on the server is equal to TotalFileSize, then the file is considered complete and no further download will be performed.

[0053] If UploadedSize is less than TotalFileSize, or if the target file size on the server is less than TotalFileSize (and UploadedSize is also less than the server file size), then a resume download will be triggered.

[0054] (5) Perform resume download:

[0055] After the FTP resume scheduler triggers the resume, it uses the FTP client library in the DataX plugin to establish a connection with the target FTP server.

[0056] Key step: Calling FTP's REST function <uploadedsize>The command instructs the server to begin transferring from the specified byte offset. Then, use the STOR command to continue uploading the remaining portion of the file.

[0057] (6) Update complete: After the file is successfully transferred, the FTP resume scheduler updates the TPMF status of the file to "COMPLETED" and records the completion time.

[0058] (7) Error handling: If an error occurs during the resume process, the FTP resume scheduler will record the error information to TPMF and set the status to "FAILED" so that it can be manually tried next time.

[0059] Specifically, this application provides the following technical solutions:

[0060] The first aspect of this application provides a method for resuming interrupted downloads in FTP unstructured file transfers based on DataX, the method comprising:

[0061] S1. Progress Recording: A progress tracking module is integrated into DataX Writer to create a TPMF (Transmission Progress Metadata File) for each unstructured file to be transmitted. DataX Writer dynamically updates the UploadedSize (number of bytes uploaded) contained in the TPMF during the transmission process and feeds back the updated progress information to the progress tracking module.

[0062] S2. Proactive Error Saving: Deploy an independent process monitor to monitor the running status of DataX tasks; when a DataX task is interrupted, the process monitor sends a save command to the progress tracking module; the progress tracking module persists the latest UploadedSize and interrupt status to the corresponding TPMF;

[0063] S3. Manually triggered resume: The user selects the file to be restored and triggers the resume operation through an independent FTP resume scheduler; the FTP resume scheduler reads the TPMF of the selected file, obtains the UploadedSize and the total file size, and queries the actual size of the target file on the server side through the FTP SIZE command;

[0064] S4. Breakpoint Detection and Resumption of Download: If the UploadedSize recorded in TPMF is less than the total file size, or if the target file size on the server side is less than the total file size and UploadedSize is less than the target file size on the server side, the FTP resume scheduler calls the REST command of the FTP client to instruct the server to start transmitting from the UploadedSize byte offset and continue uploading the remaining file content; after the resume is completed, the TPMF status is updated to "Completed" and the completion time is recorded.

[0065] Furthermore, in the method of this application, the progress tracking module in step S1 communicates with DataX Writer through a background process or HTTP interface, receives and caches the progress updates fed back by DataX Writer, and performs persistent operations asynchronously.

[0066] Furthermore, in the method of this application, the transmission progress metadata file mentioned in step S1 includes the source file path, target FTP information, total file size, UploadedSize, transmission status, and last update time.

[0067] Furthermore, in the method of this application, the interruption of the DataX task in step S2 includes interruption of the DataX task due to normal termination, manual termination, or abnormal exit.

[0068] Furthermore, in the method of this application, the process monitor in step S2 is a daemon process (such as supervisor), which triggers a save command by listening for the exit event of the DataX process; the TPMF is stored in a local file system, a shared network directory, a database, or a distributed file system.

[0069] Furthermore, in the method of this application, the FTP resume scheduler described in step S3 provides a visual interactive interface that displays a task list and file status, and supports users to select files to be resumed in batches.

[0070] Furthermore, in the method of this application, before the resumption of transmission is executed in step S4, if the UploadedSize recorded in TPMF is equal to the total file size, or the target file size on the server side is equal to the total file size, then it is determined that the file has been completely transmitted and the resumption of transmission is not executed.

[0071] Furthermore, in the method of this application, after the resume download is completed in step S4, if the FTP resume download scheduler detects that the file transfer has failed, it updates the TPMF status to "failed" and records the error information for the user to trigger the resume download again later.

[0072] A second aspect of this application provides a device for resuming interrupted FTP unstructured file transfers based on DataX, the device comprising:

[0073] The progress recording module is used to integrate the progress tracking module into DataX Writer. It creates a TPMF (Transmission Progress Metadata File) for each unstructured file to be transmitted. During the transmission process, DataX Writer dynamically updates the UploadedSize (number of bytes uploaded) contained in the TPMF and feeds back the updated progress information to the progress tracking module.

[0074] The proactive exception saving module is used to deploy an independent process monitor to monitor the running status of DataX tasks. When a DataX task is interrupted, the process monitor sends a save command to the progress tracking module. The progress tracking module persists the latest UploadedSize and interrupt status to the corresponding TPMF.

[0075] The manual-triggered resume module allows users to select the file to be restored and trigger the resume operation through an independent FTP resume scheduler. The FTP resume scheduler reads the TPMF of the selected file, obtains the UploadedSize and the total file size, and queries the actual size of the target file on the server side through the FTP SIZE command.

[0076] The breakpoint detection and resume execution module is used to perform breakpoint detection and resume transmission. If the UploadedSize recorded in TPMF is less than the total file size, or if the target file size on the server side is less than the total file size and UploadedSize is less than the target file size on the server side, the FTP resume scheduler calls the REST command of the FTP client to instruct the server to start transmitting from the UploadedSize byte offset and continue uploading the remaining file content. After the resume is completed, the TPMF status is updated to "completed" and the completion time is recorded.

[0077] The device implements the steps of the aforementioned DataX-based FTP unstructured file transfer breakpoint resumption method during operation.

[0078] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0079] Memory: Used to store computer programs;

[0080] Processor: Used to execute the computer program to implement the steps of the aforementioned DataX-based FTP unstructured file transfer breakpoint resume method.

[0081] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned DataX-based FTP unstructured file transfer breakpoint resumption method.

[0082] In summary, this invention, through targeted design, achieves breakpoint resumption and controllable recovery of FTP unstructured file transfers in the DataX environment. Compared with existing technologies, this invention has the following advantages:

[0083] (1) Precise progress tracking capability: By integrating a progress tracking module into DataX Writer, fine-grained recording of the transmission progress of each file (number of bytes transmitted) is achieved, which solves the problem that the traditional solution cannot accurately locate the breakpoint after interruption, and provides a precise basis for resuming transmission.

[0084] (2) Reliable data preservation during abnormal interruption: An independent process monitor and active saving mechanism are introduced to ensure that the current transmission progress can be persisted in a timely and accurate manner when the DataX task is abnormal or manually terminated, so as to minimize the risk of data loss.

[0085] (3) User-led controllable resume: Provides an independent "FTP resume scheduler" so that users can choose the files to be recovered. The scheduler performs the resume based on the recorded breakpoint information, avoiding the unexpected behavior of the system automatically retrying and giving users complete control over the recovery process.

[0086] (4) Transmission efficiency and resource optimization: Precise breakpoint location and resume mechanism avoid redundant retransmission, especially in large file scenarios, significantly saving time and network bandwidth resources and improving overall synchronization efficiency.

[0087] (5) Enhanced robustness and reliability: Supports interruption recovery in unstable scenarios such as network fluctuations and temporary server unavailability, and ensures the final completion of tasks by manually triggering resume transmission, thereby enhancing the stable execution capability of data synchronization.

[0088] (6) Reduced operation and maintenance complexity: Avoid potential risks such as repeated network attempts that may be caused by automatic recovery, place the recovery operation under the active control of the user, simplify system logic, and reduce the operation and maintenance burden.

[0089] (7) Low intrusion and high versatility: Through external component expansion and reasonable interface design, the core functions of DataX are minimally intrusive, and have good integration adaptability and maintenance convenience.

[0090] In summary, this invention effectively solves the key problem of DataX's lack of resume capability after interruption in FTP unstructured file transfers, resulting in significant technical gains in terms of reliability, controllability, efficiency, and maintainability.

[0091] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0092] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0093] Figure 1 This is the overall design architecture diagram of the solution proposed in this application.

[0094] Figure 2 This is a flowchart illustrating the implementation of the DataX-based FTP unstructured file transfer breakpoint resume method in this application.

[0095] Figure 3 This is a structural diagram of the FTP unstructured file transfer breakpoint resume implementation device based on DataX in this application.

[0096] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0098] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0099] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0100] Figure 2 The diagram shows the overall implementation flow of the DataX-based FTP unstructured file transfer breakpoint resumption method provided in this application, including the following steps:

[0101] S1. Progress Recording: A progress tracking module is integrated into DataX Writer to create a TPMF (Transmission Progress Metadata File) for each unstructured file to be transmitted. DataX Writer dynamically updates the UploadedSize (number of bytes uploaded) contained in the TPMF during the transmission process and feeds back the updated progress information to the progress tracking module.

[0102] S2. Proactive Error Saving: Deploy an independent process monitor to monitor the running status of DataX tasks; when a DataX task is interrupted, the process monitor sends a save command to the progress tracking module; the progress tracking module persists the latest UploadedSize and interrupt status to the corresponding TPMF;

[0103] S3. Manually triggered resume: The user selects the file to be restored and triggers the resume operation through an independent FTP resume scheduler; the FTP resume scheduler reads the TPMF of the selected file, obtains the UploadedSize and the total file size, and queries the actual size of the target file on the server side through the FTP SIZE command;

[0104] S4. Breakpoint Detection and Resumption of Download: If the UploadedSize recorded in TPMF is less than the total file size, or if the target file size on the server side is less than the total file size and UploadedSize is less than the target file size on the server side, the FTP resume scheduler calls the REST command of the FTP client to instruct the server to start transmitting from the UploadedSize byte offset and continue uploading the remaining file content; after the resume is completed, the TPMF status is updated to "Completed" and the completion time is recorded.

[0105] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0106] 1. Environment preparation and component setup

[0107] (1) DataX environment: Ensure that DataX has been deployed correctly and prepare the DataX JSON configuration file to define the data source (local file), data target (FTP server) and transfer task.

[0108] (2) Progress tracking module: Build a lightweight Java service that provides a RESTful API. DataXWriter interacts with it through HTTP requests.

[0109] Core functionality: Receive real-time updates on DataX Writer upload progress, cache the latest progress, and respond to "save progress" commands.

[0110] (3) Transmission Progress Metadata File (TPMF) Storage:

[0111] Storage method: It can be a local file system (for example, a .meta file for each file in a fixed directory of the DataX execution node), or a separate database (such as MySQL), or a distributed file system (such as HDFS, S3).

[0112] File structure: JSON format.

[0113] (4) Process monitor:

[0114] supervisor: Configures monitoring of the DataX process. When the exit of the DataX process is detected, it triggers a call to the progress tracking module through the postexit script.

[0115] 2. Detailed Implementation Process

[0116] 2.1. Integration of DataX Writer with the Progress Tracking Module

[0117] Initialization: When DataX Writer starts, it loads the configuration and determines the list of files to be transferred. For each file, it queries the TPMF storage to find the corresponding metadata. If it exists, it reads UploadedSize and TotalFileSize; if it does not exist, it creates a new TPMF with UploadedSize set to 0 and Status set to PENDING.

[0118] File transfer: DataX Writer starts reading data from the local Reader and calls the FTP Writer interface to send the data to the FTP server.

[0119] Progress update: Each time DataX Writer successfully writes a batch of data (for example, after writing to a Buffer), it sends the currently accumulated UploadedSize to the progress tracking module.

[0120] DataX Writer (external service): writer.write(record) ->currentUploadedSize +=recordSize; ->httpClient.post(" / progress / update", { fileId, uploadedSize:currentUploadedSize});

[0121] Caching and Persistence: After receiving an update, the progress tracking module caches the UploadedSize. To avoid frequent I / O, persistence operations can be performed asynchronously.

[0122] 2.2. Abnormal Termination and Progress Saving

[0123] Process monitor configuration

[0124] Using supervisor:

[0125] [program:datax-job-1]

[0126] command=python3 / path / to / datax / bin / datax.py / path / to / job.json

[0127] autostart=true

[0128] autorestart=false

[0129] redirect_stderr=true

[0130] stdout_logfile= / var / log / datax-job-1.log

[0131] stderr_logfile= / var / log / datax-job-1.err

[0132] [eventprocess]

[0133] command= / path / to / save_progress_script.sh datax-job-1

[0134] events=STOPPED

[0135] [program]: This is a configuration section header for a supervisor, indicating that the following definition is the configuration for a program.

[0136] [eventprocess]: This is another configuration section header for supervisor, indicating that the following defines the configuration for an event process. Supervisor can respond to specific events (such as start, stop, and exit) that occur in the programs it manages.

[0137] The save_progress_script.sh script will execute a command (such as curl) to trigger the save interface of the progress tracking service.

[0138] The save interface for the progress tracking service:

[0139] POST / progress / save_and_interrupt: Upon receiving a request, persist the UploadedSize and Status (set to INTERRUPTED) of all files currently in the "TRANSFERRING" state to TPMF.

[0140] 2.3. Implementation of FTP resume logic

[0141] FTP resume scheduler:

[0142] Operation and maintenance interface: Displays a list of all DataX tasks, including task ID, task status, number of associated files, number of completed files, etc.; provides functions such as "select task", "resume selected task", and "refresh".

[0143] User interaction: The user selects one or more tasks with the status "INTERRUPTED" and clicks "Continue".

[0144] File search: Retrieves all file information associated with the task from the DataX task configuration.

[0145] File resume decision:

[0146] (1) If the UploadedSize recorded in TPMF is equal to TotalFileSize, or the target file size on the server is equal to TotalFileSize, then the file is considered complete and no further upload is performed.

[0147] (2) If UploadedSize is less than TotalFileSize, or the target file size on the server is less than TotalFileSize (and UploadedSize is also less than the server file size), then resume download is triggered.

[0148] Perform a resume download:

[0149] After the FTP resume scheduler triggers the resume, it uses the FTP client library in the DataX plugin to establish a connection with the target FTP server.

[0150] Key step: Calling FTP's REST function <uploadedsize>The command instructs the server to begin transferring from the specified byte offset. Then, use the STOR command to continue uploading the remaining portion of the file.

[0151] Completion and Status Updates:

[0152] File completion: When a file resumes and completes, the scheduler updates its TPMF file status to "COMPLETED" and UploadedSize to TotalFileSize.

[0153] Task completion determination: After resuming a file transfer, the scheduler will re-examine all TPMFs for that task. If the status of all TPMFs has changed to "COMPLETED", the scheduler will call the task status manager to update the task status to "COMPLETED".

[0154] Figure 3 The image shows a DataX-based FTP unstructured file transfer resuming device proposed in this application. The device includes:

[0155] The progress recording module is used to integrate the progress tracking module into DataX Writer. It creates a TPMF (Transmission Progress Metadata File) for each unstructured file to be transmitted. During the transmission process, DataX Writer dynamically updates the UploadedSize (number of bytes uploaded) contained in the TPMF and feeds back the updated progress information to the progress tracking module.

[0156] The proactive exception saving module is used to deploy an independent process monitor to monitor the running status of DataX tasks. When a DataX task is interrupted, the process monitor sends a save command to the progress tracking module. The progress tracking module persists the latest UploadedSize and interrupt status to the corresponding TPMF.

[0157] The manual-triggered resume module allows users to select the file to be restored and trigger the resume operation through an independent FTP resume scheduler. The FTP resume scheduler reads the TPMF of the selected file, obtains the UploadedSize and the total file size, and queries the actual size of the target file on the server side through the FTP SIZE command.

[0158] The breakpoint detection and resume execution module is used to perform breakpoint detection and resume transmission. If the UploadedSize recorded in TPMF is less than the total file size, or if the target file size on the server side is less than the total file size and UploadedSize is less than the target file size on the server side, the FTP resume scheduler calls the REST command of the FTP client to instruct the server to start transmitting from the UploadedSize byte offset and continue uploading the remaining file content. After the resume is completed, the TPMF status is updated to "completed" and the completion time is recorded.

[0159] When the above-mentioned device is in operation, it implements the steps of the DataX-based FTP unstructured file transfer breakpoint resume method disclosed in this application.

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0161] like Figure 4 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the above-described DataX-based FTP unstructured file transfer breakpoint resumption method.

[0162] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0163] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0164] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0165] Conventional procedural programming languages, such as "C" or similar programming languages.

[0166] The execution methods of program code include, but are not limited to:

[0167] It runs entirely on the user's computer;

[0168] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0169] Execute as a standalone software package;

[0170] It is executed entirely on a remote computer or server.

[0171] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0172] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the DataX-based FTP unstructured file transfer breakpoint resume implementation method disclosed in this application.

[0173] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0174] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0175] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0176] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0177] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0178] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0179] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0180] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0181] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0182] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0183] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0184] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0185] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the DataX-based FTP unstructured file transfer breakpoint resumption method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0186] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0187] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.< / uploadedsize> < / uploadedsize>

Claims

1. A method for resuming interrupted downloads in FTP unstructured file transfers based on DataX, characterized in that, The method includes: S1. Progress Recording: A progress tracking module is integrated into DataX Writer to create a TPMF for each unstructured file to be transmitted. DataX Writer dynamically updates the UploadedSize contained in the TPMF during the transmission process and feeds back the updated progress information to the progress tracking module. S2. Proactive Error Saving: Deploy an independent process monitor to monitor the running status of DataX tasks; when a DataX task is interrupted, the process monitor sends a save command to the progress tracking module; the progress tracking module persists the latest UploadedSize and interrupt status to the corresponding TPMF; S3. Manually triggered resume: The user selects the file to be restored and triggers the resume operation through an independent FTP resume scheduler; the FTP resume scheduler reads the TPMF of the selected file, obtains the UploadedSize and the total file size, and queries the actual size of the target file on the server side through the FTPSIZE command; S4. Breakpoint Detection and Resumption of Download: If the UploadedSize recorded in TPMF is less than the total file size, or if the target file size on the server side is less than the total file size and UploadedSize is less than the target file size on the server side, the FTP resume scheduler calls the REST command of the FTP client to instruct the server to start transmitting from the UploadedSize byte offset and continue uploading the remaining file content; after the resume is completed, the TPMF status is updated to "Completed" and the completion time is recorded.

2. The method according to claim 1, characterized in that, In step S1, the progress tracking module communicates with DataXWriter through a background process or HTTP interface, receives and caches the progress updates fed back by DataXWriter, and performs persistent operations asynchronously.

3. The method according to claim 1, characterized in that, The TPMF mentioned in step S1 includes the source file path, target FTP information, total file size, UploadedSize, transfer status, and last update time.

4. The method according to claim 1, characterized in that, The DataX task interruption mentioned in step S2 includes interruptions caused by normal termination, manual termination, or abnormal exit of the DataX task.

5. The method according to claim 1, characterized in that, The process monitor mentioned in step S2 is a daemon process that triggers a save command by listening for the exit event of the DataX process; the TPMF is stored in a local file system, a shared network directory, a database, or a distributed file system.

6. The method according to claim 1, characterized in that, The FTP resume scheduler described in step S3 provides a visual interactive interface that displays a task list and file status, and supports users to select files to resume in batches.

7. The method according to claim 1, characterized in that, Before resuming the transfer as described in step S4, if the UploadedSize recorded in TPMF is equal to the total file size, or the target file size on the server is equal to the total file size, then it is determined that the file has been transferred and the resuming transfer is not performed.

8. The method according to claim 1, characterized in that, After the resumed transfer is completed as described in step S4, if the FTP resume scheduler detects a file transfer failure, it updates the TPMF status to "failed" and records the error information for the user to trigger the resume transfer again later.

9. A device for resuming interrupted downloads of unstructured FTP files based on DataX, characterized in that, The device includes: The progress recording module is used to integrate the progress tracking module into DataX Writer. It creates a TPMF for each unstructured file to be transmitted. DataX Writer dynamically updates the UploadedSize contained in the TPMF during the transmission process and feeds back the updated progress information to the progress tracking module. The proactive exception saving module is used to deploy an independent process monitor to monitor the running status of DataX tasks. When a DataX task is interrupted, the process monitor sends a save command to the progress tracking module. The progress tracking module persists the latest UploadedSize and interrupt status to the corresponding TPMF. The manual-triggered resume module allows users to select the file to be restored and trigger the resume operation through an independent FTP resume scheduler. The FTP resume scheduler reads the TPMF of the selected file, obtains the UploadedSize and the total file size, and queries the actual size of the target file on the server side through the FTP SIZE command. The breakpoint detection and resume execution module is used to perform breakpoint detection and resume transmission. If the UploadedSize recorded in TPMF is less than the total file size, or if the target file size on the server side is less than the total file size and UploadedSize is less than the target file size on the server side, the FTP resume scheduler calls the REST command of the FTP client to instruct the server to start transmitting from the UploadedSize byte offset and continue uploading the remaining file content. After the resume is completed, the TPMF status is updated to "completed" and the completion time is recorded.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the DataX-based FTP unstructured file transfer breakpoint resume method as described in any one of claims 1-8.

Citation Information

Patent Citations

  • Full-amount and incremental synchronization method and device for Clickhouse and storage medium

    CN112988916A

  • Data synchronization system

    CN114925140A