An SFTP task automation processing method and device and a computer readable storage medium

Abstract

This invention relates to the field of automated file transfer processing technology, and discloses an automated SFTP task processing method, apparatus, and computer-readable storage medium. The method includes: acquiring directory change data and objectifying the directory change data to generate candidate file objects and determine file objects to be preprocessed; preprocessing the file objects to be preprocessed based on file feature data and preprocessing rule data corresponding to the file objects to be preprocessed, to generate preprocessing result objects; generating structured transfer tasks based on the preprocessing result objects, and determining group keys based on the target SFTP server IP and target directory, grouping structured transfer tasks with the same group key into the same group of tasks; establishing a group-level execution context based on the group tasks, and executing the structured transfer tasks within the group in the group-level execution context to obtain task execution results; and updating the task status and / or generating result records of the structured transfer tasks based on the task execution results.

Description

Technical Field

[0001] This invention relates to the field of automated file transfer processing technology, and in particular to an automated SFTP task processing method, apparatus, and computer-readable storage medium. Background Technology

[0002] In batch file exchange scenarios, business systems typically need to obtain files from the source SFTP server, process them, and then transfer them to the target SFTP server. Although existing SFTP clients can provide basic upload, download, and directory access capabilities, most of them still remain at the protocol layer interface encapsulation level, making it difficult to perform unified and coordinated processing of directory change detection, file filtering, duplicate file exclusion, file preprocessing, aggregation execution by target location, and abnormal state control.

[0003] In large-scale file task scenarios, existing solutions generally suffer from problems such as delayed or missed directory change detection, duplicate and unstable files entering the transmission link, difficulty in coordinating the processing of massive small files and ultra-large files in the same execution link, and lack of task organization methods around the target server and target directory, resulting in repeated connection establishment, frequent directory switching, and difficulty in converging abnormal states.

[0004] Therefore, there is a need for an automated SFTP task processing solution that can collaboratively process directory change results, file object reshaping, task grouping and execution, and result state convergence. Summary of the Invention

[0005] The purpose of this invention is to provide an automated SFTP task processing method, apparatus, and computer-readable storage medium to solve the problems of insufficient utilization of directory change results, lack of unified screening and shaping before files enter the transmission link, lack of grouping and execution mechanism for tasks with the same target location, and difficulty in centralized convergence of abnormal states in existing SFTP task processing.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0007] On one hand, the present invention provides an automated SFTP task processing method, comprising: Obtain directory change data, generate candidate file objects based on the directory change data, perform filtering and duplicate identification on the candidate file objects to determine the file objects to be preprocessed, and perform preprocessing on the file objects to be preprocessed to generate preprocessed result objects; A structured transfer task is generated based on the preprocessed result object, and a group key is determined based on the target SFTP server IP and the target directory. The structured transfer task is then grouped to generate a group task. A group-level execution context is established based on the group tasks, and the group-level structured transmission tasks are executed within the group-level execution context to obtain the task execution results; Update the task status of the structured transmission task and / or generate result records based on the task execution results.

[0008] Further, the steps of acquiring directory change data, generating candidate file objects based on the directory change data, and performing filtering and duplicate identification on the candidate file objects to determine the file objects to be preprocessed include: A monitoring context is established based on the directory to be monitored, and first change data is obtained according to the event listening method and / or second change data is obtained according to the timed scanning method. The first change data and the second change data are processed in a unified manner to generate the candidate file object. The candidate file objects are subjected to rule-constrained data for rule matching, and the candidate file objects that pass the rule matching are subjected to validity verification. Duplicate identification is performed on the candidate file objects that pass the validity verification based on historical processing data to determine the file objects to be preprocessed. The validity verification includes: acquiring attribute data of the file corresponding to the candidate file object at multiple times, and performing stability determination based on the attribute data acquired at multiple times.

[0009] Further, the step of performing preprocessing on the file object to be preprocessed to generate a preprocessing result object includes: Based on the file feature data and preprocessing rule data corresponding to the file object to be preprocessed, a preprocessing strategy is determined; The preprocessing strategy includes performing at least one of merging, splitting, and compression on the file object to be preprocessed, generating a mapping result that characterizes the relationship between the processed file object and the original input file object, and generating a preprocessing result object based on the processed file object and the mapping result. The merging process includes merging multiple file objects to be preprocessed when they have the same target SFTP server IP and target directory and meet the content consistency condition. The splitting process includes splitting the target file object to be preprocessed into multiple fragmented files when the target file object to be preprocessed meets the splitting conditions.

[0010] Furthermore, the structured transmission task is generated based on the preprocessed result object, the transmission target data, and the task control data; and the group key is determined according to the target SFTP server IP and the target directory, and structured transmission tasks with the same group key are grouped into the same group of tasks.

[0011] Further, a group-level execution context is established based on the group tasks, and the group-level structured transport tasks are executed within the group-level execution context, including: Obtain the target service connection based on the target server data corresponding to the group task, perform validity verification on the target service connection, and perform directory check and / or directory creation according to the target directory corresponding to the group task if the target directory is not ready. The group-level execution context is established by validating the target service connection and the ready target directory, and the group-level structured transport task is executed in the group-level execution context.

[0012] Furthermore, before establishing a group-level execution context based on the group tasks, the following is also included: Based on group-level preparation status data, thread resource data, and task status data, a scheduling check is performed on the group tasks. The target execution thread is determined according to the scheduling check results. Lock checks and retry eligibility checks are performed on the structured transmission tasks within the group. The structured transmission tasks that pass the lock checks and retry eligibility checks are added to the set of tasks to be executed corresponding to the target execution thread.

[0013] Further, updating the task status and / or generating result records of the structured transmission task based on the task execution result includes: When the task execution result indicates successful transmission, a result record corresponding to the structured transmission task is generated, and the task status of the structured transmission task is updated to a successful status. When the task execution result indicates a transmission failure, a recovery path is determined based on the failure type and the current number of retries, and the task status of the structured transmission task is updated to a retryable state or a final failure state. When the failure type is connection anomaly and the current number of retries has not reached the preset retry threshold, the structured transmission task is updated to a retryable and scheduled state. When the failure type is a file missing exception, a preprocessing result exception, or the current number of retries reaches the preset retry threshold, the structured transmission task is updated to a final failure state.

[0014] Furthermore, after updating the task status of the structured transmission task and / or generating result records based on the task execution results, the method further includes: Obtain task context data associated with the structured transmission task, assemble a callback context object using the task context data, determine a callback processor based on the callback type, call the callback processor to execute callback processing, and update the callback status of the structured transmission task based on the callback processing result. Based on the mapping results, task execution status, and / or callback status corresponding to the structured transmission task, file cleanup processing is performed. The final status of the structured transmission task is determined according to the file cleanup processing results, and the group status of the corresponding group task is updated according to the final status of all structured transmission tasks in the group.

[0015] On the other hand, the present invention provides an automated SFTP task processing apparatus, comprising: A candidate object generation unit is used to acquire directory change data and generate candidate file objects based on the directory change data; The filtering and identification unit is used to perform filtering and duplicate identification on the candidate file objects to determine the file objects to be preprocessed. A preprocessing unit is used to perform preprocessing on the file object to be preprocessed in order to generate a preprocessing result object; The task generation unit is used to generate structured transmission tasks based on the preprocessed result object, and to determine the group key based on the target SFTP server IP and the target directory, and to group the structured transmission tasks to generate group tasks. An execution unit is configured to establish a group-level execution context based on the group tasks, and execute the group-level structured transmission tasks within the group-level execution context to obtain task execution results; The result processing unit is used to update the task status of the structured transmission task and / or generate result records based on the task execution result.

[0016] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the SFTP task automation processing method.

[0017] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: 1. Generate candidate file objects through directory change data, and perform screening and duplicate identification on the candidate file objects to exclude invalid files, duplicate files and unstable files before transmission, so that the input objects entering the subsequent processing link have consistency and validity. In the stage of preprocessing file objects, perform preprocessing such as merging, splitting and compressing, and generate the mapping result between the processed file objects and the original input file objects, thereby preserving the file form adjustment results and object association relationships, providing a unified data foundation for subsequent structured task generation, result tracking and closing processing; 2. By determining the group key through the target SFTP server IP and target directory, structured transfer tasks are grouped and a group-level execution context is established based on the group tasks. This allows multiple structured transfer tasks pointing to the same target location to be executed continuously in a unified target service connection and target directory context. This transforms the pre-shaped transfer objects into executable objects that can be organized by group. Through group task scheduling verification, lock verification, retry qualification verification, and recovery path diversion based on task execution results, a complete link is formed, including transfer execution, exception recovery, result write-back, callback processing, file cleanup, and final state convergence. Attached Figure Description

[0018] Figure 1 The diagram shown is a schematic representation of the overall process of the SFTP task automation method of the present invention. Figure 2 The diagram shows the process of generating, filtering, and identifying candidate file objects. Figure 3 The diagram shows the preprocessing and task generation and grouping process. Figure 4 The diagram shows the group-level execution and result processing flow. Detailed Implementation

[0019] This invention can be modified in many ways and has many embodiments, with specific embodiments shown in the accompanying drawings for detailed description. However, this does not mean that the invention is limited to a specific implementation; it should be understood that all modifications, equivalents, and even substitutions falling within the concept and technical scope of this invention are included in this invention. Similar reference numerals are used for similar constituent elements in the description of the drawings.

[0020] The terms “first,” “second,” “A,” “B,” etc., are used to describe a wide variety of constituent elements, but these constituent elements are not limited by these terms. These terms are used to distinguish one constituent element from others. For example, without departing from the scope of the invention, a first constituent element may be named a second constituent element, and similarly, a second constituent element may be named a first constituent element. The term “and / or” includes a combination of multiple associated descriptions, or one of multiple associated descriptions.

[0021] When it is mentioned that a certain constituent element is "connected" or "linked" to other constituent elements, it can mean not only that the element is directly connected or linked to the other constituent element, but also that there are other constituent elements in between. Conversely, when it is mentioned that a certain constituent element is "directly connected" or "directly linked" to other constituent elements, there are no other constituent elements in between.

[0022] The terminology used in this application is for illustrative purposes only and is not intended to limit the scope of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as "comprising" or "having" as used herein do not preclude the possibility of the presence or addition of features, numbers, stages, actions, constituent elements, components, or combinations thereof described in the specification.

[0023] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] Terms defined in common dictionaries should be interpreted as having the same meaning as in the context of the relevant technology, and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0025] Example 1 This embodiment provides an automated SFTP task processing device, including but not limited to one deployed in a server, gateway device, task scheduling host or other device with data processing capabilities. The device includes at least a candidate object generation unit, a filtering and identification unit, a preprocessing unit, a task generation unit, an execution unit and a result processing unit.

[0026] The candidate object generation unit is connected to the directory monitoring resource and is used to receive directory change data and organize the directory change data into candidate file objects. The candidate file objects serve as unified input objects in the subsequent processing chain and can carry file source information, change attribute information, and discovery time information.

[0027] The screening and identification unit is connected to the candidate object generation unit. It is used to read candidate file objects and perform rule screening, validity verification and duplicate identification on the candidate file objects based on rule constraint data, thereby converting file objects that meet the processing conditions and pass the stability judgment into file objects to be preprocessed.

[0028] The preprocessing unit is connected to the filtering and identification unit and is used to read the file feature data and preprocessing rule data corresponding to the file object to be preprocessed. According to the determined preprocessing strategy, it performs at least one of the following processes on the file object to be preprocessed: merging, splitting, and compression, so as to output a preprocessing result object. The preprocessing result object may contain the processed file object, the preprocessing type, and the mapping result between the processed file object and the original input file object.

[0029] The task generation unit is connected to the preprocessing unit and is used to generate structured transmission tasks based on the preprocessing result objects. It determines the group key according to the target SFTP server IP and the target directory, and groups structured transmission tasks with the same group key into the same group of tasks. Through the above grouping process, multiple tasks with the same target location can share the group-level execution context in the subsequent execution stage.

[0030] The execution unit is connected to the task generation unit and is used to establish a group-level execution context based on the group tasks, and to execute the group-level structured transmission tasks within the group in the group-level execution context. The group-level execution context is associated with at least the target service connection and the target directory status to support the continuous execution of tasks in the same group.

[0031] The result processing unit is connected to the execution unit and is used to read the task execution result and update the task status of the structured transmission task and / or generate result records based on the task execution result. In some embodiments, the result processing unit can further trigger exception recovery, callback processing and file cleanup to complete the subsequent state convergence of the structured transmission task.

[0032] Example 2 refer to Figures 1 to 4 As shown, based on the same inventive concept as Embodiment 1, the difference is that this embodiment provides an automated SFTP task processing method for batch file transfer scenarios, specifically including: S101. Obtain directory change data and objectify the directory change data to generate candidate file objects; S102. Perform rule-based filtering, validity verification, and duplicate identification on candidate file objects to determine the file objects to be preprocessed; S103. Based on the file feature data and preprocessing rule data corresponding to the file object to be preprocessed, preprocess the file object to be preprocessed to generate a preprocessing result object. S104. Generate structured transfer tasks based on the preprocessing result objects, and determine the group key based on the target SFTP server IP and the target directory, and group structured transfer tasks with the same group key into the same group of tasks. S105. Establish a group-level execution context based on the group tasks, and execute the group-level structured transmission tasks within the group-level execution context to obtain the task execution results; S106. Update the task status of the structured transmission task and / or generate result records based on the task execution results.

[0033] Specifically, the system loads the directory monitoring configuration, target transmission configuration, and execution resource configuration, and establishes a monitoring context based on the directory to be monitored. The monitoring context is used to record basic information related to directory change detection, including at least the directory source, detection method, detection cycle, and the relationship with the target transmission task.

[0034] After the monitoring context is established, directory change data is obtained according to the directory type and monitoring method. For directories that support event listening, the first change data is obtained by listening to events. For directories that do not support event listening or require periodic inspection, the current directory status is scanned periodically and compared with historical snapshots to obtain the second change data. In the case where both the first and second change data exist, the first and second change data are uniformly sorted to extract the change results. The change results represent the changes of newly added or updated files.

[0035] Based on the unified and organized change results, candidate file objects are generated. These candidate file objects serve as a unified object form for subsequent processing steps. The candidate file objects include file source information, change attribute information, and discovery time information. For multiple change results obtained from different detection methods but pointing to the same file, deduplication processing is also performed on the change results to prevent the same file from repeatedly entering subsequent processing steps due to simultaneous monitoring and scanning. This transforms the directory change data into a set of candidate file objects that can be used for rule filtering and duplicate identification.

[0036] Read the file determination data corresponding to each candidate file object. The file determination data includes file name constraint information, file attribute constraint information and historical processing association information. Based on the file name constraint information and file attribute constraint information, perform rule matching on the candidate file object to determine whether the candidate file object meets the preset processing conditions.

[0037] For candidate file objects that fail rule matching, they are moved to the elimination path and will not proceed to subsequent preprocessing steps. For candidate file objects that pass rule matching, further validity checks are performed. This validity check does not simply determine if the file currently exists; instead, it involves acquiring attribute data for the file at multiple points in time and performing a stability assessment based on this data to determine if the file is still being written. If the attribute data at multiple points indicates that the file state is still changing, the file is considered unstable and will not be included in subsequent preprocessing steps. If the attribute data at multiple points indicates that the file state is stable, the candidate file object is considered to have passed the validity check.

[0038] For file objects that pass the validity check, duplicate identification is performed by combining file digest information and historical processing results. If the identification result shows that the current file has already met the duplicate condition in historical processing, the file object is marked as a duplicate object and subsequent processing is terminated. If the identification result shows that the current file has not met the duplicate condition, the file object is determined as a file object to be preprocessed, so that the file objects received by the subsequent preprocessing link have completed rule filtering, stability determination and duplicate diversion.

[0039] The preprocessing process reads file feature data and preprocessing rule data from the file objects to be preprocessed. The file feature data characterizes the file size, content format, and subsequent target location of the file objects. The preprocessing rule data characterizes small file merging rules, large file splitting rules, compression rules, and target directory preparation rules. Based on the file feature data and preprocessing rule data, a preprocessing strategy is determined. For multiple file objects that meet the small file condition and are consistent on the target SFTP server and target directory, they are grouped into the same merging batch and merged in a preset order to generate a merged result file. A merge mapping result is also generated simultaneously to record the correspondence between the original file objects and the merged result file. For file objects that meet the large file condition, a splitting process is performed, dividing the target file object into multiple fragments and generating a split mapping result to record the correspondence between the original file object and each fragment file. For file objects that meet the compression condition, compression is performed after merging or splitting to generate a compressed result file. After compression, the file object remains associated with the original input file object through the mapping result.

[0040] During preprocessing, information related to the target SFTP server and target directory is extracted simultaneously, and association information related to the preparation of the target directory is generated so that it can be directly read in subsequent task generation and group task execution stages. The output preprocessed result object includes the processed file object, the preprocessing type, and the mapping result between the processed file object and the original input file object.

[0041] The system reads the preprocessed result object, transmission target data, and task control data, and generates a structured transmission task. The transmission target data represents the target SFTP server, target directory, and target file location information. The task control data represents task parameters related to subsequent scheduling, execution, and status control. After the structured transmission task is generated, a group key is determined based on the target SFTP server IP and target directory. If the group task corresponding to the current group key already exists, the current structured transmission task is added to the corresponding group task. If the group task corresponding to the current group key does not yet exist, a new group task is created, and the current structured transmission task is used as a member task of the new group task. Through the group key grouping method, multiple structured transmission tasks with the same target location can be uniformly scheduled and continuously executed as the same group task in subsequent stages.

[0042] After obtaining the task execution result, priority is given to performing scheduling and verification processing on the group tasks. The group-level preparation status data, thread resource data, and task status data corresponding to the group tasks are read, and scheduling verification is performed based on the read results. If the target directory preparation status corresponding to the group task has not yet met the execution conditions, the group task is temporarily suspended from entering the execution stage. When the target directory preparation status meets the conditions, the target execution thread is determined for the group task based on the thread resource data. If the current group key has a historically bound thread and the historically bound thread is in an available state, the execution thread is reused first; otherwise, a new execution thread is allocated to the group task.

[0043] After determining the execution thread, further lock checks and retry eligibility checks are performed on the structured transmission tasks within the group. For structured transmission tasks that pass the checks, they are added to the set of tasks to be executed corresponding to the target execution thread. For structured transmission tasks that fail the lock checks, lock timeout checks, or retry eligibility checks, the tasks are skipped, lock released and rebuilt, or diverted to the final state failure based on the check results, so that all tasks in the set of tasks to be executed pass the group-level and task-level checks.

[0044] A group-level execution context is established based on the set of tasks to be executed. The execution thread reads the target connection data and target path data associated with the set of tasks to be executed, and requests a target SFTP server connection from the connection pool. After obtaining the target service connection, the validity of the target service connection is checked. If the connection verification passes, the connection is bound to the current group task. If the connection verification fails, the current connection is released and a new available connection is obtained.

[0045] If the target service connection is valid, continue with target directory preparation. If the target directory is already available, reuse the target directory context directly. If the target directory is not yet ready, perform directory checks and / or directory creation, and update the group task's readiness status after directory preparation is complete. By verifying the target service connection and the ready target directory, the system establishes a group-level execution context. Since multiple structured transfer tasks in the same group have the same target SFTP server IP and target directory, tasks in the same group can be executed continuously within the same group-level execution context without repeatedly establishing connections or verifying target directories for each task.

[0046] After the group-level execution context is established, the execution thread reads the objects to be transferred corresponding to the structured transfer tasks within the group in sequence and performs file transfer processing. When processing each structured transfer task, it first checks whether the current task status allows execution. If execution is allowed, the file to be transferred is written to the target file location corresponding to the target directory, and the execution result information related to this transfer is recorded. The execution result information includes the transfer start time, transfer end time, and transfer amount, etc.

[0047] After transmission is completed, the task execution result is determined based on the transmission return result. If the execution is successful, a successful task execution result is generated. If the execution fails, the current task is further classified according to the failure type. For connection anomalies such as connection interruption or network jitter, if the current number of retries has not reached the preset retry threshold, the current structured transmission task is updated to a retryable and scheduled state to enter subsequent recovery processing. For cases such as illegal target path, insufficient permissions, missing files to be transmitted, inaccessible preprocessed files, abnormal preprocessing mapping relationships, or missing preprocessing results, the corresponding structured transmission task is updated to a final failure state, so that each structured transmission task generates a corresponding task execution result.

[0048] Read the result data and status data corresponding to the task execution result, and perform result recording and status write-back processing. For successfully executed structured transmission tasks, generate the result record corresponding to the structured transmission task, update the task status of the structured transmission task to the success status, and record the status change trajectory.

[0049] For failed structured transmission tasks, a recovery path is determined based on the failure type and the current number of retries. If the failure type is a connection anomaly and the current number of retries has not reached the preset retry threshold, the group-level execution context state is used to determine whether the current group task has the conditions for recovery. If the group-level execution context still has the conditions for recovery, the current structured transmission task is rolled back to a retryable, pending scheduling state, and its group task is re-entered into the scheduling process. If the group-level execution context does not have the conditions for recovery, the group task is transferred to the connection recovery path or the anomaly pause path. If the failure type is a file missing anomaly, a preprocessing result anomaly, or the current number of retries has reached the preset retry threshold, a failure result record is written, and the current structured transmission task is updated to the final failure state.

[0050] After writing back the task execution result, a cleanup process is performed on the structured transmission task. Specifically, the task context data associated with the structured transmission task is read, and a callback context object is assembled based on the task context data. The task context data includes task execution status data, file association data, and preprocessing association data. The corresponding callback processor is determined according to the callback type. After passing the pre-callback validation, the callback processor is called to execute the callback logic, and the callback status of the structured transmission task is updated according to the callback execution result. If the callback execution is successful, the callback status is updated to a success status. If the callback execution fails, the current task is diverted to a callback retry path or a callback termination path based on the current callback retries and a preset callback threshold.

[0051] After callback processing is complete, file cleanup is performed according to the preprocessing mapping relationship. For result files formed by merging small files, the cleanup or archiving strategy for the original small files is determined based on the mapping relationship. For fragmented files formed by splitting large files, the original large files and intermediate fragmented files are processed uniformly after all fragmentation tasks are completed. For intermediate files formed by compression, deletion or archiving is performed after confirming that the transmission is complete and no longer needs to be retained. After cleanup is completed, the final state of the structured transmission task is determined by combining the task execution state, callback state, and cleanup state. If the task execution state, callback state, and cleanup state all meet the completion conditions, the structured transmission task is updated to the completed state. If there is a case where cleanup fails but compensation is allowed, it is updated to the pending cleanup and compensation state. If the task has entered the final failure state in the aforementioned steps, the final failure state remains unchanged.

[0052] After the final state of all structured transmission tasks within a group is determined, the group state of the corresponding group task is updated based on the final state of all structured transmission tasks within the group. When all tasks within the group have entered the final or completed state, the thread binding relationship and connection occupation relationship corresponding to the group task are released. This completes the entire processing chain for a single file or multiple files, from directory change detection, candidate object generation, filtering and identification, preprocessing, task grouping, group-level execution to result convergence.

[0053] Example 3 Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program implements the SFTP task automation processing method described in Embodiment 2. When the computer program is executed, it can sequentially complete the following processes: directory change acquisition, candidate file object generation, filtering and duplicate identification, preprocessing, structured task generation, group task grouping, group-level execution context establishment, intra-group transfer execution, result write-back, exception recovery, callback processing, file cleanup, and final state convergence.

[0054] In summary, this invention does not simply piece together directory monitoring, file filtering, duplicate identification, file preprocessing, and SFTP transfer as isolated functions. Instead, it establishes a complete chain around pre-processing file shaping, structured task orchestration, and group-level execution. By performing candidate objectification on the directory change results and filtering and duplicate identification on the candidate file objects, files entering the subsequent chain have clear processing boundaries. The preprocessing step shapes the file objects at a granular level and forms a mapping relationship, transforming input objects of different forms into processed objects suitable for transmission execution. Based on the target SFTP server IP and target directory, the group key is determined and a group-level execution context is constructed, enabling multiple structured transmission tasks with consistent target locations to be executed continuously in a unified connection context and directory context. Through task execution result status updates, exception recovery, callback handling, and file cleanup, the task chain from discovery, filtering, preprocessing, execution to result convergence forms a closed loop.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for automating SFTP task processing, characterized in that, include: Obtain directory change data, generate candidate file objects based on the directory change data, perform filtering and duplicate identification on the candidate file objects to determine the file objects to be preprocessed, and perform preprocessing on the file objects to be preprocessed to generate preprocessed result objects; A structured transfer task is generated based on the preprocessed result object, and a group key is determined based on the target SFTP server IP and the target directory. The structured transfer task is then grouped to generate a group task. A group-level execution context is established based on the group tasks, and the group-level structured transmission tasks are executed within the group-level execution context to obtain the task execution results; Update the task status of the structured transmission task and / or generate result records based on the task execution results.

2. The SFTP task automation processing method according to claim 1, characterized in that, The steps of acquiring directory change data, generating candidate file objects based on the directory change data, and performing filtering and duplicate identification on the candidate file objects to determine the file objects to be preprocessed include: A monitoring context is established based on the directory to be monitored, and first change data is obtained according to the event listening method and / or second change data is obtained according to the timed scanning method. The first change data and the second change data are processed in a unified manner to generate the candidate file object. The candidate file objects are subjected to rule-constrained data for rule matching, and the candidate file objects that pass the rule matching are subjected to validity verification. Duplicate identification is performed on the candidate file objects that pass the validity verification based on historical processing data to determine the file objects to be preprocessed. The validity verification includes obtaining attribute data of the file corresponding to the candidate file object at multiple times, and performing stability determination based on the attribute data obtained at multiple times.

3. The SFTP task automation processing method according to claim 1, characterized in that, The step of performing preprocessing on the file object to be preprocessed to generate a preprocessing result object includes: Based on the file feature data and preprocessing rule data corresponding to the file object to be preprocessed, a preprocessing strategy is determined; The preprocessing strategy includes performing at least one of merging, splitting, and compression on the file object to be preprocessed, generating a mapping result that characterizes the relationship between the processed file object and the original input file object, and generating a preprocessing result object based on the processed file object and the mapping result. The merging process includes merging multiple file objects to be preprocessed when they have the same target SFTP server IP and target directory and meet the content consistency condition. The splitting process includes splitting the target file object to be preprocessed into multiple fragmented files when the target file object to be preprocessed meets the splitting conditions.

4. The SFTP task automation processing method according to claim 1, characterized in that, The structured transmission task is generated based on the preprocessed result object, the transmission target data, and the task control data; The group key is determined based on the target SFTP server IP and the target directory, and structured transfer tasks with the same group key are grouped into the same group of tasks.

5. The SFTP task automation processing method according to claim 1, characterized in that, Establish a group-level execution context based on the group tasks, and execute the group-level structured transport tasks within the group-level execution context, including: Obtain the target service connection based on the target server data corresponding to the group task, perform validity verification on the target service connection, and perform directory check and / or directory creation according to the target directory corresponding to the group task if the target directory is not ready. The group-level execution context is established by validating the target service connection and the ready target directory, and the group-level structured transport task is executed in the group-level execution context.

6. The SFTP task automation processing method according to claim 5, characterized in that, Before establishing a group-level execution context based on the group tasks, the following is also included: Based on group-level preparation status data, thread resource data, and task status data, a scheduling check is performed on the group tasks. The target execution thread is determined according to the scheduling check results. Lock checks and retry eligibility checks are performed on the structured transmission tasks within the group. The structured transmission tasks that pass the lock checks and retry eligibility checks are added to the set of tasks to be executed corresponding to the target execution thread.

7. The SFTP task automation processing method according to claim 1, characterized in that, The step of updating the task status and / or generating result records of the structured transmission task based on the task execution result includes: When the task execution result indicates successful transmission, a result record corresponding to the structured transmission task is generated, and the task status of the structured transmission task is updated to a successful status. When the task execution result indicates a transmission failure, a recovery path is determined based on the failure type and the current number of retries, and the task status of the structured transmission task is updated to a retryable state or a final failure state. When the failure type is connection anomaly and the current number of retries has not reached the preset retry threshold, the structured transmission task is updated to a retryable and scheduled state. When the failure type is a file missing exception, a preprocessing result exception, or the current number of retries reaches the preset retry threshold, the structured transmission task is updated to a final failure state.

8. The SFTP task automation processing method according to claim 1, characterized in that, After updating the task status and / or generating result records of the structured transmission task based on the task execution result, the method further includes: Obtain task context data associated with the structured transmission task, assemble a callback context object using the task context data, determine a callback processor based on the callback type, call the callback processor to execute callback processing, and update the callback status of the structured transmission task based on the callback processing result. Based on the mapping results, task execution status, and / or callback status corresponding to the structured transmission task, file cleanup processing is performed. The final status of the structured transmission task is determined according to the file cleanup processing results, and the group status of the corresponding group task is updated according to the final status of all structured transmission tasks in the group.

9. An automated SFTP task processing device, characterized in that, include: A candidate object generation unit is used to acquire directory change data and generate candidate file objects based on the directory change data; The filtering and identification unit is used to perform filtering and duplicate identification on the candidate file objects to determine the file objects to be preprocessed. A preprocessing unit is used to perform preprocessing on the file object to be preprocessed in order to generate a preprocessing result object; The task generation unit is used to generate structured transmission tasks based on the preprocessed result object, and to determine the group key based on the target SFTP server IP and the target directory, and to group the structured transmission tasks to generate group tasks. An execution unit is configured to establish a group-level execution context based on the group tasks, and execute the group-level structured transmission tasks within the group-level execution context to obtain task execution results; The result processing unit is used to update the task status of the structured transmission task and / or generate result records based on the task execution result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images