File synchronization operation method, terminal equipment and storage medium

Abstract

The invention relates to a file synchronization operation method, terminal equipment and a storage medium. The method comprises the steps that a file synchronization operation is achieved through a built universal task execution engine, wherein the task execution engine comprises a connection address configuration and more than one task component, each task component comprises a scanning componentand more than one responsibility chain component, the scanning component is used for scanning all files under a file directory, distributing the files scanned by the scanning component to each responsibility chain component in sequence, and configuring the thread number of all responsibility chain components under one task component during execution through a configurable variable, each responsibility chain component is composed of a plurality of sub-task components, the plurality of sub-task components are executed from top to bottom in sequence, and the sub-task components are used for executing file operation. According to the method, a multi-thread execution mode can be configured for each task, so that each task and each file can be processed in parallel, and the execution efficiencyis greatly improved.

Description

technical field [0001] The invention relates to the field of file operation, in particular to a file synchronization operation method, a terminal device and a storage medium. Background technique [0002] With the development of information technology, there are more and more demands for synchronization between files, including copying, filtering, and some common operations between local files, FTP service files, and HDFS files. There are mainly the following scenarios : [0003] 1. During the actual file synchronization process, it is often necessary to scan the files under a certain directory A, and copy them to multiple directories such as directory B, directory C, and directory D at the same time, and all directories A, B, C, Directory D may be a directory copy between different components such as local files, FTP directories, and HDFS. [0004] 2. When copying from data source directory A to directory B, directory C, and directory D, the actual requirements will also ...

AI Technical Summary

Problems solved by technology

[0005] 3. In the process of file copying, there are complex requirements for file encryption, file decryption, file name filtering, file size filtering, or some custom file processing, etc.

[0006] For the above complex operations such as copying between different components or the same component, file processing, file filtering, etc., the traditional method needs to tailor the development of each file copy, which greatly increases the development workload and work cycle, and seriously affects the file. The effectiveness of copying, and the copying tasks in different places are not common, low efficiency, low decoupling

Therefore, in view of the diversity and incompatibility between file copies, how to achieve unified management and unified configuration performance optimization is a great challenge

Images