Method for efficiently reading large number of small files in ftp service scene

Abstract

The invention discloses a method for efficiently reading a large number of small files in an ftp service scene, which comprises interaction localization and local file monitoring, and is characterized in that under the condition that parameters input by a front end are not changed, an input parameter obtained by the method is url of an ftp protocol, and the method is suitable for adapting strategy modes in a design mode to various ftp servers; the method aims at improving the monitoring logic efficiency of a large number of small files which are frequently created and improving the performance of cross-network service exchange.

Description

technical field [0001] The invention relates to the field of cross-network service exchange, in particular to a method for efficiently reading a large number of small files in an ftp service scene. Background technique [0002] The inter-network service exchange scenario needs to generate a large number of small files, which are exchanged to the peer network with the help of the file ferry function of the one-way optical gate, and then processed. At present, software applications mostly adopt two methods for file monitoring. [0003] The first method: through the ftp protocol, use the MLST command to poll to obtain the file list information. The information includes a file modification time in standard UTC format. The application can use this time to judge whether the time interval before and after the time is the same to judge whether the file is written (write can be judged as complete), the reason for the low efficiency of this method is that it needs to judge the time b...

AI Technical Summary

Problems solved by technology

[0003] The first method: through the ftp protocol, use the MLST command to poll to obtain the file list information. The information includes a file modification time in standard UTC format. The application can use this time to judge whether the time interval before and after the time is the same to judge whether the file is written (write can be judged as complete), the reason for the inefficiency of this method is that it needs to judge the time before and after multiple times

And the MLST command is not supported by all ftp services, and the commonly used vsftp in the industry does not support it

[0004] The second method: through the ftp protocol, use the LIST command to poll to obtain the file list information. Since the timestamp attribute of the returned file is based on the time zone of the server, it makes many differences when FTP synchronizes directories and files, because there is no way to obtain The timestamp of the current file on the server, so it can only be used to record the size of the file within a period of time, and judge the integrity of the file based on the fact that it will not change within a period of time. This method is inefficient

[0005] However, at present, the ftp protocol + polling method is used to judge, and the ftp protocol will cause a lot of network interaction, which is affected by network factors and has a large performance loss.

Use the polling method to discover files and judge files, causing most files to be processed untimely

There are limitations in using the MLST command, this method is not well compatible with various ftp services

Images