Mass file transmission method and system

Abstract

The invention relates to a mass file transmission method and system, and the system comprises a transmission center management module which is disposed on a transmission center server and is used formanaging and displaying the machine information of each node and managing a transmission rule, a file management module deployed on each node and used for creating and monitoring a file temporary cache directory, a transmission management module deployed on each node, and a file transmission module deployed on each node. Each transmission management module creates a unique host name and reports the unique host name to the transmission center management module for registration when the node is installed for the first time, and is used for reporting heartbeat packets to the transmission center server at regular time, requesting the transmission center server for transmission rule issuing at regular time and storing the information locally, and requesting a transmission center server for issuing all host lists at regular time and storing the host lists locally; and each file transmission module is deployed on each node and is used for completing a file transmission task. According to themethod, the temporary cache directory is intelligently created and managed, so that the problem that the compression of massive files is restricted by file enumeration is solved, and the method can adapt to big data TB-level file transmission.

Description

Technical field[0001]The present invention relates to the field of document transmission, and in particular to a massive document transmission method and system.Background technique[0002]With the rapid development of cloud computing, big data and Internet of Things, the amount of corporate acquisition data has grown sharply, and massive documents are produced. Massive documents are corporate business cornerstones, and it is also a treasure house that digs commercial value. In particular, the business characteristics of the big data industry will generate some or large or small data files in real time and usually store the same directory simultaneously. Since the operating system's file is stored to disk space, it is not continuous, causing the operating system in the file The time overhead of the plug-in mass file is very large.[0003]It has always been downloaded without dedicated tools since the massive document, and the traditional FTP cannot meet the requirements. FTP is fixed wh...

AI Technical Summary

Problems solved by technology

In this case, transfer tools such as FTP cannot transfer at all, and file transfer and sharing are very difficult

[0004] In order to solve the problem of mass file transfer, the enterprise transfer method is to adopt multi-threaded transfer after compression and then decompression process, which is limited by the compression efficiency of large files, which will make the Transmission efficiency is not high

Some enterprises adopt the file concatenation transmission method for massive small files, but the transmission network environment is complex and changeable, and this method will also be restricted

For example, when the network bandwidth is not ideal or the network is interrupted, files will also accumulate due to untimely transmission, which will still cause delays in the file enumeration in the transmission link

Images