File transmission method, device and distributed cluster file system

Abstract

The invention discloses a file transmission method and device and a distributed cluster file system, which belongs to the technical field of information. The method comprises the steps that the information of a file to be transmitted in a distributed coordination node cluster is acquired; according to the information of the file to be transmitted and the heartbeat information of at least one auxiliary push node, the facts that the file to be transmitted is saved in at least one auxiliary push node, the heartbeat information at least carries the file information saved by the auxiliary push node, and the file to be transmitted is pulled by at least one auxiliary push node from a file storage node cluster are determined; a file transmission path is generated, wherein the file transmission path comprises at least one destination node cluster; and the file transmission path is sent to at least one auxiliary push node. According to the invention, the distributed coordination node cluster is used to maintain the information of the file to be transmitted; and a push node cluster generates the file transmission path and a specific transmission implementation process according to the information of the file to be transmitted, which prevents the phenomenon of single point bottleneck.

Description

technical field [0001] The invention relates to the field of information technology, in particular to a file transmission method, device and distributed cluster file system. Background technique [0002] As the data scale becomes larger and larger, the file system gradually develops into a distributed cluster file system. The distributed cluster file system may include a database, an interface machine, a scheduling machine, and multiple destination nodes; among them, the database is used to record and display file information, the interface machine is used to store files, etc., and the scheduling machine is used to schedule and manage distributed clusters Tasks in the file system, the destination node is used to store files. [0003] For a distributed cluster file system, one of its important functions is how to quickly transfer files to the destination node. [0004] Generally, when using a distributed cluster file system for file transfer, the dispatcher detects whether ...

AI Technical Summary

Problems solved by technology

[0006] Since the existing transmission process is that multiple destination nodes request files from the interface machine, and because the interface machine is one or a few devices, its load capacity is limited, and when transferring files to multiple destination nodes, the pressure it bears is relatively high. It is easy to reach the bottleneck of device disk IO, which not only has a great impact on other running programs on the machine, but also affects the transmission of the entire distributed cluster file system, and the stability of the database used to maintain the file information to be transmitted is poor. When the database host fails, the scheduler cannot schedule the file transfer, resulting in a paralyzed state of the entire system

Images