FTP-Transport transmission creation method and system for big data platform, terminal and storage medium

Abstract

The invention provides an FTP-Transport transmission creation method and system for a big data platform, a terminal and a storage medium. The method comprises the steps of opening a security communication protocol of the big data platform; creating a source file directory of a distributed file system data node, creating a distributed file system directory and creating a data warehouse tool data table; verifying and confirming the permission of passing through a working node security shell protocol; adding a file service and creating a channel from a working node to the data warehouse tool through the distributed file system through the file service; and uploading the data of the working node to the data warehouse tool data table through the channel. According to the method and the system,the file can be transmitted on the big data platform by utilizing FTP-Transport, so that the safety and instantaneity of file transmission and multi-end-to-multi-end cross-cluster file transmission are met.

Description

technical field [0001] The present invention relates to the technical field of servers, in particular to a method, system, terminal and storage medium for creating an FTP-Transport on a big data platform. Background technique [0002] FTP, SFTP, and FTPS are File Transfer Protocol (File Transfer Protocol), a multi-transfer protocol, and a secure file transfer protocol. FTP, SFTP, and FTPS are three commonly used secure communication protocols, which can transfer files relatively securely. [0003] However, at present, FTP, SFTP, and FTPS can only realize end-to-end file transfer from one node to another node, and cannot adapt to the scenario of high immediacy, high concurrency, and many-to-many transmission of data on the big data platform. Therefore, on the big data platform, these three secure communication protocols cannot be used to transmit data to HDFS and hive normally, and some real-time data cannot be updated to the data center in time, which is not suitable for hi...

AI Technical Summary

Problems solved by technology

[0003] However, at present, FTP, SFTP, and FTPS can only realize end-to-end file transfer from one node to another node, and cannot adapt to the scenario of high immediacy, high concurrency, and many-to-many transmission of data on the big data platform

Therefore, on the big data platform, these three secure communication protocols cannot be used to transmit data to HDFS and hive normally, and some real-time data cannot be updated to the data center in time, which is not suitable for high-performance, high-reliability, and safe data exchange of big data , nor for file transfers within or between clusters

Images