Mass heterogeneous data parallel transmission method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a mass heterogeneous data parallel transmission method and device, equipment and a storage medium. The method comprises the following steps: activating a network authentication protocol of a distributed cluster and a parallel processing database cluster; establishing a data authentication relationship between the distributed cluster and the parallel processing database cluster according to a network authentication protocol; obtaining the access authority of the distributed cluster according to the data authentication relationship; and transmitting the first data in the HDFS to the parallel processing database cluster in parallel according to the access permission. According to the method, the data in the HDFS can be directly imported into the parallel processing database cluster, secondary data format conversion is not needed, and the transmission efficiency of mass heterogeneous data is improved.

Description

technical field [0001] The present invention relates to the field of big data technology, in particular to a method, device, equipment and storage medium for parallel transmission of massive heterogeneous data. Background technique [0002] Big data is usually used to describe a large amount of unstructured data, semi-structured data and structured data created by an enterprise. In recent years, with the rapid development of the Internet, the amount of unstructured data in big data has increased rapidly. According to IDC's survey report, 80% of the data in enterprises is unstructured data. Therefore, unstructured data has gradually become the main component of big data. [0003] When traditional big data platforms perform data aggregation, non-relational data such as unstructured data and semi-structured data are mainly stored by the distributed file system (Hadoop Distributed File System) in non-relational big data clusters, and structured data and other relational data ar...

AI Technical Summary

Problems solved by technology

Since the data types stored on the two platforms are non-relational and relational, respectively, when business data is transferred between the two platforms, it is necessary to export the non-relational data in the distributed file system to generate a text file, and then transfer it through the file. The protocol (File Transfer Protocol, FTP) sends the text file to the interface machine, and finally formats the text file and imports it into the MPP database. The process of using the above data transmission method is relatively complicated, which seriously affects the transmission efficiency of heterogeneous data.

Images