Efficient, robust and safe large data polymerization system and method

Abstract

The invention provides an efficient, robust and safe large data polymerization system and method. The efficient, robust and safe large data polymerization method comprises the steps that mobile equipment log data is sent to web log collection servers through interfaces, and a Flume Agent process is arranged on each web log collection server; the Flume Agent process sends all log data of Agent to a central server by adopting a LoadBalanece Agent strategy in an equilibrium mode; and the data enters the central server and is written in a magnetic disk, and the log data stored in an extranet disk is transmitted to an intranet through GAP. According to the method, massive log data is collected by means of the open source software Apache Flume, Kafkachannel of Flume is used as a data gathering mode, which can both guarantee the efficiency of data collection, and meanwhile guarantee the data will not be lost due to single point failure, thereby having obvious advantages compared with memorychannel and filechannel of the Flume. After the massive log data is written in the disk, Hdfs is recorded through a custom interface, and a speed of hundreds of megabytes per second has been achieved, which approaches to the bottleneck of the traditional bandwidth of hundreds of megabytes and is close to the rotating speed of the magnetic disk.

Description

technical field [0001] The invention relates to the field of big data security, in particular to an efficient, stable and reliable method for transmitting massive log data between different network domains based on Flume. Background technique [0002] With the development of network information, network information security has begun to become a more and more important aspect of some security-related departments, especially the government, public security, banks and other departments. Therefore, exploring a way to isolate internal and external networks to maintain internal network security and ensure efficient, reliable, and stable transmission and collection of massive log data between internal and external networks can bring effective protection for the rapid development of finance, banking, and national security industries. GAP (Security Isolation Gatekeeper) can realize transmission between different network domains and logics through dedicated hardware, which can increa...

AI Technical Summary

Problems solved by technology

GAP (Security Isolation Gatekeeper) can realize transmission between different network domains and logics through dedicated hardware, which can increase the security of intranet data communication interaction to a certain extent, but has disadvantages when dealing with massive data transmission throughput

In addition, there is no perfect strategy for the integrity, repeatability, and timeliness of massive data during the transmission process, and it cannot provide a reliable and safe maintenance method

Images