Big data log intelligent routing and storage system and method

Abstract

The invention discloses a big data log intelligent routing and storage system and a method. The invention belongs to the technical field of big data log management. The system comprises a Routor filedistribution routing assembly, a Procesor processing assembly, an Aggregate aggregation assembly and a Storage Adapter assembly, and is characterized in that the Procesor processing assembly is connected with the Aggregate aggregation assembly and the Storage Adapter assembly; the method comprises the following steps of: establishing a log management framework comprising a Routor file distributionrouting component, a Procesor processing component, an Aggregate aggregation component and a StorageAdapter storage adapter component on a big data platform; a processing component Procesor used forjudging log types is added between aggregation and storage of logs, different types of log files are stored in different storage media; efficient aggregation, simplification, classification and storage of multi-component massive logs are achieved, and the overall cost of operation and maintenance is reduced.

Description

technical field [0001] The invention belongs to the technical field of big data log management, and in particular relates to a big data log intelligent routing and storage system and method. Background technique [0002] Big data platforms often have many components, such as batch processing component Spark, stream processing component Storm / sparkStreaming, message component Kafka, coordination component Zookeeper, offline computing component Hadoop, etc. A large number of logs running on each component need to be effectively managed , so that the operation and maintenance personnel can keep abreast of the operating status of the platform, as well as troubleshooting and optimization when encountering failures, so this kind of platform-level massive log management puts forward higher requirements for efficiency, reliability, and low maintenance costs. [0003] At present, management platforms or methods for big data logs generally include the following: [0004] (1) Each com...

AI Technical Summary

Problems solved by technology

[0007] ①In the existing technology (1), the log management platform or method only realizes the aggregation and storage of the full amount of logs, which requires the largest amount of storage space and occupies the largest amount of network bandwidth resources. The stored data is also used for subsequent analysis. Facing the problem of massive analysis pressure

At the same time, each component maintains its own log local path before the log is aggregated. The release of a new version of the program is very likely to change the log directory, and then the log will be lost.

[0008] ②Compared with technology (1), the difference is that the existing technology (2) has an additional filtering link for logs, which effectively reduces the amount of logs, filters out unwanted logs, and saves log storage space, but However, it is necessary to manually set and maintain rules, and the workload is heavy; on the other hand, if the version of the component program changes or the coder changes the file name, the original rules will not apply, which is very likely to cause the loss of logs

Images