Method and system for collecting log data

Abstract

The invention discloses a log data collection method and system. The log data collection method comprises the steps of dividing a target region of to-be-collected log data into at least two collection regions in advance, wherein each collection region comprises a data center and at least a branch mechanism, and a log storage system is located at the data center of a first collection region of the at least two collection regions; collecting log data of web servers in each branch mechanism and each data center, and transmitting the collected log data of each branch mechanism to the data center of the collection region where the branch mechanism locates; and storing the log data of each collection region into the log storage system through a Flume first-class receiving end of the data center of the first collection region. The log collection scheme provided by the invention integrates characteristics such as high availability, high reliably and high timeliness, and increases the work efficiency.

Description

technical field [0001] The invention relates to data processing technology, specifically a log collection method and system. Background technique [0002] At present, e-commerce and Internet finance have developed rapidly. While online transactions bring convenience to users, they also face potential risks, such as user account theft, financial fraud and money laundering. Therefore, enterprises have increasingly strong demand for monitoring transaction risks. When traditional enterprises monitor transaction risks, they usually set up a risk control department within the enterprise to conduct offline analysis of user transactions and intervene after suspicious data are found. With the development of big data technology, transaction risk control is gradually becoming digitized and intelligent. With the help of big data, transaction risk monitoring not only saves manpower and material resources, improves work efficiency, but also effectively reduces losses caused by economic...

AI Technical Summary

Problems solved by technology

[0005] In order to overcome the deficiencies in dealing with abnormal events in traditional log collection, resulting in data loss, log analysis errors, etc., the embodiment of the present invention provides a log data collection method, the method includes:

Images