A method and device for distributed statistics of log access

Abstract

Embodiments of the present invention provide a method and device for distributed statistics of log visits. The method includes: obtaining multiple log data in real time in a server cluster of a distributed real-time computing system; dividing the obtained multiple log data For the first predetermined number of log data streams; create the first predetermined number of work units, and assign each log data stream to each work unit in one-to-one correspondence; in each work unit, for the current work unit Each log data in the allocated log data stream is counted, and the statistics database is updated in real time with the obtained log statistics data. Through the present invention, the load of each machine is greatly reduced; at the same time, since the execution operations on log files are greatly reduced, the data processing response speed of the entire cluster is greatly improved; fast and accurate real-time statistics are realized The number of effective user visits per day.

Description

technical field [0001] The invention relates to the technical field of computer data processing, in particular to a method and device for distributed statistics of log visits. Background technique [0002] In a large website, each time the server accepts a user request, an access log is generated. The log fields of the access log usually include access IP (Internet Protocol, a protocol for interconnection between networks), access time, access path, access duration, processing The machine of the log data and the status code of this HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol) request. If it is necessary to count the daily effective visits and total visits of the website, but because the number of website visit logs is very large and may reach hundreds of billions of records per day, it is not feasible to use conventional statistical methods to perform statistical operations on such logs Yes, because the processing speed of a single machine is far behind t...

AI Technical Summary

Problems solved by technology

[0004] 1. The load of each machine is huge and log loss often occurs, because the bandwidth of a single machine may fluctuate, so it may not be enough to receive all access logs, resulting in distortion of statistical results;

[0005] 2. Frequent access to the file system will slow down the system response;

[0006] 3. If any machine goes down, it will affect all statistics of the day;

[0007] 4. After the statistics are finished every day, the statistical data of the previous day will be obtained the next day, and the statistical data cannot be observed in real time

Images