Method and device for distributed log visitor volume counting

Abstract

Embodiments of the invention provide a method and a device for distributed log visitor volume counting. The method comprises: in a server cluster of a distributed real-time calculating system, obtaining a plurality of pieces of log data in real time; dividing the obtained plurality of pieces of log data to log data streams in a first preset number; establishing working units in the first preset number, and correspondingly distributing each log data stream to each working unit in a one-to-one manner; in each working unit, counting each piece of log data in the distributed log data streams in the current working unit, and using the obtained log statistical data to update a database in real time. Through the method and the device, load of each machine is greatly reduced. Since execution operation on log files is greatly reduced, data processing response speed of the whole cluster is greatly improved. The method and the device realize to rapidly and accurately count user effective visit times in real time every day.

Description

technical field [0001] The invention relates to the technical field of computer data processing, in particular to a method and device for distributed statistics of log visits. Background technique [0002] In a large website, each time the server accepts a user request, an access log is generated. The log fields of the access log usually include access IP (Internet Protocol, a protocol for interconnection between networks), access time, access path, access duration, processing The machine of the log data and the status code of this HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol) request. If it is necessary to count the daily effective visits and total visits of the website, but because the number of website visit logs is very large and may reach hundreds of billions of records per day, it is not feasible to use conventional statistical methods to perform statistical operations on such logs Yes, because the processing speed of a single machine is far behind t...

AI Technical Summary

Problems solved by technology

[0004] 1. The load of each machine is huge and log loss often occurs, because the bandwidth of a single machine may fluctuate, so it may not be enough to receive all access logs, resulting in distortion of statistical results;

[0005] 2. Frequent access to the file system will slow down the system response;

[0006] 3. If any machine goes down, it will affect all statistics of the day;

[0007] 4. After the statistics are finished every day, the statistical data of the previous day will be obtained the next day, and the statistical data cannot be observed in real time

Images