Log data transmission method, device and equipment

Abstract

The invention discloses a log data processing method, device and equipment. The method comprises the following steps: monitoring log data generated by an application program; sending the log data to afirst theme area of a Kafka message queue in real time; when log data pushing is abnormal, retrying to send the log data, and if retrying succeeds, stopping sending the log data to the Kafka messagequeue; if the retry is not successful, further determining whether the sending frequency of the log data reaches a threshold value or not; if the log data reaches the threshold value, sending the logdata to a second theme area; and if the log data does not reach the threshold value, retrying to send the log data to the first theme area, and carrying out counting statistics to count the sending times of the log data. According to the embodiment of the invention, when the log data of the application program is sent abnormally, the sending error of the log data is avoided by determining the resending result of the log data, and the accuracy of log data transmission is effectively improved.

Description

technical field [0001] The present application generally relates to the technical field of data processing, and in particular relates to log data processing methods, devices and equipment. Background technique [0002] With the development of e-commerce technology, the pressure on the back-end server of the network is also increasing. At the same time, the "data" that needs to be processed is also increasing geometrically. Real-time and accurate collection, transmission, and calculation of massive logs have become an e-commerce urgent demands. [0003] Tools for collecting logs in the prior art, such as Flume, Scribe, Chukwa, ELK, etc., follow the traditional log reporting method, that is, the log data is first recorded to the file system, and the log changes are reported through the collection agent, which may be incremental or Collect log data in full and send it to the message queue. The above method leads to a certain delay in log collection and reporting, resulting in...

AI Technical Summary

Problems solved by technology

The above method leads to a certain delay in log collection and reporting, resulting in poor real-time reporting effect

[0004] At the same time, after the log data is collected and reported to the message queue, only relying on the feedback mechanism of the message queue itself, resulting in errors in the reported log data

Images