Data receiving method and device

Abstract

The method provides a low-latency PingBack data receiving method and device. According to the method and device, a real-time data filtering path is separated from a batch processing path. Operating data is stored in a local queue of a PingBack server in the real-time data filtering path, so that order placing in the PingBack server is avoided; real-time filtration of the operating data is conducted in a common resource pool, the operating data stored in the local queue needs to be read only once, a plurality of filtration rules stored in the common resource pool are loaded, the operating data is filtered according to the filtration rules separately, and a task of data filtering is transferred from the PingBack server to the common resource pool. According to the method and device, reading of the operating data received from a client in the PingBack server for multiple times is avoided, the occupancy rate of CPU resources of the PingBack server is reduced, and delay of the real-time data filtering path is lowered.

Description

technical field [0001] The present invention relates to the technical field of communications, and more specifically, to a data receiving method and device. Background technique [0002] Online applications, especially various client applications, will report some running information to the server through buried points. By collecting this information, you can detect the health status of the online application, or collect some indicators of the user's use of the service, so as to improve the application or enhance the user experience. This method of collecting application information through buried points is called the Pingback data collection mechanism. The general structure is as follows figure 1 shown. [0003] At present, the process of receiving Pingback data is that each terminal or online service application sends data to each server in the Pingback cluster, and each server writes the data to the disk Disk after receiving the data, and the business setting filter rul...

AI Technical Summary

Problems solved by technology

[0004] However, the current framework PingbackServer first writes the data to the disk after receiving the data, and then reads the full amount of data from the disk when filtering, and stores it in the Kafka cluster after filtering, resulting in an increase in the delay of the real-time computing path

At the same time, if you need to add filtering rules, you need to start another program to read the full amount of data, and filter the full amount of data according to the added new filtering rules, which will cause the CPU usage of Pingback Server to increase continuously, resulting in delays in the real-time data filtering path

Images