A data aggregation method and server

Abstract

The invention discloses a data aggregation method, which is suitable for execution in a server, and the server is respectively connected to a message middleware and a data index database. type of topic data, the topic data includes a timestamp and at least one indicator value; temporarily store the topic data in a preset data flow queue, and use a preset filter flow queue to extract topics one by one from the data flow queue Data; aggregate the same topic data of the same computing node according to at least one aggregation period, and calculate the indicator statistics of the computing node in each aggregation period; aggregate the indicator statistics of multiple aggregation periods within the preset index period Summarize and store in the data index library as the data index of the computing node in the preset index period. The invention also discloses a server for executing the method.

Description

technical field [0001] The invention relates to the field of log processing, in particular to a data aggregation method and server. Background technique [0002] In recent years, high-performance computing technology has gradually developed and matured. With the increasing demand for computing power, the expansion of computing scale, and the changes in computing models, more and more users not only pay attention to the monitoring and analysis of macro performance data, but also pay great attention to micro performance. Association analysis of data. The E-level system has a large scale and supports complex subject applications. It needs to realize high-frequency and low-load collection, storage, analysis and mining of multi-dimensional management data and performance big data. For the massive performance data collected, it is very important to design data aggregation to realize effective multi-dimensional and multi-indicator performance data visual analysis, which puts forwa...

AI Technical Summary

Problems solved by technology

This method is not suitable for applications in scenarios with large amounts of data, and the single point of failure problem may produce inaccurate analysis results

Moreover, when a data aggregation request occurs, the server reads, calculates, and aggregates data synchronously, which is not friendly to some scenarios that require relatively high response speed, and for non-relational databases, the pressure will also be great. Momentary resource usage spikes may occur

[0004] In another solution, different data processing processes are designed according to different business needs, resulting in low efficiency and flexibility of data processing business; and the results of each business have strong business correlations, which cannot be provided to other businesses as input data, resulting in A lot of waste of computing power

Or each business will directly read the original data when requesting, which will cause a large number of repeated I / O operations, and even reduce the I / O performance of the original business, thereby affecting the original business system

Images