Data aggregation method and server

Abstract

The invention discloses a data aggregation method, which is suitable for being executed in a server. The server is respectively in communication connection with message middleware and a data index library. The data aggregation method comprises the following steps of: reading theme data of each computing node for different theme types from the message middleware, wherein the theme data comprises atimestamp and at least one index value; temporarily storing the theme data in a preset data flow queue, and extracting the theme data from the data flow queue piece by piece by adopting a preset filtering flow queue; aggregating the same theme data of the same computing node according to at least one aggregation period, and calculating an index statistical value of the computing node in each aggregation period; and summarizing the index statistical values of the plurality of aggregation periods in the preset index time period, taking the summarized index statistical values as data indexes of the computing nodes in the preset index time period, and storing the data indexes into a data index library. The invention further discloses a server for executing the data aggregation method.

Description

technical field [0001] The invention relates to the field of log processing, in particular to a data aggregation method and server. Background technique [0002] In recent years, high-performance computing technology has gradually developed and matured. With the increasing demand for computing power, the expansion of computing scale, and the changes in computing models, more and more users not only pay attention to the monitoring and analysis of macro performance data, but also pay great attention to micro performance. Association analysis of data. The E-level system has a large scale and supports complex subject applications. It needs to realize high-frequency and low-load collection, storage, analysis and mining of multi-dimensional management data and performance big data. For the massive performance data collected, it is very important to design data aggregation to realize effective multi-dimensional and multi-indicator performance data visual analysis, which puts forwa...

AI Technical Summary

Problems solved by technology

This method is not suitable for applications in scenarios with large amounts of data, and the single point of failure problem may produce inaccurate analysis results

Moreover, when a data aggregation request occurs, the server reads, calculates, and aggregates data synchronously, which is not friendly to some scenarios that require relatively high response speed, and for non-relational databases, the pressure will also be great. Momentary resource usage spikes may occur

[0004] In another solution, different data processing processes are designed according to different business needs, resulting in low efficiency and flexibility of data processing business; and the results of each business have strong business correlations, which cannot be provided to other businesses as input data, resulting in A lot of waste of computing power

Or each business will directly read the original data when requesting, which will cause a large number of repeated I / O operations, and even reduce the I / O performance of the original business, thereby affecting the original business system

Images