A real-time calculation big data system and method

Abstract

The invention belongs to the technical field of big data calculation, and discloses a real-time calculation big data system and method. The real-time calculation big data system comprises a data acquisition module, a data analysis module, a main control module, a distribution module, a parallel processing module, a merging module, a log processing module, a data storage module and a display module. The invention can quickly realize the development and replacement of the service logic through the data analysis module. At the same time, log data are serialized by log processing module with big data storage and calculation to form data stream, and the data collection and data stream of the whole server cluster use the same time series. The whole cluster adopts the same time series to realizea large number of servers in the cluster, which can quickly gather the time-dimension data through the time series, quickly access the server performance at a certain time, and analyze the real-time data flow of the cluster through the big data.

Description

technical field [0001] The invention belongs to the technical field of big data computing, and in particular relates to a real-time computing big data system and method. Background technique [0002] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain period of time. It is a massive, High growth rate and diverse information assets. The strategic significance of big data technology lies not in mastering huge data information, but in professional processing of these meaningful data. In other words, if big data is compared to an industry, the key to achieving profitability in this industry lies in improving the "processing capability" of data and realizing the "value-added" of data through "processing". From a technical point of view, the relationship between big data and cloud computing is as inseparable as the front and back of a coin. Big data cannot be processed by a single computer, and ...

AI Technical Summary

Problems solved by technology

However, the business logic development of the existing big data flow computing is relatively complicated, and it cannot be developed like the traditional data processing application Query. It will be more difficult to develop complex business logic with multiple iterations; The method of server operation and maintenance log data analysis and monitoring cannot realize real-time data collection to form a data stream, and the analysis can only analyze the static operation and maintenance logs for a period of time, which makes it impossible to effectively monitor and analyze the current server performance in a timely and effective manner , especially when there are a large number of large-scale cluster servers, data analysis cannot be realized.

[0003] To sum up, the problems existing in the existing technology are: the business logic development of the existing big data flow computing is relatively complicated, and it cannot be developed like the traditional data processing application Query. If complex multiple iterations of the business logic are encountered It is even more difficult to develop; at the same time, the existing server operation and maintenance log data analysis and monitoring methods cannot realize real-time data collection to form a data stream, and the analysis can only analyze the static operation and maintenance logs for a period of time, which leads to the inability to timely and effectively Effectively monitor and analyze the current server performance, especially when there are a large number of large-scale cluster servers, data analysis cannot be realized.

The existing parallel algorithms cannot make full use of the computing resources of the parallel system, the effect of parallelizing the algorithm data is not ideal, and the convergence speed of the algorithm and the characteristics of the optimal classification of the algorithm cannot be guaranteed, resulting in a decrease in parallel efficiency

Images