Real-time big data processing system and method

Abstract

The invention relates to a real-time big data processing system and method, and the method comprises the steps: asynchronously receiving, compressing / decompressing data through employing an asynchronous data receiving assembly, and transmitting the data to a multi-thread data processing assembly; analyzing the data message from the asynchronous data receiving component by adopting a multi-thread data processing component, asynchronously processing the data message, and inserting the data message into a column storage database in batches; and a plurality of column storage databases with equal roles are adopted to provide a materialized view function to improve the storage query performance. By means of the data asynchronous receiving and concurrent analysis processing technology and based on column storage and related components, application performance index data can be analyzed, processed, stored and analyzed in real time under the condition that few system resources are utilized, and the operation and maintenance efficiency is improved.

Description

technical field [0001] The invention belongs to the field of application performance monitoring, in particular to a real-time big data processing technology, which is mainly used for big data processing and analysis with high throughput and low delay for monitoring data. Background technique [0002] At present, in the field of application performance monitoring, for relevant probes (agents) to collect data in real time, to receive, process, and store the data collected by the probes in real time and efficiently, there are problems of large hardware resource consumption and poor real-time performance, and it is difficult to simultaneously Consider both throughput and latency. Generally speaking, either the throughput rate is high, but the data delay is high, and it is difficult to achieve real-time performance; or the real-time performance is strong, but the data throughput rate is low. When both are taken into consideration, high system resources need to be consumed, so th...

AI Technical Summary

Problems solved by technology

[0002] At present, in the field of application performance monitoring, for relevant probes (agents) to collect data in real time, to receive, process, and store the data collected by the probes in real time and efficiently, there are problems of large hardware resource consumption and poor real-time performance, and it is difficult to simultaneously Consider both throughput and latency

Generally speaking, either the throughput rate is high, but the data delay is high, and it is difficult to achieve real-time performance; or the real-time performance is strong, but the data throughput rate is low

When both are taken into account, high system resources need to be consumed, so the solutions around traditional data receiving and processing can no longer meet the relevant performance requirements

[0003] For example, the traditional data receiving scheme generally adopts the BIO model, that is, one thread corresponds to one request, which will cause many request threads to be in a state of waiting for io resources, so they cannot be effectively utilized

Since creating threads is a relatively expensive system resource, when the amount of concurrency increases, the resource load of the system is heavy, which cannot meet the requirements of high throughput.

In addition, the stand-alone or master-slave design mode of traditional databases cannot meet the requirements for high throughput and low latency of databases.

Images