Real-time big data processing system and method

A big data processing and data processing technology, applied in the direction of electrical digital data processing, special data processing applications, data processing input/output process, etc., can solve problems such as high throughput, large hardware resource consumption, high system resources, etc. , to achieve the effect of improving customer experience, locating performance faults, and improving operation and maintenance efficiency

Pending Publication Date: 2022-05-13
CLOUD WISDOM BEIJING TECH
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] At present, in the field of application performance monitoring, for relevant probes (agents) to collect data in real time, to receive, process, and store the data collected by the probes in real time and efficiently, there are problems of large hardware resource consumption and poor real-time performance, and it is difficult to simultaneously Consider both throughput and latency
Generally speaking, either the throughput rate is high, but the data delay is high, and it is difficult to achieve real-time performance; or the real-time performance is strong, but the data throughput rate is low
When both are taken into account, high system resources need to be consumed, so the solutions around traditional data receiving and processing can no longer meet the relevant performance requirements
[0003] For example, the traditional data receiving scheme generally adopts the BIO model, that is, one thread corresponds to one request, which will cause many request threads to be in a state of waiting for io resources, so they cannot be effectively utilized
Since creating threads is a relatively expensive system resource, when the amount of concurrency increases, the resource load of the system is heavy, which cannot meet the requirements of high throughput.
In addition, the stand-alone or master-slave design mode of traditional databases cannot meet the requirements for high throughput and low latency of databases.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time big data processing system and method
  • Real-time big data processing system and method
  • Real-time big data processing system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] In order to better explain the present invention and facilitate understanding, the present invention will be described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0042] see figure 1 , the real-time big data processing technique of one embodiment of the present invention, it comprises:

[0043]The data receiving component ApmCollector is used for receiving and compressing asynchronous data (such as probe data);

[0044] Several data processing and analysis components ApmConsumer, distributed layout, concurrent analysis, used for data cleaning;

[0045] Database column storage component, used for data storage and multi-dimensional query;

[0046] The database query analysis component provides the data analysis query results required by the user.

[0047] see figure 2 , the real-time big data processing technology of one embodiment of the present invention, wherein:

[0048] Thread includes 3 parts, the first part is...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a real-time big data processing system and method, and the method comprises the steps: asynchronously receiving, compressing / decompressing data through employing an asynchronous data receiving assembly, and transmitting the data to a multi-thread data processing assembly; analyzing the data message from the asynchronous data receiving component by adopting a multi-thread data processing component, asynchronously processing the data message, and inserting the data message into a column storage database in batches; and a plurality of column storage databases with equal roles are adopted to provide a materialized view function to improve the storage query performance. By means of the data asynchronous receiving and concurrent analysis processing technology and based on column storage and related components, application performance index data can be analyzed, processed, stored and analyzed in real time under the condition that few system resources are utilized, and the operation and maintenance efficiency is improved.

Description

technical field [0001] The invention belongs to the field of application performance monitoring, in particular to a real-time big data processing technology, which is mainly used for big data processing and analysis with high throughput and low delay for monitoring data. Background technique [0002] At present, in the field of application performance monitoring, for relevant probes (agents) to collect data in real time, to receive, process, and store the data collected by the probes in real time and efficiently, there are problems of large hardware resource consumption and poor real-time performance, and it is difficult to simultaneously Consider both throughput and latency. Generally speaking, either the throughput rate is high, but the data delay is high, and it is difficult to achieve real-time performance; or the real-time performance is strong, but the data throughput rate is low. When both are taken into consideration, high system resources need to be consumed, so th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06G06F16/2453
CPCG06F3/0613G06F3/0676G06F3/067G06F3/061G06F3/0602G06F16/2453
Inventor 赖彩林韩成冰
Owner CLOUD WISDOM BEIJING TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products