System and method for realizing real-time data association in big data environment

A technology of real-time data and big data, applied in the field of real-time monitoring of computer applications and high-throughput streaming data

A technology of real-time data and big data, applied in the field of real-time monitoring of computer applications and high-throughput streaming data

CN103646111AActive Publication Date: 2014-03-19PRIMETON INFORMATION TECH
6 Cites 85 Cited by

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for realizing real-time data association in big data environment
  • System and method for realizing real-time data association in big data environment
  • System and method for realizing real-time data association in big data environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0173] In order to describe the technical content of the present invention more clearly, further description will be given below in conjunction with specific embodiments.

[0174]The invention relates to the data association link in the real-time monitoring or analysis system of high-throughput streaming data, and includes technical solutions such as an association model, a parallel association scheduling framework, and a NoSQL database-based distributed storage framework.

[0175] The data association system in the present invention mainly involves technical solutions and architectural ideas in three aspects: one is to design a set of general association models based on the idea of ​​monitoring context and association key values; the other is to filter, deduplicate / change, generate A set of parallel association scheduling framework is designed for the processing links such as association key value, intra-group association, and inter-group association; thirdly, a set of distrib...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a system and method for realizing real-time data association in the big data environment. The system comprises an association scheduling framework, a distributed storage framework, a relational database and Redis servers, wherein the association scheduling framework is used for conducting intra-class association, inter association and concurrent scheduling on business objects; the distributed storage framework is used for establishing a connection pool for a master server of each Redis server, storing data related to current data association, sending cold data surpassing updating time limit and excessive data surpassing memory capacity limit to the relational database, and inquiring key values of intra-class association and inter association transferred to the relational database; the relational database is used for storing key values of intra-class association, key values of inter association, intra-class contexts and monitoring contexts. Through the adoption of the system and method, real-time association supporting mass streaming data can be realized, the demand of complex association in various conditions can be met, the model generality is high, and the application range is relatively wide.

Description

technical field [0001] The invention relates to the field of computer applications, in particular to the field of real-time monitoring of high-throughput stream data, and specifically refers to a system and method for realizing real-time data association in a big data environment. Background technique [0002] The era of big data computing has arrived. This field is mainly divided into two directions: batch processing of static data and real-time processing of streaming data. Real-time data association is to receive data streams generated by multiple data sources in real time, and filter, deduplicate / change, generate association key values, intra-group association, inter-group association, multi-party splitting, and follow-up link triggering according to certain business correlations Wait for a series of processing. The associated data presents a group of data contexts, which can meet the needs of subsequent processing links such as process restoration, alarm judgment, tren...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
19 Mar 2014
Publication
CN103646111A
IPC
G06F17/30
CPC
G06F3/067; G06F11/3034; G06F16/24568; G06F16/284
Inventors
苏阳