System and method for realizing real-time data association in big data environment

A technology of real-time data and big data, applied in the field of real-time monitoring of computer applications and high-throughput streaming data

Active Publication Date: 2014-03-19
PRIMETON INFORMATION TECH
View PDF6 Cites 85 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The association itself is also hierarchical, including the generation of associated key values, intra-group association, inter-group association, parallel association, etc. This set of complex pro...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for realizing real-time data association in big data environment
  • System and method for realizing real-time data association in big data environment
  • System and method for realizing real-time data association in big data environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0173] In order to describe the technical content of the present invention more clearly, further description will be given below in conjunction with specific embodiments.

[0174]The invention relates to the data association link in the real-time monitoring or analysis system of high-throughput streaming data, and includes technical solutions such as an association model, a parallel association scheduling framework, and a NoSQL database-based distributed storage framework.

[0175] The data association system in the present invention mainly involves technical solutions and architectural ideas in three aspects: one is to design a set of general association models based on the idea of ​​monitoring context and association key values; the other is to filter, deduplicate / change, generate A set of parallel association scheduling framework is designed for the processing links such as association key value, intra-group association, and inter-group association; thirdly, a set of distrib...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a system and method for realizing real-time data association in the big data environment. The system comprises an association scheduling framework, a distributed storage framework, a relational database and Redis servers, wherein the association scheduling framework is used for conducting intra-class association, inter association and concurrent scheduling on business objects; the distributed storage framework is used for establishing a connection pool for a master server of each Redis server, storing data related to current data association, sending cold data surpassing updating time limit and excessive data surpassing memory capacity limit to the relational database, and inquiring key values of intra-class association and inter association transferred to the relational database; the relational database is used for storing key values of intra-class association, key values of inter association, intra-class contexts and monitoring contexts. Through the adoption of the system and method, real-time association supporting mass streaming data can be realized, the demand of complex association in various conditions can be met, the model generality is high, and the application range is relatively wide.

Description

technical field [0001] The invention relates to the field of computer applications, in particular to the field of real-time monitoring of high-throughput stream data, and specifically refers to a system and method for realizing real-time data association in a big data environment. Background technique [0002] The era of big data computing has arrived. This field is mainly divided into two directions: batch processing of static data and real-time processing of streaming data. Real-time data association is to receive data streams generated by multiple data sources in real time, and filter, deduplicate / change, generate association key values, intra-group association, inter-group association, multi-party splitting, and follow-up link triggering according to certain business correlations Wait for a series of processing. The associated data presents a group of data contexts, which can meet the needs of subsequent processing links such as process restoration, alarm judgment, tren...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F3/067G06F11/3034G06F16/24568G06F16/284
Inventor 苏阳
Owner PRIMETON INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap