Big data processing and solving system simultaneously supporting offline data and real-time online data

A technology of big data processing and data processing module, applied in the field of big data processing solution system, can solve complex and arduous problems, and can no longer process the results, etc., to achieve the effect of improving stability and reliability, and improving performance and stability

Inactive Publication Date: 2016-06-15
BEJING HSRT INFORMATION TECH CO LTD
View PDF3 Cites 43 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Neither of the two tools at their disposal could quite solve the problem: a scalable high-latency batch processing system for historical data, and a low-latency streaming system that could no longer process the results
[0003] Hadoop framework brings batch data processing, but real-time processing of web-scale big data remains a challenge
There are many techniques that can be used to build such a complete data processing system, but choosing the right tools and orchestrating their use can be complex and daunting

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data processing and solving system simultaneously supporting offline data and real-time online data
  • Big data processing and solving system simultaneously supporting offline data and real-time online data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] Such as figure 1 As shown, the big data processing system that supports both offline data and real-time online data includes: data acquisition module, data preprocessing module, unified configuration center, distributed file storage module, distributed real-time stream computing module, offline data processing module, database , Data comprehensive analysis query module and comprehensive display module.

[0042] Data acquisition module:

[0043] (1) Read the configuration information from the unified configuration center, and incrementally import the data in a relational database (such as MySQL, Oracle, etc.) into a distributed file storage module, such as HDFS, through regular cycle scheduling. For example, import the user information tables and product details tables stored in the Oracle database, and use this data as basic data in the subsequent log processing to cooperate with log data for analysis, calculation, and other processing. The data acquisition module can...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a big data processing and solving system simultaneously supporting offline data and real-time online data. The system comprises a data collecting module, a preprocessing module, a distributed storage module, a distributed real-time flow calculating module, an offline data processing module, a database, a data comprehensive analysis and query module, a comprehensive showing module and a uniform configuration center. The big data processing and solving system can process the real-time data and the offline data, and is timely in processing and high in processing efficiency.

Description

technical field [0001] The invention relates to a big data processing solution, in particular to a complete big data processing solution system that simultaneously supports offline data and real-time online data. Background technique [0002] As technology develops, there is an increasing need to build complex and low-latency processing systems. Neither of the two tools at their disposal could quite solve the problem: a scalable high-latency batch processing system for processing historical data, and a low-latency streaming system that could no longer process the results. But by linking these two tools together, a usable solution can be built. [0003] The Hadoop framework brings batch data processing, but real-time processing of web-scale big data remains a challenge. There are many techniques that can be used to build such a complete data processing system, but selecting the appropriate tools and orchestrating their use can be complex and daunting. Contents of the inve...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/182G06F16/13
Inventor 许丹霞刘寅汪伟郑宇
Owner BEJING HSRT INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap