Big-data-analysis-oriented mixing computing system

A hybrid computing and big data technology, applied in the field of distributed computing, can solve problems such as the differentiated requirements of big data analysis, achieve efficient and fast data related processing, save configuration, and improve cluster utilization

Inactive Publication Date: 2015-06-24
JIANGSU R & D CENTER FOR INTERNET OF THINGS
View PDF2 Cites 41 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to solve the problem of differentiated requirements for big data analysis, and provide a hybrid computing system oriented to big data analysis, so as to achieve the purpose of efficiently analyzing big data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big-data-analysis-oriented mixing computing system
  • Big-data-analysis-oriented mixing computing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] The present invention will be further described below in conjunction with specific drawings and examples.

[0023] In order to achieve the best performance of the hybrid computing framework, after an in-depth analysis of the mainstream computing frameworks in the industry, it is evaluated from the perspectives of supported development languages, fault tolerance, fault recovery time, reliability, scalability, and code maturity. After considering the trade-offs of various indicators, the Spark distributed memory computing framework is selected to cooperate with MapReduce to realize hybrid computing of massive heterogeneous data. Through the integration of existing computing frameworks, the current big data analysis workflow is improved, and the lack of support for data analysis algorithms by a single computing framework is made up. While improving the degree of parallelism, the operating efficiency of the system is also improved.

[0024] The present invention designs a ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a big-data-analysis-oriented mixing computing system which comprises a cluster resource management system. The cluster resource management system operates on a Hadoop cluster and manages and dispatches Hadoop cluster resources in a unified mode, so that sharing of cluster resources by two computing frames of MapReduce and Spark is achieved. The cluster resource management system comprises a global resource manager, special application managers for the computing frames (namely a MapReduce application manager and a Spark application manager) and node managers on nodes. Big data analysis is carried out in the mode based on a mixing computing frame, the MapReduce computing frame and the Spark computing frame are integrated and are set up into the same Hadoop cluster, data sharing is achieved, in addition, different features of the MapReduce computing frame and the Spark computing frame can be used for the mode of computing frame mixing, in general, the cluster using rate is improved, energy loss is lowered, and data related processing can be carried out efficiently and quickly.

Description

technical field [0001] The invention relates to a hybrid computing system for big data analysis, which belongs to the technical field of distributed computing. Background technique [0002] With the popularization and development of the Internet and information technology, the data generated by humans is increasing exponentially, and big data has become the hottest topic today. Whether it is social networks, cloud computing, mobile Internet of Things, or smart cities, they are all closely related to big data. Big data refers to massive data sets with a very large amount of data and a wide variety of data, which cannot be managed and analyzed using traditional database tools. Big data has "5V" characteristics: large data volume (Volume), large data category (Variety), fast data processing speed (Velocity), high data authenticity (Veracity), and huge implied value (Value). The purpose of processing big data is to dig out the huge value contained in it through the analysis of...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/44
Inventor 田佳琦陈曙东
Owner JIANGSU R & D CENTER FOR INTERNET OF THINGS
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap