Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method for unified analysis and processing of big data based on cloud computing

A technology of analysis and processing and cloud computing, applied in the fields of digital data processing, special data processing applications, computing, etc., can solve the problems of increasing the complexity of big data analysis and processing, many types, and complex structure, and achieves the goal of overcoming the continuous growth and Real-time requirements, solving complex problems, and improving the effect of value

Active Publication Date: 2018-04-27
湖南建工德顺电子科技有限公司
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The technical challenges brought by big data to traditional data analysis and processing technologies (such as parallel databases and data warehouses) are as follows: 1) Traditional data warehouse technologies can generally only handle TB-level data volumes. However, big data is often PB-level or even EB-level level, most parallel databases support limited expansion, and generally can be expanded to hundreds of nodes, and there are no application cases with thousands of nodes. Traditional data analysis and processing technologies cannot handle the high scalability and massive demands of big data; 2) big data It covers various types of data, including structured, semi-structured and unstructured data. The analysis of different types of data is different. Traditional data analysis and processing often only target a certain type of data and are relatively single. Big data analysis The methods are also diversified, including data mining, pattern recognition, data fusion and integration, time series analysis, etc. The increase of data types leads to the increase of the dimension of existing data space, which greatly increases the complexity of big data analysis and processing; 3) traditional The improvement of database processing capacity depends on the update and upgrade of CPU / memory / storage / network, while the processing mode of big data is a "scale-out" mode, and its performance improvement depends on the continuous increase in the distributed system. Low-cost computing and storage nodes; 4) The traditional data processing method is processor-centric, but in the big data environment, it is necessary to adopt a data-centric model to reduce the overhead caused by data movement. The traditional data processing method , can no longer meet the needs of big data
[0003] In short, compared with traditional relational databases, big data has the characteristics of huge data volume, complex structure, and many types, which pose new challenges to the storage, processing and analysis of big data. Moreover, the problem of big data has only recently been recognized by people. However, the existing methods cannot realize the analysis and processing of big data well.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method for unified analysis and processing of big data based on cloud computing
  • A method for unified analysis and processing of big data based on cloud computing
  • A method for unified analysis and processing of big data based on cloud computing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0023] Apply the method of the present invention to the big data real-time query analysis platform:

[0024] Such as figure 1 As shown, it is a processing flowchart of the method for unified analysis and processing of big data based on cloud computing in this embodiment.

[0025] Applying the method of the present invention to realize the big data real-time query and analysis platform can provide real-time data query and analysis for OLTP type big data applications. In order to shorten the real-time data query time, a distributed query engine (including a distributed parallel scheduling layer and a query analysis execution engine layer) is realized by referring to the idea of ​​a traditional parallel relational database, accessing data and performing data analysis and processing on distributed data storage nodes. The overall structure of figure 2 As shown, the following layers are included:

[0026] 1) Data service provision layer: use cloud data services based on distribu...

Embodiment 2

[0032] Apply the method of the present invention to the big data comprehensive query analysis platform:

[0033] Applying the method of the present invention to realize the big data comprehensive query and analysis platform can realize the comprehensive query and analysis of structured, unstructured and semi-structured data, and provide a basic platform for the big data application of OLAP type. The overall structure of its implementation is as follows: image 3 As shown, the following layers are included:

[0034] 1) Data service provider layer: Provide cloud data services for OLAP-type big data applications.

[0035] 2) Integrated access interface layer: Provide SQL and MapReduce query and analysis interfaces, integrate structured data query and analysis interfaces and programming interfaces for unstructured data analysis and processing.

[0036] 3) Hadoop MapReduce layer: analyze big data query and analysis requests, and dispatch them to Hadoop and MPP relational database ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a method for unified analysis and processing of big data based on cloud computing. The method includes: building a highly scalable distributed storage platform for massive structured, unstructured and semi-structured data based on cloud computing technology; Realize distributed parallel processing of massive heterogeneous data on the platform, analyze query and analysis requests of heterogeneous data, schedule data processing and calculation according to the location of the data object for query and analysis, distribute data analysis, processing and calculation to each data storage node, and realize massive Parallel analysis and processing of data; integrate structured data query and analysis interfaces and unstructured data query and analysis interfaces, realize parallel analysis and processing of heterogeneous data, and provide unified data access interfaces; provide structured data services for big data applications based on cloud service technology and unstructured data services. It has the advantages of overcoming the complexity and challenges of big data analysis and processing, and meeting the growing scale and real-time requirements of big data processing.

Description

technical field [0001] The invention relates to a distributed data processing technology, in particular to a method for unified analysis and processing of big data based on cloud computing. Background technique [0002] With the rapid development of applications such as the Internet, mobile Internet, and the Internet of Things, the amount of global data has exploded. According to the Digital Universe research report released by IDC, the total amount of global information will double every two years. In 2011, the total amount of data created and copied worldwide was 1.8ZB. IDC believes that by the next decade (2020), all IT departments in the world will have 10 times more servers than they do now, and manage 50 times more data than they do now. It is estimated that by 2020, the world will have a total of 35ZB of data volume. The rapid growth of data volume indicates that we have now entered an era of big data. However, at present, not only the scale of data is getting bigg...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08G06F17/30
Inventor 林伟伟齐德昱
Owner 湖南建工德顺电子科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products