A method for unified analysis and processing of big data based on cloud computing

Abstract

The invention relates to a method for unified analysis and processing of big data based on cloud computing. The method includes: building a highly scalable distributed storage platform for massive structured, unstructured and semi-structured data based on cloud computing technology; Realize distributed parallel processing of massive heterogeneous data on the platform, analyze query and analysis requests of heterogeneous data, schedule data processing and calculation according to the location of the data object for query and analysis, distribute data analysis, processing and calculation to each data storage node, and realize massive Parallel analysis and processing of data; integrate structured data query and analysis interfaces and unstructured data query and analysis interfaces, realize parallel analysis and processing of heterogeneous data, and provide unified data access interfaces; provide structured data services for big data applications based on cloud service technology and unstructured data services. It has the advantages of overcoming the complexity and challenges of big data analysis and processing, and meeting the growing scale and real-time requirements of big data processing.

Description

technical field [0001] The invention relates to a distributed data processing technology, in particular to a method for unified analysis and processing of big data based on cloud computing. Background technique [0002] With the rapid development of applications such as the Internet, mobile Internet, and the Internet of Things, the amount of global data has exploded. According to the Digital Universe research report released by IDC, the total amount of global information will double every two years. In 2011, the total amount of data created and copied worldwide was 1.8ZB. IDC believes that by the next decade (2020), all IT departments in the world will have 10 times more servers than they do now, and manage 50 times more data than they do now. It is estimated that by 2020, the world will have a total of 35ZB of data volume. The rapid growth of data volume indicates that we have now entered an era of big data. However, at present, not only the scale of data is getting bigg...

AI Technical Summary

Problems solved by technology

The technical challenges brought by big data to traditional data analysis and processing technologies (such as parallel databases and data warehouses) are as follows: 1) Traditional data warehouse technologies can generally only handle TB-level data volumes. However, big data is often PB-level or even EB-level level, most parallel databases support limited expansion, and generally can be expanded to hundreds of nodes, and there are no application cases with thousands of nodes. Traditional data analysis and processing technologies cannot handle the high scalability and massive demands of big data; 2) big data It covers various types of data, including structured, semi-structured and unstructured data. The analysis of different types of data is different. Traditional data analysis and processing often only target a certain type of data and are relatively single. Big data analysis The methods are also diversified, including data mining, pattern recognition, data fusion and integration, time series analysis, etc. The increase of data types leads to the increase of the dimension of existing data space, which greatly increases the complexity of big data analysis and processing; 3) traditional The improvement of database processing capacity depends on the update and upgrade of CPU / memory / storage / network, while the processing mode of big data is a "scale-out" mode, and its performance improvement depends on the continuous increase in the distributed system. Low-cost computing and storage nodes; 4) The traditional data processing method is processor-centric, but in the big data environment, it is necessary to adopt a data-centric model to reduce the overhead caused by data movement. The traditional data processing method , can no longer meet the needs of big data

[0003] In short, compared with traditional relational databases, big data has the characteristics of huge data volume, complex structure, and many types, which pose new challenges to the storage, processing and analysis of big data. Moreover, the problem of big data has only recently been recognized by people. However, the existing methods cannot realize the analysis and processing of big data well.

Images