Big-data parallel computing method and system based on distributed columnar storage

A distributed columnar and parallel computing technology, applied in the field of big data processing, can solve problems such as slow computing speed, reduce time consumption, improve data query efficiency, and ensure real-time query analysis.
CN107329982AInactive Publication Date: 2017-11-07SOUTH CHINA UNIV OF TECH
4 Cites 44 Cited by

Patent Information

Authority / Receiving Office
CN Β· China
Current Assignee / Owner
SOUTH CHINA UNIV OF TECH
Publication Date
2017-11-07
Estimated Expiration
Not applicable Β· inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a big-data parallel computing method and system based on distributed columnar storage. Data which is most often accessed currently is stored by using the NoSQL columnar storage based on a memory, the cache optimizing function is achieved, and quick data query is achieved; a distributed cluster architecture, big data storing demands are met, and the dynamic scalability of the data storage capacity is achieved; combined with a parallel computing framework based on Spark, the data analysis and the parallel operation of a business layer are achieved, and the computing speed is increased; the real-time data visual experience of the large-screen rolling analysis is achieved by using a graph and diagram engine. In the big-data parallel computing method and system, the memory processing performance and the parallel computing advantages of a distributed cloud server are given full play, the bottlenecks of a single server and serial computing performance are overcome, the redundant data transmission between data nodes is avoided, the real-time response speed of the system is increased, and quick big-data analysis is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the technical field of big data processing, in particular to a large data parallel computing method and system based on distributed columnar storage. Background technique

[0002] The rapid development of the Internet and the continuous upgrading and replacement of hardware have caused the data scale of various units such as governments and enterprises to show explosive growth, and gradually move towards massive data. Faced with the storage and processing requirements of massive data, traditional relational databases are mainly based on the operation of tables and data rows, which has gradually failed to meet user needs, and even restricts the storage and processing of massive data. Therefore, relying solely on traditional storage technology cannot meet the development and needs of the times. It is necessary to establish a new big data storage technology based on traditional processing technology to ensure that data sto...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More