Big data distributed storage method and system

A distributed storage and cloud storage system technology, applied in the field of big data distributed storage methods and systems, can solve the problems of massive data analysis cost bottlenecks and unproposed solutions, so as to improve distributed storage methods and increase query parallelism , good loading performance effect

Inactive Publication Date: 2014-09-24
SICHUAN FEDSTORE TECH
View PDF5 Cites 59 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, based on cost considerations, with the popularization of cloud computing service platforms, large-scale data analysis tasks are transferred from high-end servers deployed in parallel databases to cheaper low-end server clusters with a shared-nothing architecture. Cost bottlenecks that really need to be addressed
[0003] Therefore, for the above-mentioned problems existing in related technologies, effective solutions have not yet been proposed

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data distributed storage method and system

Examples

Experimental program
Comparison scheme
Effect test

specific Embodiment approach

[0042] One aspect of the present invention provides a large data distributed storage method and system. figure 1 It is a flowchart of a method for distributed storage of big data according to an embodiment of the present invention. Such as figure 1 Shown, implement the specific embodiment of the present invention as follows:

[0043] The cloud storage system is deployed on a shared-nothing cluster, uses Hadoop as the computing layer, and uses a single-node database as the storage layer to implement middleware technology. The cloud storage system is mainly divided into three parts: master node, distributed computing node (Hadoop node) and data node. Running the engine of the present invention on the master node is responsible for receiving user queries, compiling, converting and optimizing queries, generating query execution plans and executing queries, and also responsible for metadata management and node monitoring; Hadoop server processes running on Hadoop nodes, Responsi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a big data distributed storage method and system. The method comprises the steps of operating a data management engine on a main node, conducting compiling, conversion and optimization on user queries, generating and executing a query executing plan, and conducting metadata management and node monitoring; operating server processes on a distributed computational node and executing a distributed computation task; deploying the working processes of distributed computation and a single-node database on a data node; executing a subquery in the database or in a distributed computation frame. According to the big data distributed storage method and system, the opportunities that the queries are pushed down to the database to be executed are increased, data transmission cost caused by cross-node connection is avoided, and query performance is improved.

Description

technical field [0001] The invention relates to cloud storage, in particular to a large data distributed storage method and system. Background technique [0002] With the rapid development of applications such as the mobile Internet and the Internet of Things, the amount of global data has exploded. The rapid growth of data volume indicates that we have entered the era of big data. Network operators have a large number of users, and at the same time have the ability to control terminals and user Internet access channels, so that they have a good data foundation in user behavior analysis, in-depth analysis of user traffic behavior characteristics and rules, and discovering users' potential consumption needs. Effective means of value and operating levels. However, not only the scale of data is getting bigger and bigger, but also the variety of data types and real-time processing requirements have greatly increased the complexity of big data processing. Big data brings techn...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2255G06F16/24532G06F16/2471
Inventor 蒲思羽
Owner SICHUAN FEDSTORE TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap