Distributed storage method

A technology of distributed storage and distributed computing, applied in the field of data processing, can solve the problems of unprovided solutions and the cost bottleneck of massive data analysis, so as to improve query parallelism, improve distributed storage methods, and avoid data transmission costs. Effect

Pending Publication Date: 2020-03-24
四川中讯易科科技有限公司
View PDF4 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, based on cost considerations, with the popularization of cloud computing service platforms, large-scale data analysis tasks are transferred from high-end servers deployed in parallel databases to cheaper low-end server clusters with a shared-nothing architecture. Cost bottlenecks that really need to be addressed
Therefore, for the above-mentioned problems existing in related technologies, effective solutions have not yet been proposed

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed storage method

Examples

Experimental program
Comparison scheme
Effect test

specific Embodiment approach

[0022] One aspect of the present invention provides a distributed storage method and system. figure 1 It is a flowchart of a distributed storage method according to an embodiment of the present invention. Such as figure 1 Shown, implement the specific embodiment of the present invention as follows:

[0023] The cloud storage system is deployed on a shared-nothing cluster, uses Hadoop as the computing layer, and uses a single-node database as the storage layer to implement middleware technology. The cloud storage system is mainly divided into three parts: master node, distributed computing node (Hadoop node) and data node. Running the engine of the present invention on the master node is responsible for receiving user queries, compiling, converting and optimizing queries, generating query execution plans and executing queries, and also responsible for metadata management and node monitoring; Hadoop server processes running on Hadoop nodes, Responsible for executing Hadoop ta...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed storage method. The method is used for realizing storage and query of big data in a cloud storage system. The cloud storage system comprises a main node, a distributed computing node and a data node, a data management engine is operated on the main node, a user query is received, the query is compiled, converted and optimized, a query execution plan is generated and the query is executed, and metadata management and node monitoring are carried out at the same time; a server process is operated on the distributed computing node, and a distributed computingtask is executed; the working process of distributed computing and a single-node database are deployed at a data node, and a data table is stored in the database of the data node; the sub-queries converted from the user query are executed in a database or in a distributed computing framework. A hybrid data warehouse architecture incorporating a database and a distributed computing framework is presented.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a distributed storage method. Background technique [0002] With the rapid development of applications such as the mobile Internet and the Internet of Things, the amount of global data has exploded. The rapid growth of data volume indicates that we have entered the era of big data. Network operators have a large number of users, and at the same time have the ability to control terminals and user Internet access channels, so that they have a good data foundation in user behavior analysis, in-depth analysis of user traffic behavior characteristics and rules, and discovering users' potential consumption needs. Effective means of value and operating levels. However, not only the scale of data is getting bigger and bigger, but also the variety of data types and real-time processing requirements have greatly increased the complexity of big data processing. Big data brings te...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/22G06F16/2458
CPCG06F16/27G06F16/2471G06F16/2282
Inventor 康俊忠蒲思羽
Owner 四川中讯易科科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap