Data storage system based on Hadoop architecture

A data storage system and Hadoop cluster technology, applied in the field of big data storage, can solve the problem that the real-time performance of MapReduce-led data processing cannot meet the requirements, the scalability and fault tolerance have not changed, and the parallel database-led scalability and fault tolerance Poor and other issues

CN107800808AInactive Publication Date: 2018-03-13GUANGDONG AOFEI DATA TECHNOLOGY CO LTD
9 Cites 8 Cited by

Patent Information

Authority / Receiving Office
CN · China
Current Assignee / Owner
Publication Date
2018-03-13
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure 1
    Figure 1
Patent Text Reader

Abstract

The invention relates to a data storage system based on a Hadoop architecture. The storage system comprises at least one application server, a backup server, a database cluster and at least one core layer switch; the database cluster comprises a first sub-storage cluster and a second sub-storage cluster; basic data of structured data are stored in the first sub-storage cluster, and unstructured and semi-structured loose data are stored in the second sub-storage cluster; the application server, the backup server and the database cluster are respectively connected to the core layer switch; and the application server is connected with the backup server and the database cluster, and used for managing real time data of the backup server and the database cluster. The data storage system based onthe Hadoop architecture provided by the invention uses a distributed storage mode to store the data and uses redundant storage to ensure the reliability of the data. An HDFS (Hadoop Distributed FileSystem) module can reliably store massive files across machines, and stores various files as data block sequences of the same size.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of big data storage, and in particular relates to a data storage system based on Hadoop architecture. Background technique

[0002] Data can be divided into structured data, semi-structured data and unstructured data by type. Structured data refers to a data type that can be expressed in a two-dimensional structure and can be stored in a relational database; semi-structured data refers to A data type with a certain structure but not clear semantics, such as emails, HTML web pages, etc. Some of their fields are definite, and some of them are not. Unstructured data refers to a data type that cannot be represented by a two-dimensional structure. Various data types, mainly including office documents, texts, pictures, audio and video files, etc., cannot be processed by relational databases. With the rise and development of social networks, a large amount of UGC (User Generated Content, User Generated Content) ha...

Examples

Embodiment Construction

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other implementations obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

[0031] A data storage system based on Hadoop architecture, the storage system includes at least one application server, backup server, database cluster and at least one core layer switch;

[0032] The database cluster includes a first sub-storage cluster and a second sub-storage cluster; the basic data of the structured data is stored in the first sub-storage cluster, and the unstructured and semi-structured loose data is stored in the second sub-storage cluster...