SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data

A hybrid storage and big data technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., to achieve the effect of promoting progress and development

Inactive Publication Date: 2014-12-17
TIANJIN UNIV
View PDF5 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to overcome the above-mentioned prior art, the present invention proposes a hybrid storage method of SSD disks and HDD disks for distributed big data, comprehensively utilizes HDD disks and SSD disks, and implements a hybrid storage solution based on SSD / HDD through the JS-model model , solve big data storage management problems in a distributed environment, and finally achieve the effectiveness and efficiency of big data management, and prepare for big data storage and public release

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data
  • SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data
  • SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, but the implementation scope of the present invention is not limited thereto.

[0027] The specific implementation mode that the present invention adopts comprises following flow process:

[0028] Step 1. Research on the existing big data storage is mainly based on HDD. JS-model is different from the traditional data storage method. The stored data is persisted in files. The files include a journal file and multiple segment files; for large-scale data, build JS -model storage model: binary group, data record, data item, timestamp, basic concept of JS collection, journal file, segment file, and four data file operations on it: build, move, split, and merge;

[0029] Step 2, build a new SSD / HDD hybrid distributed storage solution based on the JS-model, namely HDStore;

[0030] Step 3, use HDStore to manage journal and segment files on different ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data. The SSD and HDD hybrid storage method comprises the following steps of step (1) establishing a JS-model storage model aiming at the distributed big data of each node in a cluster system, wherein the JS-model storage model comprises a Journal file and multiple segment files, the Journal file is used for buffering data and carrying out quick read-write operation on the data, and the segment files are used for persistently and stably storing the data and supporting the data to be added with reading and data random access; step (2) establishing an SSD and HDD hybrid distributed storage model HDStore based on the JS-model storage model; step (3) managing the Journal file and the segment files by utilizing the HDStore, and optimizing data reading and writing; step (4) downloading and generating a Lubm data set, preprocessing the data set, loading the data in a Bigdata system, and testing the loading time and the searching time. According to the SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method disclosed by the invention, a hybrid distributed storage scheme is adopted for effectively and efficiently managing semantic big data, and thus the improvement and the development of large-scale data storage management are facilitated.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to hybrid distributed fast storage of semantic big data. Background technique [0002] With the popularization and development of modern Web technology and the rapid growth of information, human beings have entered the era of big data, which poses a huge challenge to traditional data management methods. At the same time, big data technology is gradually emerging. Big data technology is composed of various technologies, such as parallel computing, distributed file system, distributed database, scalable storage system and so on. Among them, one of the key technologies is how to effectively and efficiently store and manage big data. In order to solve the above problems, there are several options for data storage management, such as classifying data according to their importance, making reasonable arrangements for data processing, or using distributed storage technology to improve...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/172G06F16/182
Inventor 冯志杰冯志勇王鑫饶国政
Owner TIANJIN UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap