A file fragmentation method based on HBase

A file fragmentation and file technology, applied in the computer field, can solve problems such as high single-point load, low data security, and inefficient storage

Active Publication Date: 2019-05-03
BEIJING SCISTOR TECH
View PDF6 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Aiming at the problems of high single-point load, inefficient storage and low data security when storing large files, the present inventio

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A file fragmentation method based on HBase
  • A file fragmentation method based on HBase
  • A file fragmentation method based on HBase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0045] An HBase-based file fragmentation method uses HBase as a storage medium for distributed storage of data content and a storage medium for fragment metadata information storage when large file data is stored in fragments. Such as figure 1 As shown, when large file data is written, it is divided into slices according to a certain granularity, and then concurrently written to HBase to distribute and store the data content of each slice in different node machines, and at the same time, store the storage elements of each slice Data information is added to the metadata information table of the large file in the form of a new column to complete the efficient and reliable storage of large file fragments. It mainly includes four parts: large file fra...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a file fragmentation method based on HBase, and belongs to the field of computers. The method comprises the following steps: firstly, reading a configuration file for fragmenting a large file and a configuration file read according to fragmentation granularity from a system, and fragmenting a certain large file; and writing and reading the file content according to the sequence of the fragments, and storing the content of each fragment in the HBase data table in an independent key value mode through a plurality of parallel threads. And meanwhile, storing the metadata information of each fragment in an HBase metadata information table in a newly added column manner. If the writing and reading of the file content is the specified slice, writing and reading the file content according to the specified slice number; if the write-in reading of the file content is random reading, setting a byte starting position and a read content size of the random reading; and finally, when the server is abnormal to cause writing or reading interruption, uploading or reading the fragment of the breakpoint again after the service is recovered. According to the method, the large file storage is more efficient and reliable, and the practicability and the adaptability are higher.

Description

technical field [0001] The invention belongs to the field of computers, in particular to an HBase-based file fragmentation method. Background technique [0002] With the development and application of Internet technology, the data generated by various industries such as social networks, mobile communications, online video, and e-commerce have shown explosive growth, and our lives are gradually entering the era of big data. In the era of big data, how to effectively store a large amount of data and then analyze and apply it has increasingly become the key to the development and advancement of enterprises in all walks of life. Especially in the face of large file storage, how to avoid excessive single-point load and how to achieve efficient and reliable storage are problems that need to be solved urgently. [0003] As a column-oriented distributed storage system for unstructured data, HBase provides a solution for distributed high-performance and reliable storage of unstructu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/13G06F16/16
Inventor 王振宇李斌斌苏连超
Owner BEIJING SCISTOR TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap