Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A high-efficiency storage method for massive large object data

A large-object, high-efficiency technology that is applied in the field of massive data storage to achieve the effect of reducing storage space and deployment costs

Active Publication Date: 2020-12-01
FOCUS TECH
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problems existing in the existing large object storage technology, the present invention divides the large object record into two parts, meta and blob, and stores them in two mutually independent distributed Key-Value storage systems (hereinafter respectively referred to as: meta -cluster and blob-cluster)

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A high-efficiency storage method for massive large object data
  • A high-efficiency storage method for massive large object data
  • A high-efficiency storage method for massive large object data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] The following are explanations of technical terms that may appear in this program:

[0043] blob MD5: A 32-byte hexadecimal string calculated by Message-Digest Algorithm 5 (Information-Digest Algorithm 5).

[0044] meta-cluster: used to store the meta information part, its key is the large object record identifier, and its value is the meta information of the large object record, the meta information includes the MD5 code corresponding to the blob value.

[0045] blob-cluster: used to store the blob part, its key is the MD5 code of the blob value of the large object record, and its value is the blob value of the large object record.

[0046] Blob data reference: In the present invention, meta information and blob values ​​are stored separately in meta-cluster and blob-cluster, and large object records can share a certain blob record in blob-cluster (if the MD5 codes of their blob parts are the same); In order to accurately track whether a blob record is referenced by a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a high-efficiency storage method for massive big object data. It is characterized in that the method comprises the following steps: 1, deploying a distributed Key-Value clusterwhich is in communication connection with the client for receiving big object data uploaded by a client, wherein one set of cluster is meta-cluster, and one set is a blob-cluster, wherein the meta-cluster is used for storing a meta information part, the blob-cluster comprises a refresh-db used for receiving the reference, and the blob-cluster is used for storing a blob value; 2, processing a large object record writing scene; 3, processing a large object record deletion scene; 4, processing a large object record copy scene; and step 5, processing a large object record updating scene. According to the method, the effects of removing the duplicate of the blob value, greatly reducing the storage occupied space of the data so as to reduce the deployment cost, greatly improving the efficiencyof updating the meta information recorded by the large object, independently operating and maintaining and the like can be achieved.

Description

technical field [0001] The invention relates to the field of massive data storage, in particular to a high-efficiency storage method for massive large object data. Background technique [0002] In recent years, with the vigorous development of Internet applications, massive amounts of multimedia data such as video, audio, and pictures have been generated. In order to store these massive large object data, a large number of distributed NOSQL storage products have emerged. A considerable part of them are distributed storage systems with key-value structure. Due to their good scalability and simple data structure, they are used by many Internet companies in the storage scenarios of massive large object data, and many enterprises Use them to store PB or even EB-level data. However, with the rapid growth of storage capacity, people have gradually realized that in many Internet application scenarios, the blob part of massive large object data actually has a lot of duplication; fo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/22
CPCG06F16/2219
Inventor 梁峰
Owner FOCUS TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com