An object storage-based large-scale data cloud storage method

An object storage and data cloud technology, which is applied in electrical digital data processing, special data processing applications, instruments, etc., can solve problems such as reducing the efficiency of compressed data access, unable to effectively exert the advantages of object storage performance expansion, and save time and cost. Effect

Active Publication Date: 2017-07-28
GENETALKS BIO TECH CHANGSHA CO LTD
View PDF5 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since the size of the compressed package of large data is not small (more than a few GB), the Get and Put operations for such a large object cannot effectively take advantage of the performance expansion advantages of object storage and greatly reduce the access efficiency of compressed data.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • An object storage-based large-scale data cloud storage method
  • An object storage-based large-scale data cloud storage method
  • An object storage-based large-scale data cloud storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 2

[0050] The method of this embodiment is basically the same as that of the first embodiment. The difference is that the large data file to be stored is different. The large data file to be stored in this embodiment is a non-FASTQ file, and the non-FASTQ file is different from the non-FASTQ file. Therefore, this embodiment Step 1) When forming at least one data substream from the read file stream, it essentially forms a data substream (binary file stream) from the read file stream. When compressing, first adopt block sorting compression Do preprocessing, and then use arithmetic coding based on the bit-level dynamic prediction model to compress the data in each block again.

[0051] Combining the files in the first embodiment and the second embodiment, it can be seen that the large-scale data cloud storage method based on object storage of the present invention is not limited to a specific large-scale data file type. The large-scale data file's own characteristics are used to control...

Embodiment 3

[0053] The method of this embodiment is basically the same as that of the first embodiment. The difference is that this embodiment is a cloud platform-oriented large-scale data cloud storage method based on object storage. The method of this embodiment only requires the client to provide the data to be stored in the required format. The general instance of a large data file does not limit the specific implementation form of the client to the general instance.

[0054] In this embodiment, the large-scale data cloud storage method based on object storage, the implementation steps include:

[0055] S1) The cloud platform establishes the root container object containing the block container object based on the object;

[0056] S2) The cloud platform receives the output example sent by the client for the large data file to be stored. The output example includes the data block and its description information. The description information includes the data substream to which the data block be...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an object storage-based large-scale data cloud storage method. The method comprises the steps that a client reads to-be stored large-scale data files and forms at least one kind of data sub-stream, forms data blocks with a fixed size through continuous accumulation in memory, compresses the data blocks and description information thereof to form output examples, and sends the output examples to a cloud platform; the cloud platform establishes root container objects containing block container objects, receives the output examples sent by the client for the to-be stored large-scale data files and stores the received output examples into the corresponding root container objects as objects, wherein output examples of each data sub-flow are stored in more than one block container object. The method is based on data streams, data blocks and concurrent compression, supports synchronization of data compression and transmission to cloud and supports targeted compression schemes for data blocks of different data sub-streams; the method can greatly save time cost of data uploading and economic cost of data storage.

Description

Technical field [0001] The invention relates to cloud storage technology for large-scale data, in particular to a large-scale data cloud storage method based on object storage. Background technique [0002] The era of large-scale data and the era of cloud have both come, and cloud computing platforms have become effective platforms for large-scale data processing. Typical industries represented by biology, finance, and communications, produce hundreds of gigabytes or even terabytes of data locally every day. Limited by the bandwidth of the wide area network, the speed at which these massive amounts of large data are transmitted to the cloud platform has become a bottleneck restricting the use of cloud computing resources for data processing in these areas. In addition, the high storage cost of large-scale data cloud platforms has also become one of the important reasons for the limitation of enterprise cloud use. [0003] Compressed storage is an effective method for data storage...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F19/28H04L29/08
CPCH04L67/1097G06F16/13G06F16/183G16B50/00H04L67/561H04L67/5651
Inventor 李根宋卓冯博伦王振国
Owner GENETALKS BIO TECH CHANGSHA CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap