Data storage method, device and equipment and computer readable storage medium

A data storage and data technology, applied in the field of data processing, can solve the problems of not improving data reading speed, bandwidth consumption, data storage space consumes a large amount of data, etc., so as to ensure high data availability and avoid large consumption

Pending Publication Date: 2022-08-05
创优数字科技(广东)有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In Hadoop, in order to achieve high availability of cluster storage, Hadoop’s HDFS file storage system provides two storage methods: Block replication (that is, multiple copies of file blocks) and Erasure coding (that is, erasure coding technology), but the storage method of Block replication is in While improving data availability, it also causes 200% additional overhead of storage space, and additional bandwidth consumption when writing data; while Erasure coding storage method is better than Blockreplication storage method in terms of storage space usage and data writing speed Great improvement, but did not improve the data reading speed
Therefore, in the file storage system with Hadoop as the core, it is a difficult choice not only to meet the high availability of data, but also to avoid a large consumption of data storage space, and to ensure the speed of reading and writing data.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data storage method, device and equipment and computer readable storage medium
  • Data storage method, device and equipment and computer readable storage medium
  • Data storage method, device and equipment and computer readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0063] The present application designs a data storage method, device, device and computer-readable storage medium, which can be applied to a file storage system with Hadoop as the core, can overcome the defects of the storage method provided by the existing Hadoop, and meet the needs of the file storage system. High availability of data, avoid massive consumption of data storage space, and ensure data read and write speed.

[0064] In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data storage method, device and equipment and a computer readable storage medium, which can be suitable for a file storage system taking Hadoop as a core, can overcome the defects of a storage mode provided by the existing Hadoop, meets the high availability of data in the file storage system, avoids the large consumption of data storage space, and ensures the read-write speed of the data. The storage method comprises the steps that access attributes of a target storage file in the file storage system are determined, and the access attributes comprise a cold attribute and a hot attribute; generating a target storage strategy matched with the access attribute for the target storage file; querying a historical storage strategy of the target storage file in the file storage system; and according to a comparison result of the target storage strategy and the historical storage strategy, executing a storage operation on the target storage file.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, to a data storage method, apparatus, device, and computer-readable storage medium. Background technique [0002] With the development of science and technology, in the field of big data, there are many big data components with Hadoop as the core, such as Hive, Spark, Presto, Impala, etc. [0003] The so-called Hadoop is a distributed system infrastructure. Through Hadoop, users can develop distributed programs without knowing the underlying details of the distribution to make full use of clusters for high-speed computing and storage. [0004] In Hadoop, in order to achieve high availability of cluster storage, Hadoop's HDFS file storage system provides two storage methods, Block replication (that is, multiple copies of file blocks) and Erasure coding (that is, erasure coding technology). While improving data availability, it also causes 200% additional overhe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/14G06F16/13G06F16/18G06F16/182
CPCG06F16/148G06F16/1815G06F16/182G06F16/13
Inventor 邹杰
Owner 创优数字科技(广东)有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap