A cold and hot judgment method for mass data in a distributed storage system

A distributed storage and distributed system technology, applied in the field of hot and cold judgment of massive data in distributed storage systems, can solve problems such as excessive disk IO, occupation of large computing resources, inability to record data access information, etc., to achieve accurate judgment , reduce storage space usage and the effect of

Inactive Publication Date: 2019-05-21
XI AN JIAOTONG UNIV
View PDF5 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, with the increase of data size, it will be impossible to record massive data access information in memory
This method will occupy a large amount of computing resources and br

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A cold and hot judgment method for mass data in a distributed storage system
  • A cold and hot judgment method for mass data in a distributed storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the basic idea of ​​the present invention, and are not used to limit the present invention. Those skilled in the art can understand other advantages and effects of the present invention from the content expressed in this specification. The present invention can also be implemented or applied in other different specific embodiments, and the details of this specification can also be based on different viewpoints and applications, and various modifications or changes can be made without departing from the spirit of the present invention.

[0024] The specific embodiment of the present invention provides a method for judging the hotness and coldness of massi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a cold and hot judgment method for mass data in a distributed storage system, which comprises the following steps of: counting the access frequency of data by adopting a multi-version hash table, and determining the cold and hot conditions of the data according to the access frequency. The method mainly comprises the steps of selecting a proper number of Hash functions, calculating corresponding Hash results of accessed indexes through the Hash functions in each time of data access, and then increasing values of the Hash results at corresponding positions in a Hash table of a current version; After multiple times of access, carrying out attenuation by switching the hash table of the current version and removing the first bits of all the hash tables of the version, so that the influence of old information is reduced; And finally, determining the cold and hot of the data by counting the data access information in all versions of hash tables. Compared with other methods in the field, the method has the advantages that higher cold and hot data judgment accuracy can be provided when the memory space with the same size is used, the cold and hot data are favorablyand respectively processed, and the system performance is improved.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a method for judging the hotness and coldness of massive data in a distributed storage system. Background technique [0002] With the extensive development and application of technologies such as the Internet, cloud computing, and the Internet of Things, data has shown explosive growth, and massive data needs to be processed and stored all the time, which poses a huge challenge to the performance and reliability of the storage system. . Distributed storage solutions are commonly used for mass data storage. The more famous distributed storage systems include Google's GFS (Google File System), the open source version of GFS HDFS (Hadoop Distributed FileSystem), Microsoft's WAS (Windows Azure Storage) and Ceph et al. However, statistics show that most of the data in the distributed storage system is cold data, which is rarely accessed, and the access is mainly concentrated on...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/13G06F16/17G06F16/182
Inventor 张兴军刘威董小社武旭瑞赵英交刘云飞
Owner XI AN JIAOTONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap