A Storage Resource Management Method Based on Storage Medium Type and Weighted Quota
A technology for storing resources and storage media, applied in resource allocation, electrical digital data processing, input/output process of data processing, etc., to achieve the effects of ensuring scalability and fault tolerance, reducing memory consumption, and high robustness
Inactive Publication Date: 2020-01-14
BEIHANG UNIV
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, in the existing storage resource management or allocation methods, there is no method that can effectively satisfy all the above conditions
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0039] The method for managing storage resources based on storage media and weighted consistent hashing proposed by the present invention will be explained in detail below in conjunction with the accompanying drawings.
[0040] The storage resource management method of the present invention should include the following steps: mounting of storage devices, mounting of user space file systems, running Hadoop applications, and distribution of read and write requests of Hadoop applications. Its system structure is as figure 1 shown. First, the storage device is mounted to different directories according to different storage media, and then the file operation requests for these directories are encapsulated through the user space file system. Then start the user space file system through multiple processes to provide FUSE file services, and mount it to the intermediate data storage directory of the Hadoop system to improve the concurrent utilization of storage resources. Then by mo...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a storage resource management method based on storage medium types and weighting quotas. The method comprises the steps of mounting of storage apparatuses and user space file systems and distribution of reading-writing requests, wherein weighting distribution characteristics are imported into a Weighting Jump algorithm adopted in distribution of reading-writing requests, so that high calculating speed, stateless scheduling and extremely low memory consumption are achieved. The method is implemented by employing a method similar to jump in probability proposed by John Lamping and Eric Veach of Google. The method can guarantee that the distribution of storage resources complies with set weights at the level of probability, minimize standard error and achieve lower time complexity and memory consumption than a resource management system Yarn of an original Hadoop cluster. The method solves the problem of low efficiency in search directory polling by a disk resource distribution algorithm achieved in the conventional resource management system Yarn, has extendibility and can minimize movement of original data in probability during extension.
Description
technical field [0001] The invention relates to a storage resource management method based on storage medium type and weighted quota, and belongs to the fields of big data storage and distributed computing. Background technique [0002] With the development of big data storage and hybrid storage media technology, storage resource management and allocation methods are becoming more and more important and facing more and more severe tests. The Hadoop big data processing system developed by the Apache open source community has become the most representative solution in the field of big data. Hadoop includes HDFS (Hadoop Distributed FileSystem) distributed file system and Yarn (Yet Another Resource Negotiator) resource management system and MapReduce data processing framework. Among them, the running MapReduce application’s access to data is divided into access to the HDFS distributed file system when reading input files in the map stage, access to the HDFS distributed file sys...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06G06F9/50
CPCG06F3/0604G06F3/0608G06F3/0611G06F3/0614G06F3/0631G06F3/0635G06F3/0643G06F3/0649G06F3/0659G06F3/067G06F9/5083
Inventor 吴文峻冯梦琦
Owner BEIHANG UNIV