Method and system for merging and storing massive small files

A technology of massive small files and storage systems, applied in the field of combined storage of massive small files, can solve problems such as storage structure and storage scheme defects, efficiency problems, modification, deletion defects, etc.

Active Publication Date: 2017-07-25
AEROSPACE INFORMATION
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Although the existing patents retrieved in the patent database have solved the problem of deployment on the Windows platform, there are serious defects in the storage structure and storage scheme. Although the files can be written and

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for merging and storing massive small files
  • Method and system for merging and storing massive small files
  • Method and system for merging and storing massive small files

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0042] Example 1

[0043] The reason why the continuous reading speed of batch small files is slow is that the file header is frequently read or the storage space of each small file on the disk is not continuous. Therefore, in order to solve the above two defects, it is necessary to merge small files into large files and then store them. In this implementation, the merged files are referred to as library files for short.

[0044] This embodiment provides a processing flow of a method for merging and storing massive small files as follows: figure 1 As shown, in this embodiment, the processing steps are as follows:

[0045] Step 11, pre-establish a map file as a positioning file index library file; wherein, the map file and the library file establish a mapping relationship according to a two-dimensional family sequence; such as figure 2 As shown, a two-dimensional family sequence with the same number of rows and columns is established for each map file mapping, wherein each...

Example Embodiment

[0054] Embodiment 2

[0055] This embodiment provides a combined storage system for massive small files, the specific implementation structure of which is as follows image 3 As shown, it may specifically include the following modules: establishing module 31, receiving module 32, determining module 33, and storing module 34; wherein,

[0056] Establishment module 31: it is used for pre-establishing a map file as a positioning file index library file; wherein, the map file and the library file establish a mapping relationship according to a two-dimensional family sequence;

[0057] Establishment module 31: it is specifically used to map each graph file to establish a two-dimensional family sequence with the same number of rows and columns, wherein each graph file corresponds to a library file, and each family in the two-dimensional family sequence records a library corresponding to unit.

[0058] Receiving module 32: it is used for receiving small files;

[0059] Determini...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a method and a system for merging and storing massive small files. The method comprises the steps of establishing a chart file as a locating file in advance to index a library file, wherein a mapping relation is established between the chart file and the library file according to a two-dimensional family sequence; receiving the small files; determining an idle storage space in the library file according to the chart file; merging and storing the small files by use of the idle storage space, wherein the chart file records positions and lengths of the small files in the library file and returns identifiers of the corresponding small files; and determining the positions and the lengths of the small files in the library file through the identifiers, which are returned by the chart file, of the small files. According to the method and the system, the problem of storage of the massive small files is solved, the file backup is easy, the occupation memory is less during running, the access speed is rapid, the occupation rate of a CPU (Central Processing Unit) is low, concurrent writing or reading can be carried out, and the method and the system bring low-cost operation and efficient processing capacity for an enterprise.

Description

technical field [0001] The invention relates to the field of data storage, in particular to a method and system for combining and storing massive small files. Background technique [0002] With the development of science and technology and the advancement of electronic equipment, we are in the era of data explosion. The number of picture data, attachments, and posts in ordinary web servers is increasing rapidly. The total amount of various files in general web servers is calculated as 1TB , the average website picture is about 30KB, and the result will be more than 35 million files. The traditional method is to store the above files directly in the server hard disk, which may easily cause some files to be damaged and cannot be read, because the huge number of files cannot be backed up, etc. Difficulties that plague enterprises and operation and maintenance personnel. [0003] Currently existing file storage solutions, such as GFS, HDFS, Luster, Ceph, mogileFS, FastDFS, Hado...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/13
Inventor 冯尔斌张学军朱兴
Owner AEROSPACE INFORMATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap