A method and system for merging and storing massive small files
What is AI technical title?
AI technical title is built by Patsnap AI team. It summarizes the technical point description of the patent document.
A massive small file and storage system technology, applied in the field of combined storage of massive small files, can solve problems such as storage structure and storage solution defects, efficiency problems, modification, deletion defects, etc.
Active Publication Date: 2020-12-08
AEROSPACE INFORMATION
View PDF3 Cites 0 Cited by
Summary
Abstract
Description
Claims
Application Information
AI Technical Summary
This helps you quickly interpret patents by identifying the three key elements:
Problems solved by technology
Method used
Benefits of technology
Problems solved by technology
[0004] Although the existing patents retrieved in the patent database have solved the problem of deployment on the Windows platform, there are serious defects in the storage structure and storage scheme. Although the files can be written and read normally, the There are serious flaws in the modification and deletion of
In addition, existing patents use a linked list structure when storing and locating header files, and there is an efficiency problem when accessing the last file
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more
Image
Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
Click on the blue label to locate the original text in one second.
Reading with bidirectional positioning of images and text.
Smart Image
Examples
Experimental program
Comparison scheme
Effect test
Embodiment 1
[0043] The reason why the continuous reading speed of batches of small files is slow is that the file headers are frequently read or the storage space of each small file on the disk is discontinuous. Therefore, in order to solve the above two defects, it is necessary to merge small files into large files and then store them. In this implementation, we refer to the merged files as library files for short.
[0044] This embodiment provides a processing flow of a method for merging and storing a large number of small files as follows: figure 1 As shown, in this embodiment, its processing steps are as follows:
[0045] Step 11, pre-establishing the map file as the positioning file index library file; wherein, the map file and the library file establish a mapping relationship according to the two-dimensional family sequence; as figure 2 As shown, a two-dimensional family sequence with the same number of rows and columns is established for each map file mapping, wherein each map f...
Embodiment 2
[0055] This embodiment provides a massive small file merge storage system, its specific implementation structure is as follows image 3 As shown, specifically, the following modules may be included: an establishment module 31, a reception module 32, a determination module 33, and a storage module 34; wherein,
[0056] Building module 31: it is used for pre-establishing the map file as the positioning file index library file; wherein, the map file and the library file establish a mapping relationship according to a two-dimensional family sequence;
[0057] Establishment module 31: it is specifically used to map each image file to establish a two-dimensional family sequence with the same number of rows and columns, wherein each image file corresponds to a library file, and each family in the two-dimensional family sequence corresponds to a library unit.
[0058] Receiving module 32: it is used for receiving small files;
[0059] Determining module 33: it is used for determinin...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More
PUM
Login to View More
Abstract
The embodiment of the invention provides a method and a system for merging and storing massive small files. The method comprises the steps of establishing a chart file as a locating file in advance to index a library file, wherein a mapping relation is established between the chart file and the library file according to a two-dimensional family sequence; receiving the small files; determining an idle storage space in the library file according to the chart file; merging and storing the small files by use of the idle storage space, wherein the chart file records positions and lengths of the small files in the library file and returns identifiers of the corresponding small files; and determining the positions and the lengths of the small files in the library file through the identifiers, which are returned by the chart file, of the small files. According to the method and the system, the problem of storage of the massive small files is solved, the file backup is easy, the occupation memory is less during running, the access speed is rapid, the occupation rate of a CPU (Central Processing Unit) is low, concurrent writing or reading can be carried out, and the method and the system bring low-cost operation and efficient processing capacity for an enterprise.
Description
technical field [0001] The invention relates to the field of data storage, in particular to a method and system for combining and storing massive small files. Background technique [0002] With the development of science and technology and the advancement of electronic equipment, we are in the era of data explosion. The number of picture data, attachments, and posts in ordinary web servers is increasing rapidly. The total amount of various files in general web servers is calculated as 1TB , the average website picture is about 30KB, and the result will be more than 35 million files. The traditional method is to store the above files directly in the server hard disk, which may easily cause some files to be damaged and cannot be read, because the huge number of files cannot be backed up, etc. Difficulties that plague enterprises and operation and maintenance personnel. [0003] Currently existing file storage solutions, such as GFS, HDFS, Luster, Ceph, mogileFS, FastDFS, Hado...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More
Application Information
Patent Timeline
Application Date:The date an application was filed.
Publication Date:The date a patent or application was officially published.
First Publication Date:The earliest publication date of a patent with the same application number.
Issue Date:Publication date of the patent grant document.
PCT Entry Date:The Entry date of PCT National Phase.
Estimated Expiry Date:The statutory expiry date of a patent right according to the Patent Law, and it is the longest term of protection that the patent right can achieve without the termination of the patent right due to other reasons(Term extension factor has been taken into account ).
Invalid Date:Actual expiry date is based on effective date or publication date of legal transaction data of invalid patent.
Login to View More 
Login to View More