A method for fast aggregation and reading of multiple files in a shared file system
A technology of sharing files and aggregation methods, which is applied in the field of fast aggregation of multiple files, can solve the problems of occupying the input and output capacity of disk arrays, SAN network transmission bandwidth, reducing metadata server storage bandwidth, and slowing down SAN network operation, etc., to improve cluster computing speed, reduced data storage bandwidth, and reduced aggregation time
Inactive Publication Date: 2017-11-17
BEIJING DAYANG TECH DEV
View PDF8 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
This additional physical aggregation process brought about by using distributed technology takes a long time to aggregate, and once there are too many nodes for cluster computing, a large number of files to be aggregated will be generated, thus the time spent on aggregation also longer
At the same time, if this physical aggregation method is used, it will take a very long aggregation time, making the operation of the entire SAN network slow down, and reducing the storage bandwidth of the metadata server for external services for a long time
Not only that, physical aggregation requires a read operation for each file, and at the same time, a write operation to the shared storage (usually a disk array) for the newly generated physical aggregation file. The read and write operations of these files also occupy a large number of disk arrays. The input and output capabilities and the transmission bandwidth of the SAN network
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment 1
[0041] This embodiment is a method for fast aggregation of multiple files in a shared file system and reading the aggregated file. The hardware system used by the method includes: multiple clients ( figure 1 Only 3 clients are drawn in the figure, there can be more clients in practice) through the SAN network that transmits video files ( figure 1 middle thick solid line and double line) are connected to the metadata server and the disk array, and the multiple clients described above transmit metadata through the LAN network ( figure 1 Indicated by a thin solid line) to connect to the metadata server, such as figure 1 shown.
[0042] The client can be an ordinary PC workstation or a server, which has the ability to connect to a SAN network and can process large files, such as high-definition video files. The SAN network described in this embodiment is an optical fiber network made up of optical fiber switches and optical cables. It is a broadband network with a bandwidth exc...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to methods for rapid aggregation and reading of multiple files of a shared-file system. The methods comprise the following steps of rapidly aggregating multiple files, opening the files, obtaining metadata, reading or modifying the files, and closing the files. The step of rapidly aggregating multiple files includes: detecting validity of a sub aggregation file; detecting whether the aggregation is the additional one; establishing an aggregation file; establishing a mapping relation between the aggregation file and the sub aggregation file; and carrying out processing on the sub aggregation file. The step of obtaining metadata includes: opening up a storage space unit; carrying out selective processing; detecting whether a file of obtaining metadata is an aggregation one; obtaining a mapping relation between the aggregation file and a sub aggregation file; and obtaining metadata. According to the invention, on the basis of the rapid aggregation method, files generated by multiple clients are aggregated rapidly into a large file in short time or the multiple files are added and aggregated to the end of an existing large file rapidly. During the aggregation process, no file copy tool is needed. A file obtained by rapid aggregation and a file generated by using a physical aggregation method are completely identical.
Description
technical field [0001] The invention relates to a method for fast aggregation and reading of multiple files in a shared file system, which is a method for fast aggregation of multiple files generated by cluster computing such as distributed transcoding and distributed packaging in the field of video and audio in a shared file system. It is a method for fast aggregation of multiple files under a shared file system suitable for broadcasting and television applications. Background technique [0002] At present, the common shared file system is composed of metadata server (MDS), shared storage and multi-client through LAN network and SAN network. Both metadata server and client can directly access shared storage through FC or ISCSI protocol, respectively FC -SAN and IP-SAN architecture. Due to the optical fiber transmission, the SAN network has large bandwidth, large capacity, and fast speed. It is often used to transmit files with a huge amount of data, such as video files. ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04N21/2343H04N21/4402H04N21/231H04N21/433
CPCH04N21/23113H04N21/234336H04N21/2393H04N21/4335H04N21/440236
Inventor 褚震宇徐荣波王付生
Owner BEIJING DAYANG TECH DEV