An Organizational Method for Auxiliary Records
A technology for organizing and appending records, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve problems such as memory allocation failure, affecting user experience, recycling failure, etc., to improve processing speed, control memory consumption, The effect of improving satisfaction
Active Publication Date: 2018-05-29
DAWNING CLOUD COMPUTING TECH CO LTD
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
When recycling, it is necessary to load all the records in the file into the memory and establish a memory hash structure to facilitate the pairing of records and their cancellation records. When the system memory resources themselves are relatively tight, the recovery may fail due to memory allocation failure.
[0008] 2) If the logical log to data log conversion process of the log system is on the critical path, this implementation method is unacceptable, which will cause jagged performance in the distributed file system, and even fail to respond to users within 1 minute, seriously affecting user experience
When the record file is very large, even with the read-ahead mechanism, the time consumption of this process is very large compared to the service capability of the file system
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0033] The present invention will be described in further detail below in conjunction with the accompanying drawings.
[0034] Such as figure 1 As shown, the disk structure of the organization method is that the metadata file of each type of record contains the unique identifier fid of each type of record and the number of invalid records invalidnr. Recycling is performed when the number of invalid records reaches a certain percentage, which is 30% by default and can also be adjusted freely. There are multiple recording files on each type of recording disk.
[0035] Such as figure 2 As shown, the memory structure of the organization method includes the read-write lock rwlock used to protect the structure content, this class records the current minimum subid number min_subid (easy to obtain the starting file for recycling and sequential reading), and this class records the current maximum subid number max_subid (used for subid allocation when the file is full and creating a...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a method for organizing auxiliary records, which includes the process of converting logic logs into data logs, adding records in memory and recycling files. The method provided by the invention splits the file into small files, so that the memory consumption during recycling is more controllable. At the same time, since the recycling process is run by an independent background thread, the cost of memory allocation failure is also greatly reduced. The time-consuming stage that seriously affects system performance is separated from the critical path, which improves the processing speed of the critical path and greatly improves the satisfaction of the user experience.
Description
technical field [0001] The invention relates to a method for operating distributed files, in particular to a method for organizing auxiliary records. Background technique [0002] In a parallel storage system, it is often necessary to record some auxiliary records or indexes to achieve specific purposes, such as recording which disks save the data of the file during the creation process, so that we can know exactly which files are saved on that disk. When a disk failure occurs, these files can be quickly repaired; when the server receives a deletion request from the client, it only deletes the dentry item, and does not recycle the disk data, but records the deletion and then responds to the client. In the background, an independent recycling thread completes the real Delete, in order to achieve the purpose of responding to users quickly. [0003] In the log distributed file system, these records also need to be logged to achieve the same protection level as metadata, otherw...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 马照云杨浩马振杰
Owner DAWNING CLOUD COMPUTING TECH CO LTD