MDS log disk falling method and device for distributed storage file system

A distributed storage and file system technology, applied in multi-program devices, resource allocation, special data processing applications, etc., can solve the problems of MDS business delay fluctuation, long log storage time, freeze and other problems, reducing time The effect of prolonging the impact and shortening the processing time

Active Publication Date: 2020-10-09
INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] When the MDS is doing the log segment disk operation, all the business threads of the MDS will be suspended. Only after the log segment disk task is completed, other business threads can continue; if a log file operation involves many log segments processing, and each log segment involves too many log events (in a large directory scenario, the number of directories involved in a single log segment will be less, but there will be more directory items that need to be processed in a single directory. This section only involves one directory, and there are 1024 directory items to be processed in this directory that need to be processed sequentially) will cause the log log to disk take too long this time, which will lead to fluctuations in the corresponding business delay of MDS, resulting in lag

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • MDS log disk falling method and device for distributed storage file system
  • MDS log disk falling method and device for distributed storage file system
  • MDS log disk falling method and device for distributed storage file system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0041] This embodiment provides a method for dumping logs in a distributed storage file system MDS. During the operation of the MDS, the busyness of the business changes dynamically. When the business is relatively idle, this method performs the dumping operation on as many log segments as possible that exceed the limit. , reduce this operation when the business is busy, so as to realize the dynamic management of log storage and reduce the delay impact of MDS log segment disk on normal business; Log events (each log event corresponds to a directory entry) are processed concurrently, and after all the log events in the directory are processed, a disk operation is performed, so as to achieve the purpose of shortening the processing time of a single directory, thereby reducing the MDS log. Latency impact of segment disks on normal services.

[0042] Such as figure 1 As shown, the method includes the following steps:

[0043] SS1: Obtain the business idle state of the node where...

Embodiment 2

[0066] This embodiment provides a distributed storage file system MDS log storage device, which can realize the method described in the first embodiment.

[0067] Such as Figure 4 As shown, the device includes the following functional modules.

[0068] Node business idle state acquisition module 1: obtain the business idle state of the node where the MDS is located;

[0069] Determination module 2 for the number of log segments placed on the disk: determine the number of log segments that need to be placed on the disk according to the idle state of the node business;

[0070] Packaging and placing processing module 3: During the carrying out of each log segment, the packing and placing process is performed on multiple pending log events in the log segment through multi-threaded parallel processing.

[0071] Wherein, the service idle state of the node refers to the CPU occupation ratio and the memory occupation ratio of the node.

[0072] The log segment quantity determinat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an MDS log disk falling method and device for a distributed storage file system, and the method comprises the steps: obtaining the business idle state of a node where an MDS islocated, and determining the number of log segments needing disk falling according to the business idle state of the node; and in the process of executing disk falling on each log segment, packagingand disk falling processing is carried out on the multiple to-be-processed log events in the log segment in a multi-thread parallel processing mode. According to the method, the number of the log segments needing disk falling is determined according to the service idle state of the node, disk falling operation is reduced when the service is busy, log disk falling is dynamically controlled according to the busy degree of the node service, and the time delay influence of log disk falling on normal service is reduced. According to the method, the multiple to-be-processed log events in the log segment are packaged and subjected to disk falling processing in a multi-thread parallel processing mode, the purpose of shortening the processing time of a single directory is achieved, and then the time delay influence of MDS log paragraph disks on normal services is further reduced.

Description

technical field [0001] The invention relates to the field of distributed storage file systems, in particular to a method and device for dumping MDS logs into disks of a distributed storage file system. Background technique [0002] In the current distributed file storage system, all operations of MDS (Metadata Server, the metadata service in the distributed file storage system) will be recorded in the log segment, and each log segment can record 1024 log events, and each metadata The operation will be recorded as a log event, and the directory entry object of the corresponding metadata operation will be marked as pending, waiting for the log event to be updated to the disk. [0003] MDS will regularly start corresponding threads to check the number of log segments of the current MDS. If the number of log segments of the current MDS exceeds the limit, MDS will write the log segments exceeding the limit to the disk one by one in order, and In the writing process, it is carrie...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50G06F16/182
CPCG06F9/5016G06F16/182
Inventor 李伟李雪生
Owner INSPUR SUZHOU INTELLIGENT TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap