A storage volume management system
By introducing a storage volume management system into a distributed file system, using the metadata management of the node file management system and the interface layer, the problem of slow acquisition of file directory structure and complex management of the distributed file system is solved, and efficient file system management is achieved and data recovery.
Patent Information
- Application Number
- CN202010145511.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-05
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-03-05
AI Technical Summary
When managing massive data, existing distributed file systems have problems such as slow file directory structure and complex file management, which are difficult to meet the needs of large data-intensive I/O concurrent access.
Provide a storage volume management system, including multiple nodes, databases and interface layers, each node contains multiple disks and file systems, monitors and collects file metadata through the node file management system, and manages databases based on user requirements through the interface layer. The metadata in it realizes virtualization and intelligent management of the file system.
Through disk virtualization and metadata management, the problem of slow acquisition of file directory structure and complex management of distributed file systems is solved, file system management is simplified, file retrieval and statistical efficiency is improved, and cluster health scores and data recovery capabilities are improved.
Smart Images

Figure CN111488321B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of storage management, and in particular to a management system for storage volumes. Background Art
[0002] The development of information technology has brought about an explosive growth in data. PB-scale unstructured data is becoming more and more common. How to effectively manage this data and further explore its value has become an issue that IT managers must pay attention to. At the same time, the 4V characteristics of big data have also put forward higher requirements for storage systems in terms of large capacity, high performance, easy scalability, and ease of use. Traditional SAN and NAS storage architectures have been unable to meet the intensive I / O concurrent access management requirements for massive data.
[0003] like Figure 1 As shown in the figure, in the era of big data, Internet and cloud computing, distributed file systems have developed rapidly. Distributed File System (DFS) or Network File System (NFS) is a file system that allows files to be shared on multiple hosts through the network, allowing multiple users on multiple machines to share files and storage space. In such a file system, the client does not directly access the underlying data storage block, but communicates with the server through the network using a specific communication protocol. Through the design of the communication protocol, both the client and the server can limit access to the file system based on the access control list or authorization. It is a combination of various storage servers, which are integrated with Ethernet or Infiniband technology and remote direct memory access RDMA to form a large parallel file system network. Distributed systems run in remote data centers and cannot fully control the infrastructure, which makes management and monitoring more difficult than single-machine deployment. Applications must be in a public running information state so that administrators can use this information to manage and monitor the system. In addition, existing distributed file systems have problems such as slow access to file directory structure, complex file management, and lack of intelligence. Summary of the invention
[0004] In order to solve the above-mentioned deficiencies in the prior art, the present invention provides a management system for storage volumes.
[0005] The present invention provides a storage volume management system, comprising:
[0006] Multiple nodes, databases, and interface layers;
[0007] Each node includes multiple disks, at least one file system and a node file management system; wherein the file system is virtualized from multiple disks in the node, the file system includes storage volumes, the storage volumes are used to store user-created files, and the node file management system is used to monitor and collect metadata of files in the file system on the node;
[0008] The database is used to store metadata of files in each file system;
[0009] The interface layer is used to manage the metadata of the database based on the services required by the user.
[0010] Preferably, the node file management and control system includes: a monitoring module and a collection module;
[0011] The monitoring module is used to monitor whether the file system in each node is operated by connecting to the monitoring file system component in the operating system kernel on the node, and insert the operation into the events event list in the form of an event, and generate an event notification of the file system based on the events event list, and generate an event alarm task of the file system based on the event notification. The monitoring module is also used to send the event alarm task to the acquisition module;
[0012] The collection module is used to collect metadata of the corresponding file system based on the event alarm task and store the collection results in a database;
[0013] The file system being operated includes creation, deletion, reading, writing and moving of files in the file system; and the event includes creation, deletion, reading, writing and / or moving of files.
[0014] Preferably, the acquisition module includes:
[0015] Message middleware, task execution unit and task execution result unit;
[0016] The message middleware is used to provide a task scheduling queue for storing event alarm tasks;
[0017] The task execution unit is used to collect metadata of the file system based on the event alarm task in the task scheduling queue and store the collection results in the database;
[0018] The task execution result unit is used to store the execution result of the task execution unit.
[0019] Preferably, the task execution unit includes:
[0020] A task scheduling subunit, a plurality of metadata collection subunits, and a data storage subunit;
[0021] The task scheduling subunit is used to obtain the event alarm task from the message middleware, and schedule the metadata collection subunit to execute the event alarm task;
[0022] The metadata collection subunit is used to collect metadata of the file system based on the task assigned by the task scheduling subunit, and send the metadata of the file system to the data storage subunit;
[0023] The data storage subunit is used to store the collection results of the metadata collection subunit in a database;
[0024] The metadata of the file system includes the name, type, path, size, volume, node, file system, creation time and modification time of the file.
[0025] Preferably, the task scheduling subunit is further used to determine the metadata collection subunit to be scheduled by judging the busyness and the number of task execution of each metadata collection subunit before scheduling the metadata collection subunit.
[0026] Preferably, the interface layer includes: a statistics unit, a retrieval unit and a data healing unit;
[0027] The statistical unit is used to count the file type, file quantity and file distribution in the database;
[0028] The retrieval unit is used to search the database according to the name, type, path, size, volume, node, file system, creation time and / or modification time of the file;
[0029] The data healing unit is used to determine the original and the copy of the damaged file according to the file name, type, path, and volume when a damaged file on the node needs to be healed; secondly, select the recovery basis according to the creation time, modification time, and size of the damaged file; then select the strategy according to the creation time, modification time, and size of the damaged file; and finally restore the damaged file through the rsync tool.
[0030] Preferably, the statistical unit includes:
[0031] The file type statistics subunit is used to call the aggregate method to perform clustering based on the type field in the database and count the number of files of all types in the metadata;
[0032] The file quantity counting subunit is used to retrieve all file metadata in the database and perform a count algorithm to count the number of all files;
[0033] The file distribution statistics subunit is used to call the aggregate method to perform clustering in the database according to the node field, volume field, and file field of the file, and count the number of all files under the node, the number of files in the volume, the number of files in the file system, and the location of the original and duplicate copies in the distributed file system.
[0034] Preferably, the interface layer further includes:
[0035] The scoring unit is used to score and obtain the scoring result according to the cluster file inconsistency ratio and file split-brain ratio when a node crashes or a network anomaly occurs.
[0036] Preferably, the interface layer further includes:
[0037] The data display unit is used to display statistical results, search results and cure results.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] The management system provided by the present invention comprises a plurality of nodes, a database and an interface layer; each node comprises a plurality of disks, at least one file system and a node file management and control system; wherein the file system is obtained by virtualizing a plurality of disks in the node, the file system comprises a storage volume, the storage volume is used to store files constructed by users, and the node file management and control system is used to monitor and collect metadata of files in the file system on the node; the database is used to store metadata of files in each file system; the interface layer is used to manage metadata of the database based on services required by users, and the present invention virtualizes the disks of a plurality of nodes constituting a distributed file system to obtain a plurality of file systems, and by managing metadata of files in the file system, the problem of slow acquisition of file directory structure by the distributed file system is solved, and the problem of file management in the file system is simplified. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a schematic diagram of the structure of a typical existing distributed file management system;
[0041] Figure 2 The figure is a schematic diagram of a storage volume management system according to the present invention. DETAILED DESCRIPTION
[0042] In order to better understand the present invention, the content of the present invention is further described below in conjunction with the accompanying drawings and examples.
[0043] With the development of distributed storage, the standardization process of the storage industry is also constantly advancing. Intelligent volume management technology gives priority to the use of industry standard interfaces for storage access. At the platform level, by abstracting heterogeneous storage resources and encapsulating traditional storage device-level operations into storage resource-oriented operations, the operation of heterogeneous storage infrastructure is simplified, and the entire storage life cycle process such as creation, modification, and recycling can be automatically executed. Each member of this whole will have some status, such as its own responsible module, its own load situation, and the control of certain data. And these data related to other processes become very important when it comes to file monitoring, data statistics, data recovery, etc. Figure 2 As shown, according to the technical architecture of the above-mentioned intelligent volume management technology, the present invention provides a management system for storage volumes, including: multiple nodes, a database and an interface layer;
[0044] Each node includes multiple disks, at least one file system and a node file management system; wherein the file system is virtualized from multiple disks in the node, the file system includes storage volumes, the storage volumes are used to store user-created files, and the node file management system is used to monitor and collect metadata of files in the file system on the node;
[0045] The database is used to store metadata of files in each file system;
[0046] The interface layer is used to manage the metadata of the database based on the services required by the user.
[0047] The management system provided by the present invention is based on virtualizing multiple disks of each node into different file systems, collecting and monitoring all file systems through existing collection technology, and storing metadata of file system addition, deletion, modification and query events through mogodb. Finally, by using different modules of Python algorithm to conduct comprehensive analysis on metadata, it realizes file metadata monitoring of distributed file system volumes, rapid management of file metadata, rapid retrieval of massive files, file type statistics, file quantity statistics, file distribution statistics, cluster health score, recovery after file damage and other intelligent management functions, and solves the problems of slow acquisition of file directory structure and complex file management of distributed file system. Wherein MongoDB is a database based on distributed file storage.
[0048] The virtualization of the disk in the present invention includes building the disk into RAID and then creating a VolumeGroup. This process is also called the resource pooling process of the disk; creating an LVM in the VolumeGroup and then creating a file system in the LVM. The storage volume is built on the created file system. Then the user can store files on the storage volume. As long as there is a file operation in the storage volume, the monitoring module will monitor it and collect the metadata of the file.
[0049] In a distributed file system, each node has several file systems, and each node has a monitoring module and a collection module. The monitoring module monitors the changes of the file system in each node by connecting to the operating system kernel monitoring file system component on the node. This embodiment takes the Linux operating system as an example. When operations such as file creation, deletion, reading, writing and moving occur, the kernel inserts the event into the events event list, then wakes up the program waiting in the collection module, and sends the relevant event alarm task to the collection module.
[0050] The collection module is used to collect metadata of the file system based on the event alarm task sent by the monitoring module and store the collection results in a database.
[0051] The metadata of multiple disks in a node are integrated into a resource pool, and the resource pool virtualizes the metadata into several file systems according to classification rules.
[0052] The acquisition module in the present invention uses a producer-consumer model;
[0053] The acquisition module includes: message middleware, task execution unit and task execution result unit.
[0054] Message middleware: The message middleman is a task scheduling queue, an independent service, and a producer-consumer model. The monitoring module, as a producer, puts the event alarm task of the file system into the task scheduling queue, and the consumer takes the task out of the task scheduling queue for execution.
[0055] Task execution unit: a program for executing tasks, which may have multiple concurrent tasks and is used to take out the event alarm task from the task scheduling queue, collect metadata of the file system based on the event alarm task and store the collected results in the database;
[0056] Task execution result unit: Since the execution of tasks is separated from the main program, if the main program wants to obtain the results of task execution, it must be stored through the middleware. It is used to store the results of event alarm task execution in order to handle failed tasks.
[0057] The task execution unit in the present invention comprises: a task scheduling subunit, a plurality of metadata collection subunits and a data storage subunit; and mainly realizes task scheduling, metadata collection and data storage.
[0058] The task scheduling subunit is used to obtain the event alarm task from the message middleware, select the metadata collection subunit by judging the busyness of each metadata collection subunit and the number of times the task is executed, and schedule the event alarm task to the selected metadata collection subunit;
[0059] The metadata collection subunit is used to collect metadata of the file system based on the task assigned by the task scheduling subunit, and send the metadata of the file system to the data storage subunit;
[0060] The data storage subunit is used to store the metadata of the file system corresponding to the event alarm task in a database;
[0061] The metadata of the file system includes: the name, type, path, size, volume, node, file system, creation time and modification time of the file.
[0062] The specific process of the monitoring module and the acquisition module working together in this embodiment includes:
[0063] In the first step, the monitoring module acts as a producer to monitor the event notifications of the file system in real time and put the event notifications into the message middleware.
[0064] In the second step, the task execution unit acts as a consumer. After receiving the task, the task scheduling subunit schedules the metadata collection subunit to execute the task in the queue and collect the metadata of the file system according to the task, and passes the collected metadata to the data storage subunit.
[0065] In the third step, the task passes through the data storage subunit and finally stores the data in a distributed database.
[0066] The fourth step is to place the task execution result in the task execution result unit after the task execution unit is completed. If the task execution is successful, the task execution is completed. If the task execution fails, the file is specifically analyzed.
[0067] The interface layer in the present invention includes: a statistical unit, which is used to perform a clustering algorithm based on the name, type, path, size, volume, node, file system, creation time, modification time and other contents of the files stored in the database to count the file type, file quantity and file distribution.
[0068] Furthermore, the statistical unit specifically includes:
[0069] File type statistics subunit: used to call the aggregate method to perform clustering based on the type field in the database, so as to count the number of files of all types in the metadata.
[0070] File quantity statistics subunit: used to retrieve all file metadata in the database and perform count algorithm to get the number of all files.
[0071] File distribution statistics subunit: It is used to call the aggregate method to perform clustering in the database according to the node field, volume field, and file system field where the file is located. It can count the number of all files under the node, the number of files in the volume, the number of files in the file system, and the location of the original and duplicate copies in the distributed file system.
[0072] The interface layer of the present invention also includes: a retrieval unit, which is used to retrieve the massive files stored in the database according to the fields such as name, type, path, size, volume, node, file system, creation time, modification time, etc., so as to replace the direct search in the file system, thereby greatly improving the retrieval efficiency. After the file is retrieved, the node, volume and path of the file can be quickly known according to the metadata of the file, and then the files can be deleted and downloaded in batches.
[0073] The interface layer of the present invention also includes: a scoring unit, which is used to score according to the cluster file inconsistency ratio and file brain split ratio when a node crashes or a network anomaly occurs, that is, when a cluster file is inconsistent or a file brain split occurs, and the distributed file system tool is used to count the number of file system inconsistencies and the number of brain split files, and the scoring is implemented according to the number ratio, file age, and importance level. After scoring, the user can understand the health status of the cluster according to the score and decide whether to enable healing.
[0074] The interface layer in the present invention also includes: a healing unit, because the database stores the file name, type, path, size, volume, node, file system, creation time, modification time and other contents, when the node has a damaged file and needs to be cured, firstly obtain the location of the original and duplicate of the damaged file in the file system according to the file name, type, path and volume, and then select the recovery basis according to the creation time, modification time and size of the damaged file; then select the strategy according to the creation time, modification time and size of the damaged file, and finally restore the damaged file through the rsync tool, that is, through the rsync tool, copy the selected correct file from the correct file system to the damaged file system, so as to achieve rapid recovery of the business. Because the distributed file system is built on each file system, if it is a replica volume, the same file will be written in multiple file systems.
[0075] The interface layer in this implementation also provides a data display unit for obtaining and displaying request results from the database based on client requests, and also displays scoring results, cure results, and retrieval results, etc.
[0076] The intelligent volume management method provided by the present invention uses the acquisition technology, file system synchronization technology, metadata storage technology, Python and other underlying technical algorithms to realize the storage volume. Through real-time monitoring of the file system, it can realize file metadata monitoring, file metadata rapid management, mass file rapid retrieval, file type statistics, file quantity statistics, file distribution statistics, cluster health score, file damage recovery, etc., thus having the following functional advantages:
[0077] (1) High performance, easy to deploy and use;
[0078] (2) Simple, flexible and reliable;
[0079] (3) Applicable to cloud storage, enterprise-level distributed storage and other environments;
[0080] (4) Improve the data management performance of distributed file systems;
[0081] (5) Monitor the operating status of the file system in real time to improve operation and maintenance capabilities;
[0082] (6) Rapid synchronization and healing improves data security;
[0083] (7) Improved the availability of business data;
[0084] (8) Reduces the complexity of storage planning and management.
[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0089] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.
Claims
1. A storage volume management system, characterized in that: include: Multiple nodes, databases, and interface layers; Each node includes multiple disks, at least one file system and a node file management system; The file system is obtained by virtualizing multiple disks in the node, the file system includes storage volumes, the storage volumes are used to store files constructed by users, and the node file management and control system is used to monitor and collect metadata of files in the file system on the node; The database is used to store metadata of files in each file system; The interface layer is used to manage the metadata of the database based on the services required by the user; The node file management and control system includes: a monitoring module and a collection module; The monitoring module is used to monitor whether the file system in each node is operated by connecting to the monitoring file system component in the operating system kernel on the node, and insert the operation into the events event list in the form of an event, and generate an event notification of the file system based on the events event list, and generate an event alarm task of the file system based on the event notification. The monitoring module is also used to send the event alarm task to the acquisition module; The collection module is used to collect metadata of the corresponding file system based on the event alarm task and store the collection results in a database; The file system being operated includes the creation, deletion, reading, writing and moving of files in the file system; the event includes the creation, deletion, reading, writing and / or moving of files; The interface layer includes: a statistics unit, a retrieval unit and a data healing unit; The statistical unit is used to count the file type, file quantity and file distribution in the database; The retrieval unit is used to search the database according to the name, type, path, size, volume, node, file system, creation time and / or modification time of the file; The data healing unit is used for, when a file on a node is damaged and needs to be healed, first determining the original and duplicate of the damaged file according to the file name, type, path, and volume, then selecting a recovery basis according to the creation time, modification time, and size of the damaged file, then selecting a strategy according to the creation time, modification time, and size of the damaged file, and finally restoring the damaged file through the rsync tool; The statistical unit comprises: The file type statistics subunit is used to call the aggregate method to perform clustering based on the type field in the database and count the number of files of all types in the metadata; The file quantity counting subunit is used to retrieve all file metadata in the database and perform a count algorithm to count the number of all files; The file distribution statistics subunit is used to call the aggregate method to perform clustering in the database according to the node field, volume field, and file field of the file, and to count the number of all files under the node, the number of files in the volume, the number of files in the file system, and the location of the original and duplicate copies in the distributed file system; The interface layer also includes: The scoring unit is used to score and obtain the scoring result according to the cluster file inconsistency ratio and file split-brain ratio when a node crashes or a network anomaly occurs.
2. The system according to claim 1, characterized in that The acquisition module comprises: Message middleware, task execution unit and task execution result unit; The message middleware is used to provide a task scheduling queue for storing event alarm tasks; The task execution unit is used to collect metadata of the file system based on the event alarm task in the task scheduling queue and store the collection results in the database; The task execution result unit is used to store the execution result of the task execution unit.
3. The system according to claim 2, characterized in that The task execution unit comprises: A task scheduling subunit, a plurality of metadata collection subunits, and a data storage subunit; The task scheduling subunit is used to obtain the event alarm task from the message middleware, and schedule the metadata collection subunit to execute the event alarm task; The metadata collection subunit is used to collect metadata of the file system based on the task assigned by the task scheduling subunit, and send the metadata of the file system to the data storage subunit; The data storage subunit is used to store the collection results of the metadata collection subunit in a database; The metadata of the file system includes the name, type, path, size, volume, node, file system, creation time and modification time of the file.
4. The system according to claim 3, characterized in that The task scheduling subunit is further used to determine the metadata collection subunit to be scheduled by judging the busyness and the number of task execution of each metadata collection subunit before scheduling the metadata collection subunit.
5. The system according to claim 1, wherein: The interface layer also includes: The data display unit is used to display statistical results, search results and cure results.
Citation Information
Patent Citations
File management method, equipment and memory system
CN102004769A
Cloud storage system and implementation method thereof
CN102307221A
Cluster-based visible media storage system
CN102624911A
Storage state monitoring mechanism for virtual machine
CN104268061A
Virtual machine management method and system
CN105718304A
