Metadata repair method and electronic device

Abstract

The application discloses a metadata repairing method and an electronic device, and particularly relates to the field of data processing. A target storage volume with closed write cache is added in a storage volume of a storage system, so that first metadata is stored through the target storage volume, thereby ensuring that the first metadata will not be lost due to an abnormality of the storage system. Subsequently, in the case that the storage system satisfies a repairing trigger condition, the first metadata is directly obtained from the target storage volume with closed write cache, so that the loss or damage of the first metadata due to the abnormality of the storage system is reduced, the storage system can be normally started and the target metadata stored in a target database can be loaded or repaired, the repairing capability of the target metadata is improved as much as possible, and thus the storage system can normally access the target metadata stored in the target database.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a metadata repair method and an electronic device. BACKGROUND

[0002] In a storage system, in order to improve the throughput performance of data, the underlying object storage device (OSD) is usually replaced with a storage volume medium that opens a read-write medium. However, when the storage system fails, the metadata that has not been stored in the storage device in the cache device may be lost, thereby causing the storage system to access errors.

[0003] Therefore, how to repair the metadata to ensure that the storage system can be normally accessed is a problem to be solved. SUMMARY

[0004] The present application provides a metadata repair method and an electronic device to realize the repair of metadata, thereby ensuring that the storage system can be normally accessed.

[0005] The present application provides a metadata repair method, comprising:

[0006] adding a target storage volume that closes the write cache in the storage volume of the storage system, and storing first metadata in the target storage volume; wherein the first metadata is used to provide a management interface for the target metadata;

[0007] in response to the storage system satisfying a repair trigger condition, obtaining the first metadata from the target storage volume of the storage system;

[0008] performing a repair operation on the target metadata in the target database through the first metadata; wherein the repair operation includes at least one of a log playback operation, a file scanning operation, a metadata repair operation, and a directory tree reconstruction operation.

[0009] The present application also provides a metadata repair device, comprising:

[0010] a metadata storage module for adding a target storage volume that closes the write cache in the storage volume of the storage system, and storing first metadata in the target storage volume; wherein the first metadata is used to provide a management interface for the target metadata;

[0011] a metadata acquisition module for obtaining the first metadata from the target storage volume of the storage system in response to the storage system satisfying a repair trigger condition;

[0012] a metadata repair module for performing a repair operation on the target metadata in the target database through the first metadata; wherein the repair operation includes at least one of a log playback operation, a file scanning operation, a metadata repair operation, and a directory tree reconstruction operation.

[0013] The application also provides an electronic device, comprising: at least one processor; and,

[0014] a storage device in communication connection with the at least one processor; wherein,

[0015] The storage device stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of any of the above methods.

[0016] The application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above methods.

[0017] The application also provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the above methods.

[0018] The application provides a metadata repair method and an electronic device. A target storage volume with write cache closed is added in a storage volume of a storage system, so that a first metadata is stored through the target storage volume, thereby ensuring that the first metadata will not be lost due to an exception of the storage system. Then, in a case where the storage system satisfies a repair triggering condition, the first metadata is directly obtained from the target storage volume with write cache closed, so that the loss or damage of the first metadata due to the exception of the storage system is reduced, the storage system can be normally started and the target metadata stored in a target database is loaded or repaired, the repair capability of the target metadata is improved as much as possible, and the normal access of the storage system to the target metadata stored in the target database is ensured.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 A structural schematic diagram of a storage system provided by the embodiments of the present application;

[0022] Figure 2A flowchart of a metadata repair method provided by an embodiment of the present application is shown in FIG. 1.

[0023] Figure 3 A flowchart of a metadata repair method provided by an embodiment of the present application is shown in FIG. 1.

[0024] Figure 4 A flowchart of a metadata repair method provided by an embodiment of the present application is shown in FIG. 1.

[0025] Figure 5 A flowchart of a metadata repair method provided by an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0027] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0028] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0029] In some cases, in order to better manage and store data, a storage system is usually deployed in an electronic device so that the electronic device can access data efficiently, safely and orderly. The storage system can be a distributed storage system (or distributed file system), such as a Ceph storage system. The Ceph storage system can provide block storage service, object storage service and storage system storage service at the same time.

[0030] Specifically, the Ceph storage system can include a CephFS and a metadata service (MDS). The CephFS is a storage system interface of the Ceph storage system, which can provide a file storage service. The metadata service is a core component of the Ceph storage system for managing storage system metadata. The metadata service is responsible for managing metadata (such as name, permission, size, inode number, etc.) of files and directories, and caching the metadata in memory to speed up access. At the same time, the metadata service is also responsible for managing the namespace of the Ceph storage system, and coordinating access to the storage data.

[0031] It should be noted that the metadata managed by the metadata service is ultimately stored in the underlying data pool. In this way, even if the electronic device indirectly accesses the metadata through the metadata service, the case that the electronic device directly operates the underlying storage pool and reduces the access efficiency can be reduced.

[0032] In some embodiments, in order to improve the throughput performance of the data, the Ceph storage system can replace the underlying object storage device (OSD) with a storage volume medium that opens the read-write medium. When the Ceph storage system appears abnormal scenarios such as disk failure, software crash, all nodes of the system are powered off at the same time, etc., it may cause the metadata in the cache device that has not been stored to the storage device to be damaged or lost, and further cause the Ceph storage system to access errors.

[0033] In an implementation manner, in order to repair the metadata in the Ceph storage system, the metadata repair can be performed on each node in the Ceph storage system through a command line tool. The metadata repair can include a log replay phase, a clipping and file scanning phase, and a directory tree scanning and reconstruction phase. However, each phase of the metadata repair needs to rely on all nodes in the Ceph storage system to complete the task of the previous phase. This requires that after all nodes complete the task of the same phase, it is considered that the storage system enters the next phase, that is, there is a lack of effective repair state synchronization mechanism between nodes, and it relies on manual operation of the user. In this way, not only a large amount of repair time is wasted, the metadata repair efficiency is reduced, but also human resources are wasted, and the user experience is affected.

[0034] In addition, when the Ceph storage system appears the above abnormal scenarios, in response to the restart operation of the CephFS, the Ceph storage system can initialize a target database, and read configuration information from the target database. The configuration information can include file and directory metadata kv, and space allocation information. The space allocation information can include metadata stored in the form of key-value pairs.

[0035] Exemplarily, the target database can be RocksDB, which is equivalent to an embedded key-value storage engine in CephFS for efficiently managing metadata. The RocksDB can ensure the durability and crash recovery capability of metadata operations through a WAL (write-ahead log). Moreover, the RocksDB, as a core component of the BlueStore storage engine, stores metadata (such as inodes and directory structures) in the BlueFS storage system in the form of key-value pairs.

[0036] The BlueFS storage system is a lightweight log-structured file system specially designed for RocksDB in the Ceph storage system, which is used to efficiently manage metadata files (such as SST files and LOG files) of the target database and ensure metadata consistency through a log transaction mechanism. It divides the storage space into three layers: WAL (ultra-high speed), DB (high speed), and Slow (slow), and supports multiple device types to optimize performance. BlueFS simplifies the directory structure, and all metadata is resident in memory, which is restored through log replay at restart, significantly improving the efficiency of metadata operations of the BlueStore storage engine.

[0037] However, if the key-value pair information in the target database is damaged, the target database cannot be initialized. This requires repairing the target database, but the repair process requires disk space, and the loss of key-value pair information will cause space allocation failure, that is, the disk space cannot be successfully applied, resulting in a circular dependency problem of target database repair and space allocation failure during metadata repair, which causes the Ceph storage system to fail to repair the target database in time.

[0038] Therefore, in order to realize the repair of metadata and ensure that the storage system can normally access the metadata in the target database, the embodiments of the present application provide a metadata repair method. In the method, a target storage volume with closed write cache is added in the storage volume of the storage system, and first metadata is stored in the target storage volume. The first metadata is used to provide a management interface for the target metadata. Then, in response to the storage system satisfying a repair trigger condition, the first metadata is obtained from the target storage volume of the storage system. Then, through the first metadata, a repair operation is performed on the target metadata in the target database. The repair operation includes at least one of a log replay operation, a file scanning operation, a metadata repair operation, and a directory tree reconstruction operation.

[0039] In the embodiments of the present application, a target storage volume with write cache closed is added in the storage volume of the storage system, so as to store the first metadata through the target storage volume, thereby ensuring that the first metadata will not be lost due to an abnormality of the storage system. Subsequently, in the case that the storage system satisfies the repair triggering condition, the first metadata is directly obtained from the target storage volume with write cache closed, so as to reduce the occurrence of the loss or damage of the first metadata due to the abnormality of the storage system, ensure that the storage system can be normally started and the target metadata stored in the target database can be loaded or repaired, and improve the repair capability of the target metadata as much as possible, thereby ensuring that the storage system can normally access the target metadata stored in the target database.

[0040] In some examples, the electronic device in the embodiments of the present application can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, and the like, which includes a storage device, and the specific form of the electronic device is not specially limited in the embodiments of the present application.

[0041] Figure 1 FIG. 1 is a structural schematic diagram of a Ceph storage system in the embodiments of the present application.

[0042] As shown in FIG. 1, the Ceph storage system can include two nodes (node 1 and node 2). Figure 1 It should be noted that the Ceph storage system can include multiple nodes, and the two nodes in the embodiments of the present application are only exemplary and the specific number of nodes is not limited.

[0043] Each node in the Ceph storage system can include a target database (RocksDB), a sub-storage system (BlueFS storage system), a space allocator, an underlying storage volume, and a cluster trivial database (CTDB), etc. The RocksDB can include files, directory metadata kv, and space allocation information. The space allocation information can include information stored in the form of key-value pairs. The BlueFS storage system can include first metadata (or BlueFS metadata) and first data (or BlueFS data). The BlueFS metadata can record the space address of the reserved space. The space allocator is used to apply for storage space (or reserved space) for temporary data generated in the metadata repair process. The underlying storage volume can include a storage volume with write cache closed and a storage volume with write cache opened. The CTDB is a lightweight cluster database, which mainly provides distributed TDB database support for applications, and realizes cross-node data synchronization and high availability. The CTDB can be used to create a state database (node_db.tdb file), which can include key-value pair information (key-value) of each node in the Ceph storage system. The key is the node name or node ID, and the value is the node repair result. The node repair result includes the repair status of each repair stage in the node.

[0044] In some cases, considering that the storage volume with write cache closed can improve the security of data, that is, it can be determined that the data can be written to the disk in time and accurately, and the situation of data loss due to the above abnormal scene can be reduced. Therefore, the BlueFS metadata in the BlueFS storage system can be stored in the storage volume with write cache closed. In this way, as long as the disk is not damaged, the BlueFS metadata will not be lost, thereby reducing the situation of BlueFS metadata loss or damage due to storage system failure, and providing a convenient condition for subsequent better repair of target metadata in RocksDB.

[0045] In other cases, considering that the storage volume with write cache opened can speed up the data write cache speed, and ensure that the data can be quickly written to the disk. Therefore, the BlueFS data in the BlueFS storage system can be stored in the storage volume with write cache opened. In this way, the cache efficiency of the BlueFS data can be improved.

[0046] Specifically, for each node in a Ceph storage system, if a repair phase is completed on one node, the repair status of that phase needs to be synchronized to other nodes. This allows the next repair phase to proceed only after all nodes have completed their repair phase tasks. This timely synchronization of repair status not only significantly reduces repair time and improves metadata repair efficiency, but also reduces labor costs, avoids frequent manual intervention, lowers operational costs, and ultimately improves the user experience.

[0047] The metadata repair method and electronic device of this disclosure will be described below with reference to the accompanying drawings, taking the electronic device as the subject of the metadata repair method.

[0048] Figure 2 This is a flowchart illustrating a metadata repair method provided in an embodiment of this application.

[0049] like Figure 2 As shown, the method includes the following steps:

[0050] S200: Add a target storage volume with write caching disabled in the storage volume of the storage system, and store the first metadata in the target storage volume.

[0051] The first metadata is used to provide a management interface for the target metadata. For example, in a Ceph storage system, the first metadata is BlueFS metadata.

[0052] It's important to note that for better metadata repair, the electronic device requires secondary configuration of the storage system. Specifically, the electronic device can add a target storage volume with write caching disabled in the underlying storage volume. This target storage volume is used to store the primary metadata, namely the BlueFS metadata. This ensures that the BlueFS metadata will not be lost due to storage system anomalies, reducing the likelihood of BlueFS metadata loss or corruption caused by storage system failures, and facilitating better subsequent repair of the target metadata in RocksDB.

[0053] In this application, considering that disabling the write cache in the target storage volume can improve data security—that is, ensure that data can be written to persistent storage devices (such as disks) in a timely and accurate manner, reducing the possibility of data loss due to the aforementioned abnormal scenarios—BlueFS metadata in the BlueFS storage system can be stored in the target storage volume with the write cache disabled. In this way, as long as the persistent storage device remains intact, the BlueFS metadata will not be lost, thereby reducing the possibility of BlueFS metadata loss or corruption due to storage system anomalies and facilitating better subsequent repair of the target metadata in RocksDB.

[0054] Furthermore, the electronic device can modify the writing logic of the target log to obtain a modified log writing logic. The target log (or BlueFS log) records the processing source of BlueFS metadata. The modified log writing logic allows the target log to bypass the caching device and be written directly to a persistent storage device in the storage system. For example, the persistent storage device is a WAL type disk. This persistent storage device can be bound to the aforementioned target storage volume. Thus, by modifying the log writing logic, newly generated logs can bypass the cache and be written directly to the persistent storage device, ensuring that the first metadata update is immediately persisted. This reduces the possibility of BlueFS metadata loss or corruption due to storage system failures, and provides convenient conditions for better subsequent repair of target metadata in RocksDB.

[0055] Optionally, if the space capacity of the first log is less than the preset space capacity, the first logs with space capacities less than the preset space capacity are aggregated until the space capacity of the aggregated first log is greater than or equal to the preset space capacity. The aggregated first log is then written to the storage device as the target log. Here, the first log refers to the initial BlueFS log generated by the BlueFS storage system. The preset space capacity can be pre-set according to actual conditions. For example, the preset space capacity can be 4KB.

[0056] In other words, when the accumulated first log reaches the preset space capacity, the electronic device can trigger the write_super instruction to write the accumulated first log as the target log to the persistent storage device in batches. In this way, aggregated writing of data can be achieved, reducing the occurrence of increased write times due to insufficient log space capacity, reducing the performance loss caused by multiple log writes to the storage device, and improving the utilization of the storage device.

[0057] In some embodiments, the electronic device can deploy database services on each node in the storage system and create a database cluster through the database services of multiple nodes. Specifically, the electronic device can establish a database cluster using the IP addresses of each node. For example, the IP address of node 1 can be 192.168.0.101, and the IP address of node 2 can be 192.168.0.102.

[0058] Next, a state database is created in the database cluster. This state database can be a `node_db.tdb` file, containing key-value pairs for each node in the storage system. The key is the node name or node ID, and the value is the node's repair result. This node repair result includes the repair status of each repair stage within that node. In other words, the state database includes node information and repair results for each node in the storage system, with the repair results including the repair status of each repair stage. For example, the key-value pair could be `node1: running`, or `mds0: rocksdb repair running`, `mds1: rocksdb repair success`, etc.

[0059] It's understandable that the TDB files in the database cluster are persistent, meaning the data in these TDB files can be permanently saved. This ensures that the repair progress of each node can be synchronized in a timely manner, thereby coordinating the repair of all nodes simultaneously. This not only reduces repair time significantly and improves the efficiency of metadata repair, but also reduces labor costs, avoids frequent manual intervention, lowers operational costs, and ultimately improves the user experience.

[0060] In other embodiments, the electronic device can deploy metadata services for each node in the storage system. Then, a reserved space field is added to the superblock structure of the first metadata. The superblock structure of the first metadata is stored in a persistent storage device within the storage system.

[0061] For example, the aforementioned reserved space field can be represented by the following code:

[0062] struct bluefs_superblock_t { ...

[0063] bluefs_extent_t repair_reserved_extents;

[0064] }

[0065] Subsequently, for each node's metadata service, in response to the metadata service's initial startup and its active state, a reserved space is requested, and the space address of the reserved space is saved in the reserved space field. This reserved space can be the space requested by the metadata service from the space allocator. The capacity of this reserved space can be pre-set according to actual needs. For example, the capacity can be 2 GiB. The space address of the reserved space can include a logical address offset and an address length.

[0066] It is understandable that, since data stored in persistent storage devices will not be corrupted or lost, storing the superblock structure of the first metadata in a persistent storage device in the storage system can ensure that the space address stored in the reserved space field will not be corrupted or lost. This can solve the circular dependency problem between target database repair and space allocation failure during the metadata repair process, thereby ensuring that the storage system can repair the metadata in the target database in a timely manner and improve the efficiency of metadata repair.

[0067] S201, in response to the storage system meeting the repair trigger conditions, obtain the first metadata from the target storage volume of the storage system.

[0068] The aforementioned repair triggering conditions may include at least one of the following: all nodes in the storage system experience a power outage, the metadata in the cache device and the storage device is inconsistent, log (mdlog) replay fails, and a user's repair operation is detected.

[0069] Specifically, when the storage system meets the repair trigger conditions, by directly obtaining the first metadata from the target storage volume with write caching disabled, the occurrence of loss or corruption of the first metadata due to storage system anomalies can be reduced. This ensures that the storage system can start normally and load or repair the target metadata stored in the target database, thereby maximizing the repair capability of the target metadata and ensuring that the storage system can access the target metadata stored in the target database normally.

[0070] In some cases, after obtaining the aforementioned first metadata, the electronic device can retrieve the space address of the reserved space from the reserved space field of the superblock structure of the first metadata. Then, a temporary file is created using the reserved space. This temporary file stores temporary data generated during the target metadata repair process. Thus, since data stored in persistent storage devices will not be corrupted or lost, storing the superblock structure of the first metadata in persistent storage devices within the storage system ensures that the space address stored in the reserved space field will not be corrupted or lost. This resolves the circular dependency problem between target database repair and space allocation failure during metadata repair, thereby ensuring that the storage system can repair the metadata in the target database in a timely manner, reducing the occurrence of metadata repair interruptions due to insufficient storage space, and improving the efficiency of metadata repair.

[0071] S202, Repair the target metadata in the target database using the first metadata.

[0072] Specifically, after obtaining the aforementioned first metadata, the electronic device can use this first metadata to perform repair operations on the target metadata in the target database. These repair operations may include at least one of the following: log replay operation, file scanning operation, metadata repair operation, and directory tree reconstruction operation.

[0073] In some embodiments, such as Figure 3 As shown, the process by which an electronic device repairs metadata for a target database can specifically include:

[0074] S2021, Scan the target database to obtain the scan results.

[0075] The target database can be RocksDB. RocksDB can include file and directory metadata (key-value pairs) and space allocation information. This space allocation information can include information stored in key-value pairs.

[0076] Specifically, RocksDB is scanned for each node in the storage system to obtain scan results. These results may include the file size of RocksDB and the last time (mtime) when the metadata of files or directories in RocksDB was modified.

[0077] Optionally, after the scan is completed, the electronic device can save the results that indicate whether the scan was successful to the status database, so that other nodes can know whether the scanning phase of the node has been completed, which provides convenient conditions for timely implementation of the next repair phase.

[0078] S2022, Repair the target database to obtain target metadata.

[0079] Specifically, after obtaining the scan results, the electronic device can repair the target database to obtain the target metadata. This target metadata can be either file metadata or directory metadata from RocksDB.

[0080] In one implementation, the electronic device can divide the target database into multiple repair partitions based on the total number of repair threads. Each repair partition corresponds to one repair thread. Then, for each repair partition in the target database, the partition is repaired to obtain the target metadata within it. Thus, by using multi-threaded concurrent repair, the entire repair process can be shortened, the metadata repair progress can be accelerated, and the efficiency of metadata repair can be improved.

[0081] The electronic device can calculate the hash value of files in the target database to obtain the hash value of the filename. Then, the electronic device can divide the hash value of the filename into partitions based on the total number of repair threads, resulting in a repair partition for each repair thread. For example, if the total number of repair threads is 2, the electronic device can divide the hash value of the filename into two equal parts. Then, the electronic device can control repair thread 1 to repair files with a hash value modulo 0, and control repair thread 2 to repair files with a hash value modulo 1.

[0082] Optionally, after the repair is completed, the electronic device can save the result that indicates whether the repair was successful to the status database, so that other nodes can know whether the repair phase of the node has been completed, which provides convenient conditions for the timely implementation of the next repair phase.

[0083] S2023, Reconstruct the directory tree of the target database.

[0084] Specifically, after obtaining the target metadata from the target database, the electronic device can also reconstruct the directory tree of the target database. In this way, by reconstructing the directory tree, the integrity of the entire RocksDB path can be checked and completed, ensuring that the structural information in the files is complete, providing a foundation for subsequent access to files lost due to structural corruption.

[0085] In some embodiments, the metadata in the target database is traced back to obtain path tracing results. These results characterize whether all path entries in the target database carry linked target structure information. Target structure information may include memory structure information (dentry) and core structure information (inode). Dentry is a memory structure connecting filenames and inodes, used to organize the tree hierarchy of the storage system. Dentry can record filenames, parent directory pointers, and associated inodes. Inodes are the core structure of metadata, storing attributes such as file size, permissions, and timestamps, and pointing to data distribution locations (e.g., object IDs). Each inode is associated with a directory entry (dentry) through a unique identifier (ino), supports hard links and extended attributes (xattr), and is uniformly managed by MDS to maintain global consistency.

[0086] Specifically, the electronic device can obtain the metadata structure information of RocksDB. The metadata structure information (backtrace) is a metadata structure that records file path backtrace information, used to reconstruct file paths when the directory tree is corrupted. This data is stored in the extended attributes (xattr) of the inode and maintained by the MDS to ensure the recoverability of the storage system. The metadata structure information can include the absolute path name of RocksDB from the root. Then, the electronic device can use this absolute path name to check, layer by layer from bottom to top, whether each path entry carries the linked target structure information.

[0087] In some cases, if the path entries in the target database representing the path tracing results all carry information about the linked target structures, it indicates that the file's full path is complete, meaning no information completion is needed. Therefore, the electronic device can directly generate a completion notification. This provides a foundation for subsequent file access.

[0088] In other cases, if the path tracing results indicate that any path entry in the target database lacks information about the linked target structure, it means the file's full path is incomplete, requiring information completion. Therefore, the electronic device can combine the scan results to generate new memory structure information. Afterward, the electronic device can generate a completion notification. In this way, information completion can be performed on files with incomplete paths, ensuring the structural information within the file is intact, providing a foundation for subsequent access to files lost due to structural corruption.

[0089] In some embodiments, after the target metadata repair is completed, for the metadata service of each node in the storage system, if the metadata service has been started and is running, the electronic device can release the unused space in the aforementioned reserved space. Then, a new reserved space is requested, and the address of the newly requested reserved space is saved in the reserved space field. This facilitates the next metadata repair, resolving the circular dependency problem between target database repair and space allocation failure during the next metadata repair process, thereby ensuring that the storage system can repair the metadata in the target database in a timely manner and improving the efficiency of metadata repair. Furthermore, releasing unused space in the reserved space reduces the occurrence of storage space reuse.

[0090] It should be noted that during the repair operation of target metadata in the target database, if an interruption in the metadata repair of any node is detected and the metadata service of that node is restarted, the electronic device can directly obtain the node repair result from the status database. Then, based on the node repair result, the metadata repair operation continues. In this way, by using the repair status of each node included in the status database, the electronic device can automatically control the entire repair process without frequent manual intervention, reducing maintenance costs while improving the efficiency of metadata repair.

[0091] Specifically, the electronic device can determine at least one repair stage in the pending repair state from the node repair results. Then, following the execution order of the at least one repair stage, it executes the repair tasks of each repair stage sequentially. In this way, subsequent repair operations can be performed directly based on the repair status of each node recorded in the status database. This allows the electronic device to automatically control the entire repair process without frequent manual intervention, reducing maintenance costs while improving the efficiency of metadata repair.

[0092] When there are multiple repair stages in the pending repair state, the electronic device can determine the first repair stage from among the multiple repair stages according to the execution order of the multiple repair stages and execute the repair task of the first repair stage. The first repair stage is the first repair stage that needs to be executed among the multiple repair stages. Subsequently, if any node in the storage system has an unrepaired first repair stage while the first repair stage is in a repaired state, the system waits for that unrepaired node to execute its first repair stage task until the first repair stages of all nodes are in a repaired state, at which point the second repair stage task is executed. If every node in the storage system has an unrepaired first repair stage while the first repair stage is in a repaired state, the system continues to execute the second repair stage task until all repair stages have been completed. The second repair stage is the adjacent repair stage following the first repair stage.

[0093] In other words, if the current repair phase of one node is completed, it must wait for the current repair phases of other nodes to also be completed before proceeding to the next repair phase. This allows for state synchronization across nodes, enabling coordinated repair across all nodes. This not only significantly reduces repair time and improves the efficiency of metadata repair, but also reduces labor costs, avoids frequent manual intervention, lowers operational costs, and ultimately enhances the user experience.

[0094] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0095] Corresponding to the metadata repair method described above, this disclosure also proposes a metadata repair apparatus. Since the apparatus embodiments of this disclosure correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to the method embodiments described above, and will not be repeated here.

[0096] Figure 4 This is a schematic diagram of the structure of a metadata repair device provided in an embodiment of this disclosure, as shown below. Figure 4 As shown, it includes: a metadata storage module 30, a metadata acquisition module 31, and a metadata repair module 32.

[0097] Metadata storage module 30 is used to add a target storage volume with write caching disabled in the storage volume of the storage system, and to store the first metadata in the target storage volume; wherein, the first metadata is used to provide a management interface for the target metadata;

[0098] Metadata acquisition module 31 is used to acquire first metadata from the target storage volume of the storage system in response to the storage system meeting the repair trigger condition;

[0099] Metadata repair module 32 is used to repair target metadata in the target database using first metadata; wherein the repair operation includes at least one of log replay operation, file scan operation, metadata repair operation and directory tree reconstruction operation.

[0100] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair device also includes a system configuration module 33.

[0101] System configuration module 33 is used to deploy metadata services for each node in the storage system;

[0102] A reserved space field is added to the superblock structure of the first metadata; the superblock structure of the first metadata is stored in a persistent storage device in the storage system.

[0103] For each node's metadata service, in response to the first startup of the metadata service and while the metadata service is running, a reserved space is requested, and the space address of the reserved space is saved in the reserved space field.

[0104] Furthermore, in one possible implementation of this embodiment, such as Figure 5As shown, the metadata repair module 32 is also used to obtain the space address of the reserved space from the reserved space field of the superblock structure of the first metadata;

[0105] Use the reserved space to create temporary files; these temporary files are used to store temporary data generated during the target metadata repair process.

[0106] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the system configuration module 33 is also used to deploy database services for each node in the storage system and create a database cluster through the database services of multiple nodes;

[0107] Create a state database in the database cluster; the state database includes node information and node repair results for each node in the storage system, and the node repair results include the repair status of each repair stage.

[0108] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to save the node information and node repair results of each node in the storage system to the status database; wherein, the node repair results include the repair status of each repair stage in the node;

[0109] If a node's metadata repair is interrupted, and the node's metadata service is restarted, the node repair result is retrieved from the status database.

[0110] Based on the node repair results, continue with the metadata repair operation.

[0111] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to determine from the node repair results of the node that the repair status is in at least one repair stage that is in the pending repair state;

[0112] The repair tasks of each repair phase are executed sequentially according to the execution order of at least one repair phase.

[0113] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to determine the first repair stage from the multiple repair stages according to the execution order of the multiple repair stages, and execute the repair task of the first repair stage; wherein, the first repair stage is the first repair stage that needs to be executed among the multiple repair stages;

[0114] If any node in the storage system is in an unrepaired state during the first repair phase while it is in a repaired state during the first repair phase, the repair task of the first repair phase will be executed on the unrepaired node until the first repair phase of all nodes is in a repaired state. Then the repair task of the second repair phase will be executed.

[0115] If the first repair phase is in a repaired state, and if the first repair phase of each node in the storage system is in a repaired state, the repair task of the second repair phase continues to be executed until the repair tasks of multiple repair phases are completed; wherein, the second repair phase is the adjacent repair phase after the first repair phase.

[0116] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to scan the target database and obtain the scan results;

[0117] The target database is divided into multiple repair partitions based on the total number of repair threads; each repair partition corresponds to one repair thread.

[0118] For each repair partition in the target database, the repair partition is repaired to obtain the target metadata in the repair partition.

[0119] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to rebuild the directory tree of the target database.

[0120] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to backtrack the metadata in the target database to obtain path tracing results; wherein, the path tracing results are used to characterize whether all path items in the target database carry the target structure information that has been linked, and the target structure information includes memory structure information;

[0121] If the path tracing results represent path entries in the target database that all carry linked target structure information, a message indicating that the repair is complete is generated.

[0122] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to generate new memory structure information in combination with the file scan results when any path item in the target database does not carry the target structure information that has been linked.

[0123] Generate a message indicating that the repair is complete.

[0124] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the system configuration module 33 is also used to modify the writing logic of the target log to obtain the modified log writing logic; wherein, the target log is used to record the processing source of the first metadata, and the modified log writing logic is used to enable the target log to bypass the cache device and be directly written to the persistent storage device in the storage system.

[0125] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used to aggregate the first log with a space capacity less than the preset space capacity when the space capacity of the first log is less than the preset space capacity, until the space capacity of the aggregated first log is greater than or equal to the preset space capacity, and then write the aggregated first log as the target log to the storage device.

[0126] Furthermore, in one possible implementation of this embodiment, such as Figure 5 As shown, the metadata repair module 32 is also used for metadata services for each node in the storage system. If the metadata service has been started and is in running state, it releases unused space in the reserved space.

[0127] Apply for a new reserved space and save the address of the newly applied reserved space to the reserved space field.

[0128] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0129] For a description of the features in the embodiment corresponding to the metadata repair device, please refer to the relevant description of the embodiment corresponding to the metadata repair method, which will not be repeated here.

[0130] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described metadata repair method embodiments.

[0131] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described metadata repair method embodiments at runtime.

[0132] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0133] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described metadata repair method embodiments.

[0134] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described metadata repair method embodiments.

[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] The foregoing has provided a detailed description of a metadata repair and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A metadata repair method, characterized in that, include: Add a target storage volume with write caching disabled in the storage volume of the storage system, and store the first metadata in the target storage volume; wherein, the first metadata is used to provide a management interface for the target metadata; In response to the storage system meeting the repair trigger condition, the first metadata is obtained from the target storage volume of the storage system; Using the first metadata, a repair operation is performed on the target metadata in the target database; wherein, the repair operation includes at least one of log replay operation, file scanning operation, metadata repair operation, and directory tree reconstruction operation; Before retrieving the first metadata from the target storage volume of the storage system in response to the storage system meeting the repair trigger condition, the method further includes: Deploy metadata services for each node in the storage system; In the superblock structure of the first metadata, a reserved space field is added; wherein, the superblock structure of the first metadata is stored in a persistent storage device in the storage system; For the metadata service of each node, in response to the first startup of the metadata service and the metadata service being in a running state, a reserved space is requested, and the space address of the reserved space is saved in the reserved space field.

2. The method according to claim 1, characterized in that, After obtaining the first metadata from the target storage volume of the storage system, the method further includes: Obtain the space address of the reserved space from the reserved space field of the superblock structure of the first metadata; Using the reserved space, a temporary file is created; wherein the temporary file is used to store temporary data generated during the target metadata repair process.

3. The method according to claim 1, characterized in that, The method further includes: Deploy database services for each node in the storage system, and create a database cluster through the database services of multiple nodes; A status database is created in the database cluster; wherein, the status database includes node information and node repair results for each node in the storage system, and the node repair results include the repair status for each repair stage.

4. The method according to claim 3, characterized in that, After obtaining the first metadata from the target storage volume of the storage system, the method further includes: For each node in the storage system, the node information and node repair results are saved to the status database; wherein, the node repair results include the repair status of each repair stage in the node; If an interruption is detected in the metadata repair of the node, and the metadata service of the node is detected to be restarted, the node repair result of the node is obtained from the status database. Based on the node repair results, continue with the metadata repair operation.

5. The method according to claim 4, characterized in that, The step of continuing the metadata repair operation based on the node repair results includes: Determine at least one repair stage from the node repair results of the node that is in a state of pending repair; The repair tasks of each repair stage are executed sequentially according to the execution order of the at least one repair stage.

6. The method according to claim 5, characterized in that, When there are multiple repair stages in the pending repair state, the step of sequentially executing the repair tasks of the repair stages according to the execution order of the at least one repair stage includes: According to the execution order of the multiple repair stages, a first repair stage is determined from the multiple repair stages, and the repair task of the first repair stage is executed; wherein, the first repair stage is the first repair stage that needs to be executed among the multiple repair stages; If any node in the storage system is in an unrepaired state during the first repair phase, and the unrepaired node is waiting to perform the repair task of the first repair phase until the first repair phase of all nodes is in a repaired state, then the repair task of the second repair phase is performed. If the first repair phase is in a repaired state, and the first repair phase of each node in the storage system is in a repaired state, the repair task of the second repair phase continues to be executed until the repair tasks of the multiple repair phases are all completed; wherein, the second repair phase is an adjacent repair phase following the first repair phase.

7. The method according to any one of claims 1-6, characterized in that, The step of repairing the target metadata in the target database using the first metadata includes: The target database is scanned to obtain the scan results; The target database is divided into multiple repair partitions based on the total number of repair threads; each repair partition corresponds to one repair thread. For each repair partition in the target database, the repair partition is repaired to obtain the target metadata in the repair partition.

8. The method according to claim 7, characterized in that, After repairing the repair partition and obtaining the target metadata in the repair partition, the method further includes: The target database is reconstructed into a directory tree.

9. The method according to claim 8, characterized in that, The process of reconstructing the directory tree of the target database includes: The metadata in the target database is traced back to obtain path tracing results; wherein, the path tracing results are used to characterize whether all path items in the target database carry target structure information that has been linked, and the target structure information includes memory structure information; When the path tracing results indicate that all path entries in the target database carry linked target structure information, a reconstruction completion notification is generated.

10. The method according to claim 9, characterized in that, The method further includes: If the path tracing results indicate that any path entry in the target database does not carry the target structure information that has been linked, new memory structure information is generated by combining the file scanning results. Generate a notification message indicating that the reconstruction is complete.

11. The method according to any one of claims 1-6, characterized in that, Before retrieving the first metadata from the target storage volume of the storage system in response to the storage system meeting the repair trigger condition, the method further includes: The writing logic of the target log is modified to obtain the modified log writing logic; wherein, the target log is used to record the processing source of the first metadata, and the modified log writing logic is used to enable the target log to bypass the caching device and be directly written to the persistent storage device in the storage system.

12. The method according to claim 11, characterized in that, The method further includes: If the space capacity of the first log is less than the preset space capacity, the first log with a space capacity less than the preset space capacity is aggregated until the space capacity of the aggregated first log is greater than or equal to the preset space capacity, and the aggregated first log is written to the storage device as the target log.

13. The method according to any one of claims 1-6, characterized in that, After repairing the target metadata in the target database using the first metadata, the method further includes: For the metadata service of each node in the storage system, if the metadata service has been started and is in running state, release the unused space in the reserved space; Re-apply for a reserved space and save the address of the newly applied reserved space to the reserved space field.

14. An electronic device, characterized in that, include: At least one processor; as well as A storage device communicatively connected to the at least one processor; wherein, The storage device stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the metadata repair method according to any one of claims 1-13.

Images