Lustre mirror high availability scenario copy consistency self-recovery method and system

Abstract

The application provides a Lustre mirror high-availability scenario copy consistency self-recovery method and system, which is applied to a Lustre file system FLR mirror scenario, and the method comprises the following steps: after a client read-write request exception and retry failure, determining an OST exception and switching a new master copy for retry; an external coordinator periodically detects an OST state, and when the OST state is abnormal, a non-active marking instruction is issued to trigger copy switching; after the OST is reconnected, incremental synchronization is performed through a change log and is marked as active. The application also discloses a corresponding system, which is realized based on a Lustre native interface, does not need to modify a kernel, can automatically complete fault self-recovery and copy consistency repair, does not need manual intervention, and effectively improves service availability.

Description

Technical Field

[0001] This invention belongs to the field of distributed file system technology, and in particular relates to a self-healing method and system for replica consistency in Lustre image high availability scenarios. Background Technology

[0002] The Lustre file system supports File-Level Redundancy (FLR), which creates synchronous mirror copies of a single file across multiple object storage targets. When writing data, all replicas must be written simultaneously to ensure data consistency. However, in practice, when the object storage target corresponding to the primary replica experiences a crash or network interruption, client read / write requests will wait for an extended period at the communication layer, causing business threads to block. Even if other healthy replicas of the file exist, read / write services cannot function normally. Current maintenance methods rely on manually marking failed object storage targets as unavailable and then rebuilding replicas using tools. This process has low automation and slow response times, failing to meet the operational requirements of high-availability services. Furthermore, after the failed object storage target recovers, the replica data may differ, and a full replica rebuild consumes significant cluster network and storage resources. There is a lack of efficient consistency repair methods adapted to this scenario. Summary of the Invention

[0003] This application provides a replica consistency self-healing system for high availability scenarios of Lustre images. In the Lustre file system FLR image read and write scenario, it can not only realize the rapid recovery of read and write services when the object storage target is abnormal, but also ensure the consistency of replica data after the faulty node is recovered, while not modifying the system kernel to adapt to the deployment requirements of the existing production environment.

[0004] This application discloses a self-healing method for improving replica consistency in high-availability Lustre image scenarios, applied to Lustre file system FLR image scenarios. FLR is Lustre's built-in file-level redundancy feature, used to create synchronized image replicas of a single file across multiple OSTs. OSTs are Lustre object storage targets, including: The client receives read / write requests from business applications and sends them to the corresponding OST of the primary replica. If the request triggers a timeout or I / O exception and fails to retry, the client determines that the OST is abnormal, queries the MDS for the file storage layout, selects a healthy replica as the new primary replica, and retryes the request. The MDS is Lustre's metadata service node. The external coordinator checks the online status of the OST at a preset period. When an anomaly is detected, it sends an inactive flag instruction to the kernel through Lustre native tools. The kernel then terminates the pending I / O requests of the OST and triggers client replica switching. After detecting that an inactive OST has come back online, extract the change log of the faulty area to generate a file list, call the Lustre native image recovery tool to perform incremental synchronization, and mark the OST as active after completion.

[0005] Optionally, before the external coordinator issues an inactive flag instruction to the Lustre kernel, it performs the following steps: The external coordinator receives I / O exception events reported by the client and performs online status supplementary checks on the corresponding OST based on the exception events; By combining the periodic detection results and the supplementary detection results, the abnormal state of the target OST is confirmed. The target OST is the OST that is currently waiting to be issued an inactive marking instruction.

[0006] Optionally, in the process of extracting the change log of the fault area to generate a file list, the change log entries are deduplicated based on the file identifier. Only the latest entry is retained among multiple change entries in the same file, and the file list is generated based on the deduplicated entries. After performing incremental synchronization, a copy verification and comparison is performed on the synchronized files to confirm that the copy on the re-uploaded OST is consistent with the healthy copy data.

[0007] Optionally, before the external coordinator issues the inactive flag instruction, the following steps are performed: Based on the minimum replica redundancy of the Lustre cluster FLR image, calculate the maximum number of concurrent isolations N, where N is the minimum replica redundancy minus one. Create a fixed number of N isolation slot files on the MGS node of the Lustre cluster, and configure a DLM exclusive lock for each slot file. The MGS is the global management node of the Lustre cluster, and the DLM is the native distributed lock manager of the Lustre kernel, which is used to ensure that a slot file can only be held by one isolation operation at the same time. For OSTs that are confirmed to be abnormal, request an exclusive lock on the slot file from MGS. Only after the exclusive lock is successfully requested will an inactive flag command be issued.

[0008] Optionally, before requesting an exclusive lock for the slot file from MGS, the following steps are performed: Based on the directory-level FLR strategy of the Lustre cluster, a service priority is marked for each abnormal OST; Create isolated queues for abnormal OSTs in descending order of business priority; In the order of the isolation queue, the abnormal OST requests an exclusive lock on the slot file in turn; If all slot files are occupied by low-priority abnormal OSTs, and there are high-priority abnormal OSTs in the queue, then release the slot files occupied by the low-priority OSTs and allocate slot files for the high-priority OSTs.

[0009] Optionally, after the exclusive lock application is successful but before the inactive flag instruction is issued, the following steps are performed: Retrieve a list of all image files stored on the exception OST that acquired the exclusive lock; For each image file in the list, verify the number of remaining healthy copies of the file after isolating the abnormal OST; If the number of remaining healthy copies of all image files is greater than or equal to one, then an inactive marking instruction is issued. If the number of remaining healthy copies of an image file is less than one, the exclusive lock on the corresponding slot file is released, triggering the creation process of a temporary image copy of that file.

[0010] Optionally, after marking the target OST as inactive, the following steps are performed: Write the inactive status information and corresponding slot number of the target OST into the corresponding slot file on the MGS; The inactive status information of the target OST is synchronized to all MDS, OST and client nodes in the cluster through the MGS native configuration synchronization mechanism. Once feedback is received that the status synchronization of more than two-thirds of the nodes in the cluster is complete, the isolation operation is confirmed to be closed-loop, and the exclusive lock of the corresponding slot file is released.

[0011] Optionally, after the isolation operation is confirmed to be completed, the following steps are performed: After the exclusive lock on a slot file is released, the slot file is marked as available. According to the order of the isolation queue, apply for an exclusive lock on the slot file of the first abnormal OST in the queue that has not been isolated; Repeat the steps of replica redundancy pre-verification, inactive marking, and cluster state synchronization until all abnormal OSTs in the isolation queue have been isolated.

[0012] Optionally, after remarking the OST as active, the following steps are performed: Synchronize the active status information of OST to the slot file of MGS and update the full-process audit log of the corresponding isolation operation; The active status information of OST is synchronized to all MDS, OST and client nodes in the cluster through the MGS native configuration synchronization mechanism. Verify the consistency of status information of all nodes in the cluster to complete the self-healing process closed loop.

[0013] This application also discloses a replica consistency self-healing system to improve the high availability of Lustre images, including: an enhanced client module, an external coordinator module, an incremental repair service module, and an MGS global management node; The enhanced client module is used to receive read and write requests initiated by business applications for the primary replica of Lustre files. After the request triggers a timeout or I / O exception and the retry fails, it queries the storage layout information of the file from MDS, selects healthy replicas as the new primary replicas, retryes the request, and returns the execution result to the business application. The external coordinator module is used to perform online status detection of OSTs according to a preset period, issue status marking instructions to the Lustre kernel for abnormal OSTs, apply for exclusive locks of isolation slot files from the MGS global management node, and perform business priority scheduling and pre-verification of replica redundancy before isolation of abnormal OSTs. The incremental repair service module is used to extract change log entries corresponding to the fault range from MDS after an OST marked as inactive is brought back online, parse and generate a list of change files, and call the Lustre native image recovery tool to perform incremental data synchronization. The MGS global management node is used to store isolated slot files and cluster configuration information, and synchronizes OST status information to all nodes in the cluster through the native configuration synchronization mechanism.

[0014] As can be seen from the above technical solution, the client receives read and write requests from business applications and sends them to the object storage target corresponding to the primary replica. After the request triggers an exception and retry fails, the node is determined to be abnormal, and the file storage layout is queried from the metadata service node. Healthy replicas are selected and switched to become the primary replica, and the request is retried. The automatic switching of service paths can be completed directly on the read and write paths, avoiding continuous blocking of read and write requests. The periodic detection and inactive marking operations of the external coordinator can terminate the input and output requests suspended on the faulty node, forming a connection with the client's replica switching process, completing the isolation of the abnormal node and service recovery. After the faulty node comes back online, the incremental synchronization operation based on the change log can complete the consistency repair of the replica data. The three links are connected in sequence to form a complete fault self-healing process. Without manual intervention and kernel modification, the read and write service can be quickly restored in case of anomalies, while ensuring the consistency of replica data and improving the service availability of Lustre image read and write scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a method for improving replica consistency self-healing in high availability scenarios of Lustre images, as described in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a replica consistency self-healing system for improving the high availability of Lustre images in an embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without such specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0018] This application also provides a method for improving replica consistency and self-healing in high-availability Lustre image scenarios, applied to Lustre file system FLR image scenarios. FLR is Lustre's built-in file-level redundancy feature, used to create synchronized image copies of a single file across multiple OSTs. OSTs are Lustre object storage targets, such as... Figure 1 As shown, it includes: S100: The client receives read / write requests from business applications and sends them to the corresponding OST of the primary replica. If the request triggers a timeout or I / O exception and fails to retry, the OST is determined to be abnormal. The client queries the MDS for the file storage layout, selects the healthy replica as the new primary replica, and retryes the request. The MDS is the metadata service node of Lustre. S200: The external coordinator checks the online status of the OST at a preset period. When an anomaly is detected, it sends an inactive flag instruction to the kernel through the Lustre native tool. The kernel terminates the I / O requests of the OST and triggers the client replica switch. S300: After detecting that an inactive OST has come back online, it extracts the change log of the fault range to generate a file list, calls the Lustre native image recovery tool to perform incremental synchronization, and marks the OST as active after completion.

[0019] This application provides a self-healing method for improving replica consistency in high-availability scenarios of Lustre images. The client receives read and write requests from business applications and sends them to the object storage target corresponding to the primary replica. After the request triggers an exception and retry fails, the node is determined to be abnormal, and the file storage layout is queried from the metadata service node. Healthy replicas are selected and switched to become the primary replica, and the request is retried. The automatic switching of service paths can be completed directly on the read and write paths, avoiding continuous blocking of read and write requests. The periodic detection and inactive marking operations of the external coordinator can terminate the pending input and output requests on the faulty node, forming a connection with the client's replica switching process to complete the isolation of the abnormal node and service recovery. After the faulty node comes back online, the incremental synchronization operation based on the change log can complete the consistency repair of the replica data. The three links are connected in sequence to form a complete fault self-healing process. Without manual intervention and kernel modification, it can realize the rapid recovery of read and write services in case of anomalies, while ensuring the consistency of replica data and improving the service availability of Lustre image read and write scenarios.

[0020] It should be noted that the core concept of this application specifically addresses the operational requirements of Lustre file system FLR image read / write scenarios by proposing a complete self-healing method. This method automatically queries the file storage layout when a read / write request fails, selects a healthy replica as the new primary replica, and retryes the request, thus achieving automatic recovery of read / write services. An external coordinator periodically checks the online status of the object storage target; when an anomaly occurs, it issues an inactive marker command to the kernel, terminating pending input / output requests and triggering the client replica switchover process. After the failed object storage target comes back online, the change logs for the failed interval are extracted to generate a file list, and native tools are invoked to perform incremental data synchronization. Upon completion, the marked node is declared active. This method is implemented entirely based on Lustre's native interfaces and tools, requiring no modification to the system kernel. It can be directly deployed in existing production environments, achieving self-healing in read / write scenarios and improving service availability and replica data consistency.

[0021] The embodiments of this application will be described in detail below.

[0022] Specifically, in this embodiment, the Lustre file system is a parallel distributed file system that can be applied to data storage and access scenarios in large-scale clusters. The Lustre mirror read / write scenario is a file mirror copy read / write scenario built on the FLR function. FLR stands for File Level Redundancy, which is a built-in function module of the Lustre file system. It can create synchronous mirror copies of a single file on multiple OSTs. OST stands for Object Storage Target, which is a component in the Lustre file system used to store the actual data of user files. Multiple OSTs can be deployed on different physical nodes to achieve redundant data storage and access load sharing.

[0023] It should be noted that the client is deployed on the business access node of the Lustre file system. The business access node is the computing node in the cluster used to run business applications. The client is the access component of the Lustre file system deployed on the business access node. The business application runs on the business access node. During the execution of data processing tasks, the business application sends read and write operation instructions for the Lustre file to the client. The read and write operation instructions include two types: data read requests and data write requests. Data read requests are used to obtain the actual data content stored in the Lustre file, and data write requests are used to write new data content to the Lustre file or modify existing data content.

[0024] For example, after receiving a read / write request from a business application, the client first parses the file identification information contained in the read / write request. The file identification information is the encoding information used in the Lustre file system to uniquely distinguish different files. Based on the file identification information, the client retrieves the image copy configuration information of the file from the local cache. The image copy configuration information includes the primary copy information and secondary copy information of the file. The primary copy is the default copy for executing read / write operations on the file, and the secondary copy is a redundant backup copy of the file. Both the primary copy and the secondary copy are stored on the corresponding OST. Based on the primary copy information, the client sends the received read / write request to the OST corresponding to the primary copy, and the OST performs the specific read / write operation.

[0025] In this embodiment, after the client issues a read / write request, it continuously monitors the execution status of the request. The monitoring includes the response time and the execution result. The response time is the time interval between the client issuing the read / write request and receiving the response information returned by the OST. The execution result includes successful read / write operation results and failed read / write operation exception information. When the OST corresponding to the primary replica experiences hardware failure, network interruption, or storage media damage, the read / write request issued by the client cannot be processed normally, resulting in two abnormal situations: first, the read / write request does not receive any response information from the OST within a preset time, triggering a timeout exception; second, the OST returns feedback information indicating that the read / write operation failed, triggering an I / O exception. I / O stands for Input / Output, and an I / O exception is an input / output exception.

[0026] It should be noted that when the client detects a timeout or I / O exception in a read / write request, it will retry the request according to a preset retry strategy. This strategy includes a preset number of retries and a retry interval. The number of retries is the maximum number of times the client can repeatedly send the same read / write request, and the retry interval is the time interval between two consecutive retry operations. The parameters of the retry strategy can be adjusted by modifying the client's configuration file to adapt to the operational requirements of different clusters. For example, if the preset number of retries is two, after the client detects a read / write request exception, it will resend the read / write request to the OST corresponding to the primary replica after a preset interval. If the read / write request still fails after two retries, the client will determine that the OST corresponding to the primary replica is in an abnormal state and cannot perform read / write operations normally.

[0027] Those skilled in the art will understand that after a client determines that an OST is abnormal, it will not directly return a result indicating that the read / write operation failed to the business application. Instead, it will send a file storage layout query request to the MDS. MDS stands for Meta data Server, which is a service node in the Lustre file system used to manage file metadata information and storage layout information. Metadata information includes file identifiers, permissions, creation time, modification time, etc. Storage layout information records the storage location, replica status, and corresponding OST identifier of all mirror copies of the file. The MDS stores the metadata information and storage layout information of all files in the cluster and can respond to client query requests by returning relevant information for the corresponding file.

[0028] For example, after receiving a file storage layout query request from a client, MDS first parses the file identification information contained in the request, retrieves the current storage layout information of the corresponding file from the metadata storage area based on the file identification information, encapsulates the storage layout information into a standard format data packet, and returns it to the client through the cluster's internal communication network. After receiving the storage layout information, the client parses the data packet to obtain the corresponding OST information and replica status information of all mirror replicas of the file.

[0029] Specifically, after obtaining the current storage layout information of the file, the client iterates through all the mirror replicas recorded in the storage layout information. Based on the running status of the OST corresponding to the replica, it filters out healthy replicas. Healthy replicas are mirror replicas stored on OSTs that can communicate and perform read and write operations normally. The client can obtain the running status information of the OST by sending a status probe command to the OST corresponding to the replica. The status probe command is a native node status detection command of the Lustre file system, which can be directly called and executed without additional development and deployment. From the filtered healthy replicas, the client selects one replica as the new primary replica of the file. The new primary replica will replace the original abnormal primary replica and take over the task of performing read and write operations on the file. The rules for selecting healthy replicas can be executed according to a preset load balancing strategy. The load balancing strategy can prioritize the replica corresponding to the OST with the lowest current read and write load, or it can prioritize the replica corresponding to the OST with the best network link status with the client. The strategy rules can be adjusted through the configuration file.

[0030] In this embodiment, after the client completes the setting of the new primary replica, it will reissue the read / write request based on the OST corresponding to the new primary replica. The OST corresponding to the new primary replica will then perform the specific read / write operation. After the read / write operation is completed, the client will return the execution result to the business application. The business application can then continue to perform subsequent data processing tasks based on the execution result. The entire process of replica switching and request retry is transparent to the business application and will not affect the normal operation of the business application.

[0031] Furthermore, this embodiment also includes an external coordinator, deployed on the management node of the Lustre cluster. The management node is the node within the cluster used to manage the overall operational status of the cluster, and the external coordinator is a service program running on the management node, used to perform operations such as status detection, anomaly isolation, and status synchronization of OSTs within the cluster. The external coordinator performs online status detection operations on all OSTs within the cluster according to a preset period. The preset period is a pre-defined time interval between two adjacent status detection operations. This time interval can be adjusted through the configuration parameters of the external coordinator to adapt to the operational needs of different clusters. For example, the preset period can be set to ten seconds, and the external coordinator will perform a complete online status detection on all OSTs within the cluster every ten seconds.

[0032] In a more specific embodiment of this application, the external coordinator is implemented as a user-space daemon running on the MGS node (Global Management Node) or an independent management node. This daemon interacts with the Lustre kernel by calling the native Lustre lctl command-line tool. Specifically, it issues inactive flag commands by executing the `lctl deactivate<target OST number>` command; it issues active flag commands by executing the `lctl activate<target OST number>` command. The daemon also obtains cluster information such as OST online status by calling `lctl` or directly parsing Lustre-related entries in the ` / proc` and ` / sys` file systems. To ensure its high availability, the external coordinator can be deployed in an active-standby mode and managed using cluster resource managers such as Pacemaker.

[0033] It should be noted that the external coordinator performs online status checks on OSTs using Lustre's native status check tool. This tool is a built-in node status monitoring program in the Lustre file system, requiring no additional development or deployment. The external coordinator calls this tool to send status check commands to each OST in the cluster. Upon receiving the commands, each OST returns its own running status information. The external coordinator then generates status check results based on the status information returned by each OST. These results include the OST's online status, offline status, and response latency parameters. The response latency parameter is the time interval between sending the status check command and receiving the response information.

[0034] For example, if the status detection results obtained by the external coordinator show that the target OST has not returned status detection response information, or the response latency parameter of the target OST exceeds a preset threshold (the preset threshold is set based on the normal communication latency range of the cluster), the external coordinator will determine that the target OST is in an abnormal response state and cannot perform normal read / write operations. After determining that the target OST is abnormal, the external coordinator will issue an inactive marking instruction to the Lustre kernel through Lustre native tools. The Lustre kernel is the core control component of the Lustre file system, deployed on each node in the cluster, used to manage the running status of each node in the cluster, read / write request scheduling, resource allocation, and other operations. The Lustre native tools are cluster parameter configuration and status management tools that come with the Lustre file system and can be directly called to execute the status marking operation.

[0035] Specifically, after receiving an inactive marking instruction from the external coordinator, the Lustre kernel marks the specified target OST as inactive. An inactive OST cannot receive new read / write requests, nor can it respond to incomplete read / write requests. The Lustre kernel's request scheduling module blocks the read / write channel of that OST, ceasing to issue any new read / write operation instructions. Simultaneously, the Lustre kernel terminates all pending I / O requests on that target OST. Pending I / O requests are read / write requests that have been sent to the target OST but not yet completed. These requests are stored in the OST's request cache area. The Lustre kernel, through its cache cleanup module, terminates the execution of these requests and returns I / O exception information to the client that initiated the request.

[0036] In this embodiment, after the client receives the I / O exception information returned by the Lustre kernel, it will trigger the replica switching process, that is, query the storage layout information of the corresponding file from the MDS, select the healthy replica as the new primary replica, reissue the read and write request, and complete the execution of the read and write operation. This process is consistent with the replica switching process executed when the client detects the request exception, which can realize the isolation of the abnormal OST and the recovery of the read and write service, and avoid the read and write request being blocked for a long time.

[0037] In addition, the external coordinator continuously monitors the operational and access status of all OSTs within the cluster. When an OST fails, maintenance personnel will repair the faulty hardware or network. After repair, the OST will reconnect to the cluster and send registration access information to the cluster management node. By listening to the node registration information in the cluster, the external coordinator can detect the re-entry of OSTs marked as inactive. After detecting the re-entry of an inactive OST, the external coordinator obtains the fault interval information of that OST. The fault interval information includes the fault start time and recovery completion time of the OST. The fault start time is the time when the OST was marked as inactive, which is recorded in the status log of the cluster management node. The recovery completion time is the time when the OST reconnects to the cluster and completes registration, which is generated by the node registration process.

[0038] It should be noted that after the external coordinator obtains the fault zone information, it sends a change log query request to the MDS to extract change log entries within the fault zone. The change log is a log in the Lustre file system used to record file metadata changes and data change events. A change log entry is the smallest record unit in the change log, and each entry corresponds to one file change operation. The entry contains information such as file identifier, change operation type, operation execution time, and operation execution node. Change log entries are stored sequentially in the MDS log storage area according to the chronological order of operation execution. The external coordinator constructs a continuous time interval based on the fault start time and recovery completion time, extracts all change log entries within that time interval, parses the extracted change log entries to obtain the file identifiers of files whose data has been changed within the fault zone, and generates a list of changed files based on these file identifiers. This list is stored in list form and records the information of all files whose data content has been updated within the fault zone.

[0039] For example, after the external coordinator generates a list of changed files, it calls the Lustre native image recovery tool. Based on the list of changed files, it performs incremental data synchronization on the re-uploaded OST. The Lustre native image recovery tool is a built-in image copy data synchronization tool of the Lustre file system, which can achieve data consistency synchronization between multiple image copies. The incremental data synchronization operation only transmits the file data that has changed within the fault period, without needing to perform a full data transfer of all files stored on the OST, thus reducing the amount of data transfer and reducing the consumption of cluster network and storage resources. After the incremental data synchronization operation is completed, the external coordinator issues an active marking instruction to the Lustre kernel through the Lustre native tool. After receiving the instruction, the Lustre kernel remarkes the OST as active. An active OST can re-participate in the cluster's read and write request scheduling. The Lustre kernel's request scheduling module restores the read and write channel permissions of the OST and can issue new read and write operation instructions to the OST.

[0040] Before the external coordinator issues the inactive flag instruction to the Lustre kernel, it performs the following steps: The external coordinator receives I / O exception events reported by the client and performs online status supplementary checks on the corresponding OST based on the exception events; By combining the periodic detection results and the supplementary detection results, the abnormal state of the target OST is confirmed. The target OST is the OST that is currently waiting to be issued an inactive marking instruction.

[0041] In this embodiment, when a client detects that a read / write request triggers an I / O exception and the retry operation fails, in addition to executing the replica switchover process, it also reports the I / O exception event to the external coordinator. The I / O exception event contains information such as the OST identifier that triggered the exception, the time of the exception occurrence, and the exception type. The client sends the exception event information to the external coordinator through the cluster's internal communication network. The external coordinator is equipped with a dedicated event receiving module to receive the I / O exception event information reported by each client.

[0042] It should be noted that after receiving an I / O exception event reported by the client, the external coordinator will parse the OST identifier contained in the exception event to locate the target OST that triggered the exception. Based on the I / O exception event, it will perform an online supplementary status check operation on the target OST. The execution process of the supplementary check operation is the same as that of the periodic status check operation. The external coordinator calls the Lustre native status check tool to send a status check command to the target OST and obtain the target OST's running status information and response information. The supplementary check operation can be performed within the interval of the periodic check without waiting for the next periodic check time node, which can improve the response speed of exception status detection.

[0043] For example, the external coordinator's periodic detection interval is ten seconds. During the interval between two periodic detection operations, if the external coordinator receives an I / O anomaly event reported by the client for a specific OST, it will immediately perform a supplementary detection operation on that OST. This eliminates the need to wait for the next periodic detection time, allowing the external coordinator to obtain the OST's operational status information in advance and shorten the detection time for anomalies. The external coordinator will combine the results of the previously completed periodic detection with the results of this supplementary detection operation to comprehensively confirm the abnormal status of the target OST. If both the periodic detection results and the supplementary detection results indicate that the target OST is unreachable or has an abnormal response, the external coordinator will confirm that the target OST is in an abnormal state and can then execute the subsequent inactive marking instruction issuance operation.

[0044] Those skilled in the art will understand that by combining client-side abnormal event reporting with supplementary detection operations, a dual-path abnormality detection mechanism can be formed. Periodic detection operations can cover all OSTs in the cluster, including OSTs that have not yet been requested for read or write by clients. Supplementary detection operations can target OSTs that are currently being accessed by clients, enabling rapid response to abnormal states. The two detection methods work together to improve the coverage and response speed of abnormal state detection, avoiding the abnormal detection delay problem that occurs with a single periodic detection method. At the same time, combining the results of the two detection methods to confirm the abnormal state of an OST can improve the accuracy of abnormal state judgment and avoid misjudgments caused by temporary factors such as network fluctuations.

[0045] Specifically, if the periodic detection results show that the target OST is in a normal online state, but the client reports an I / O anomaly event for that OST, and the supplementary detection results show that the OST is responding abnormally, the external coordinator will combine the time interval between the two detections and the detection results to confirm whether the OST's running status has changed. If the supplementary detection results confirm that the OST is in an abnormal state, the external coordinator will confirm the abnormal state of the OST and perform subsequent isolation operations. If the supplementary detection results show that the OST is in a normal online state, the external coordinator will determine that the I / O anomaly is an occasional anomaly caused by temporary network fluctuations and will not perform inactive marking operations to avoid performing erroneous isolation operations on normally running OSTs and affecting the normal operation of the cluster.

[0046] During the process of extracting the change log of the fault range and generating the file list, the change log entries are deduplicated based on the file identifier. Only the latest entry is retained among multiple change entries in the same file. The file list is generated based on the deduplicated entries. After performing incremental synchronization, a copy verification and comparison is performed on the synchronized files to confirm that the copy on the re-uploaded OST is consistent with the healthy copy data.

[0047] In this embodiment, after the external coordinator extracts the change log entries within the fault zone from the MDS, it performs deduplication on the change log entries. The deduplication is based on the file identifier contained in the change log entries. Each file has a unique file identifier in the Lustre file system. The same file may have multiple data change operations within the fault zone, corresponding to multiple change log entries. These change log entries correspond to the same file identifier. The external coordinator performs deduplication on the multiple change log entries corresponding to the same file identifier, retaining only the most recent change log entry and deleting the remaining duplicate entries.

[0048] It's important to note that after deduplication, the external coordinator will parse the corresponding file identifiers from the deduplicated change log entries and generate a list of changed files. Deduplication avoids duplicate file information in the change file list, reducing redundant data transmission in subsequent incremental synchronization operations and lowering cluster resource consumption. For example, if a file underwent five data write operations within the fault period, corresponding to five change log entries, deduplication will retain only the latest change log entry. This file will then appear only once in the change file list, and subsequent incremental synchronization operations will only require one data synchronization operation for that file, eliminating the need for five repeated synchronization operations.

[0049] In addition, after the external coordinator calls the Lustre native image recovery tool to complete the incremental data synchronization operation, it performs a replica checksum comparison operation on all synchronized files. The checksum is a fixed-length value calculated from the file data content using a hash algorithm. Identical data content in the same file will yield the same checksum value; if the file data content differs, the calculated checksum values ​​will also differ. The external coordinator calculates the checksum values ​​of the file replicas stored on the re-uploaded OST and the checksum values ​​of the healthy replicas within the cluster, and compares the two checksum values. If the two values ​​are completely identical, it means that the data content of the two replicas of the file is completely identical, and the data synchronization operation is successful. If the two values ​​differ, it means that the data content of the two replicas of the file differs, and the data synchronization operation is incomplete. The external coordinator will then call the image recovery tool again to perform the data synchronization operation on the file until the checksum values ​​of the two replicas are completely identical.

[0050] Those skilled in the art will understand that replica verification and comparison operations can verify the execution results of incremental data synchronization operations, ensuring that the replicas on the re-uploaded OST are completely consistent with the healthy replicas within the cluster. This avoids data discrepancies caused by network fluctuations, transmission interruptions, or other factors during data synchronization, thus guaranteeing replica data consistency. For example, after the incremental synchronization operation is completed, the external coordinator performs replica verification and comparison operations sequentially on one hundred files in the changed file list. Ninety-nine files show consistent verification and comparison results. If the verification and comparison result for one file differs, the external coordinator will re-perform the data synchronization operation for that file. After synchronization, the verification and comparison will be performed again until the comparison results are consistent, ensuring that the replica data of all files is completely consistent.

[0051] Before the external coordinator issues the inactive flag instruction, the following steps are performed: Based on the minimum replica redundancy of the Lustre cluster FLR image, calculate the maximum number of concurrent isolations N, where N is the minimum replica redundancy minus one. Create a fixed number of N isolation slot files on the MGS node of the Lustre cluster, and configure a DLM exclusive lock for each slot file. The MGS is the global management node of the Lustre cluster, and the DLM is the native distributed lock manager of the Lustre kernel, which is used to ensure that a slot file can only be held by one isolation operation at the same time. For OSTs that are confirmed to be abnormal, request an exclusive lock on the slot file from MGS. Only after the exclusive lock is successfully requested will an inactive flag command be issued.

[0052] In this embodiment, after the external coordinator confirms the abnormal state of the target OST, it performs an isolation permission control operation before issuing an inactive marking command. This operation limits the number of OST isolation operations that can be performed simultaneously, preventing insufficient file mirror replicas within the cluster due to a large number of OSTs being isolated at the same time, thus affecting the normal operation of the cluster's read / write services. First, the external coordinator obtains the minimum replica redundancy of the Lustre cluster's FLR image. The minimum replica redundancy is the minimum number of replicas set for all files configured with FLR image replicas within the cluster. This parameter is set by the cluster administrator based on the security requirements of the business data and is stored in the configuration file of the cluster's global management node. The external coordinator can directly retrieve this parameter value from the global management node.

[0053] It's important to clarify that the isolation slot file is a regular file created in the global configuration directory of the MGS node, with a filename following a preset rule, such as / mnt / mgs / .isolate_slots / slot_1. Configuring a DLM exclusive lock for this slot file does not modify the Lustre kernel locking mechanism; rather, it utilizes the native file locking capabilities of the Lustre file system. Specifically, when the external coordinator's isolation operation process requests an exclusive lock, it attempts to open and lock the corresponding slot file in exclusive mode (write lock) (e.g., using the flock(fd, LOCK_EX) system call). Since Lustre's DLM coordinates file lock requests across nodes, this ensures that only one isolation operation in the entire cluster can successfully hold the file lock at any given time. If the lock is successful, it means the isolation slot has been acquired, and subsequent isolation can be performed; if the lock fails, it must wait, and releasing the exclusive lock involves closing the file descriptor or explicitly unlocking it.

[0054] It's important to note that after the external coordinator obtains the minimum replica redundancy value, it calculates the maximum number of concurrent isolations, N. N is calculated by subtracting one from the minimum replica redundancy value. This maximum number of concurrent isolations ensures that after performing an OST isolation operation, any file within the cluster retains at least one healthy image copy, preventing the inability to perform read / write operations due to a lack of usable image copies. For example, if the minimum replica redundancy of the FLR image within the cluster is three, the calculated maximum number of concurrent isolations, N, is two. This means that at most two OSTs can be isolated simultaneously, and after isolating two OSTs, any file within the cluster will still have at least one healthy image copy remaining, allowing for normal read / write operations. If the minimum replica redundancy of the FLR image within the cluster is two, the calculated maximum number of concurrent isolations, N, is one. This means that at most one OST can be isolated simultaneously, ensuring that at least one healthy copy of the file remains.

[0055] Specifically, after the external coordinator calculates the maximum number of concurrent isolations N, it creates a fixed number of isolation slot files on the MGS node of the Lustre cluster. MGS, short for Management Server, is the core service node in the Lustre cluster used to store cluster configuration information and manage the running status of all nodes. All OST, MDS, and client nodes in the cluster maintain continuous communication with the MGS node. The configuration information stored on the MGS node has authority over the entire cluster, and all nodes recognize the configuration information stored on the MGS node. The isolation slot files are stored in the global configuration storage directory of the MGS node. Each isolation slot file corresponds to an independent isolation operation permission. The number of isolation slot files is exactly the same as the calculated maximum number of concurrent isolations N. The content of the isolation slot files includes information such as the unique identifier of the slot, its occupancy status, the corresponding OST identifier, and the operation execution time.

[0056] In this embodiment, the external coordinator configures a DLM exclusive lock for each created isolation slot file. DLM stands for Distributed Lock Manager, a native distributed mutex lock component of the Lustre kernel, widely used in metadata concurrency management scenarios of the Lustre file system. It can be directly called and used without modifying the kernel program. The configuration rule of the DLM exclusive lock is that at the same time, an isolation slot file can only be held by one OST isolation operation. Isolation operations that do not hold an exclusive lock cannot modify the state of the corresponding slot file, nor can they execute the corresponding OST isolation operation. The state of the DLM exclusive lock is uniformly managed by the MGS node, and all nodes in the entire cluster can query the lock holding status.

[0057] For example, with a maximum number of concurrent isolations N of two, two isolation slot files are created on the MGS node, namely the first slot file and the second slot file. Each slot file is configured with a DLM exclusive lock. At the same time, each slot file can only be held by one isolation operation with an exclusive lock. The two slot files can support a maximum of two isolation operations to be executed at the same time, which is consistent with the value of the maximum number of concurrent isolations N. This physically limits the maximum number of isolation operations that can be executed at the same time, avoiding the execution of excessive isolation operations.

[0058] It should be noted that for any OST that is confirmed to be abnormal, the external coordinator sends an exclusive lock request command for the isolation slot file to the MGS node. This command includes the unique identifier of the abnormal OST, the request time, and the operation type. Upon receiving the command, the MGS node checks for available isolation slot files. If an available slot file exists, it allocates the DLM exclusive lock of that slot file to the requested isolation operation and returns a successful lock request message to the external coordinator. If no available isolation slot file exists, it returns a failed lock request message. The external coordinator only issues an inactive flag command to the Lustre kernel and executes the corresponding OST isolation operation after receiving a successful exclusive lock request message. If the exclusive lock request fails, the external coordinator pauses the isolation operation for that OST, waits for the isolation slot file to be released, and then resends the exclusive lock request command.

[0059] Those skilled in the art will understand that by configuring isolation slot files and DLM exclusive locks, the maximum number of OST isolation operations that can be executed simultaneously can be rigidly limited, ensuring that the number of isolation operations will not exceed the calculated maximum number of concurrent isolations N. Regardless of how many abnormal OSTs occur simultaneously within the cluster, there will be no over-isolation, guaranteeing that any file within the cluster always retains at least one healthy mirror copy, maintaining the normal operation of the cluster's read and write services, and avoiding interruptions to cluster read and write services due to the simultaneous isolation of a large number of OSTs. Furthermore, the entire isolation permission control process is implemented based on Lustre's native MGS nodes and DLM components, requiring no modification to the kernel program or additional deployment of third-party consensus components. It can be directly deployed in existing production clusters, demonstrating strong adaptability.

[0060] Before requesting an exclusive lock for the slot file from MGS, the following steps are performed: Based on the directory-level FLR strategy of the Lustre cluster, a service priority is marked for each abnormal OST; Create isolated queues for abnormal OSTs in descending order of business priority; In the order of the isolation queue, the abnormal OST requests an exclusive lock on the slot file in turn; If all slot files are occupied by low-priority abnormal OSTs, and there are high-priority abnormal OSTs in the queue, then release the slot files occupied by the low-priority OSTs and allocate slot files for the high-priority OSTs.

[0061] In this embodiment, before the external coordinator requests an exclusive lock for the isolated slot file from the MGS node, it performs a service priority scheduling operation. This operation is used to divide the execution order of isolation operations for abnormal OSTs corresponding to different services, ensuring that abnormal OSTs corresponding to high-priority services can execute isolation operations first, thus shortening the read / write service interruption time for high-priority services. First, the external coordinator marks the service priority for each abnormal OST based on the directory-level FLR policy of the Lustre cluster. The directory-level FLR policy is pre-set by the cluster administrator and is based on the mirror replica configuration rules of the file storage directory. Different file storage directories correspond to different levels of service types, and different service types correspond to different service priorities. The priority division is set based on factors such as service access frequency, data importance, and service level requirements.

[0062] It's important to note that the OSTs within the cluster store mirror copies of files in corresponding directories according to a directory-level FLR policy. This means that each file stored on an OST corresponds to a specific storage directory and business type. The external coordinator can assign a business priority to each OST based on the directory-level FLR policy corresponding to the files stored on that OST. For example, the storage directories within the cluster are divided into three levels: the first level is the real-time business directory, corresponding to the highest business priority, storing file data for real-time computing, online transactions, and other business applications; the second level is the offline business directory, corresponding to a medium business priority, storing file data for offline data analysis, data backup, and other business applications; and the third level is the cold data directory, corresponding to the lowest business priority, storing file data for archived data, historical backups, and other business applications. The OST stores file copies in different levels of directories according to the directory-level FLR policy. The external coordinator assigns the highest priority to OSTs storing real-time business directory files, a medium priority to OSTs storing offline business directory files, and the lowest priority to OSTs storing cold data directory files.

[0063] Specifically, after the external coordinator marks the business priority of all detected abnormal OSTs, it sorts all abnormal OSTs in descending order of business priority and generates an isolation queue. The isolation queue is stored in the form of a linked list. The queue records information such as the unique identifier of the abnormal OST, business priority, and time of occurrence of the abnormality. The isolation queue limits the order in which abnormal OSTs are isolated. The higher the business priority of the OST, the higher its ranking in the isolation queue, and the more likely it is to be isolated.

[0064] In this embodiment, the external coordinator sends exclusive lock requests for isolation slot files to the MGS node according to the order of the isolation queue. Higher-priority abnormal OSTs at the top of the queue send lock requests first, acquire exclusive locks on isolation slot files first, and perform isolation operations first. Lower-priority abnormal OSTs at the bottom of the queue send lock requests only after the isolation operations of the higher-priority OSTs are completed, and then perform their isolation operations. For example, if there are five abnormal OSTs in the isolation queue, two with the highest priority, two with medium priority, and one with the lowest priority, the external coordinator first sends lock requests to the two highest-priority OSTs. After they complete their isolation operations and release their slot files, the external coordinator then sends lock requests to the two medium-priority OSTs, and finally sends a lock request to the lowest-priority OST.

[0065] It should be noted that if all isolation slot files are currently occupied by low-priority abnormal OSTs, and there are still high-priority abnormal OSTs in the isolation queue that have not yet undergone isolation operations, the external coordinator will perform a priority preemption operation. First, it will release the isolation slot files occupied by the low-priority abnormal OSTs. The release operation is performed through the lock management module of the MGS node. The MGS node will release the DLM exclusive lock held by the low-priority isolation operation, mark the corresponding isolation slot file as available, and then allocate the released isolation slot file to the high-priority abnormal OST in the isolation queue. It will perform a lock allocation operation for the high-priority OST to ensure that the abnormal OST corresponding to the high-priority service can obtain isolation permission first and perform isolation operations first, thereby shortening the service interruption time of the high-priority service.

[0066] For example, the cluster is configured with two isolation slot files. There are two high-priority abnormal OSTs and three low-priority abnormal OSTs in the isolation queue. The three low-priority OSTs occupy the two isolation slot files first. At this time, the external coordinator detects that there are unprocessed high-priority abnormal OSTs in the queue. It will release the two isolation slot files occupied by the low-priority OSTs and allocate them to the two high-priority OSTs. After the high-priority OSTs complete the isolation operation and release the slot files, the slot files will be allocated to the low-priority OSTs and the isolation operation will be performed.

[0067] Those skilled in the art will understand that by using business priority marking, isolation queuing generation, and priority preemption operations, the isolation of abnormal OSTs corresponding to high-priority businesses can be prioritized under the premise of limited isolation slot resources. This allows for rapid recovery of read and write services for high-priority businesses, adapting to the service level requirements of different businesses within the cluster, and improving the operational stability of high-priority businesses while ensuring the redundancy of cluster replicas. Furthermore, the business priority division is based on the existing directory-level FLR strategy of the cluster, eliminating the need for additional complex topology aggregation and weight calculation modules. The logic is simple, directly adaptable to existing cluster configuration rules, and deployment and adjustment are relatively easy.

[0068] After the exclusive lock application is successful but before the inactive flag instruction is issued, the following steps are performed: Retrieve a list of all image files stored on the exception OST that acquired the exclusive lock; For each image file in the list, verify the number of remaining healthy copies of the file after isolating the abnormal OST; If the number of remaining healthy copies of all image files is greater than or equal to one, then an inactive marking instruction is issued. If the number of remaining healthy copies of an image file is less than one, the exclusive lock on the corresponding slot file is released, triggering the creation process of a temporary image copy of that file.

[0069] The process of triggering the creation of a temporary mirror copy of the file is specifically executed by the external coordinator. First, the storage layout of the file is queried from MDS to obtain a list of OSTs containing all existing healthy copies. Then, through mechanisms such as cluster load balancing interfaces, a healthy OST with sufficient resources is selected as the target from the cluster, excluding the OST currently to be isolated and those containing existing healthy copies. Next, the external coordinator calls the command `lfs mirror extend -N<temporary copy flag>--ost<target OST index><file path>`. This command creates a new mirror copy of the file on the selected target OST, marked with an internal system "temporary" flag, and synchronizes data from the healthy copy. This temporary flag indicates that the copy was created as an emergency measure due to a pre-verification failure. After the failed OST recovers and completes self-healing, the administrator or an automatic cleanup script can use this flag for subsequent processing, such as deleting or upgrading it to a permanent copy.

[0070] In this embodiment, after the external coordinator successfully acquires the exclusive DLM lock for the isolated slot file, it performs a replica redundancy pre-verification operation before issuing an inactive marking instruction to the Lustre kernel. This pre-verification operation verifies in advance whether the number of remaining healthy replicas of the corresponding file in the cluster meets the operational requirements of the read / write service after isolating the abnormal OST, preventing read / write service interruption due to a lack of available replicas caused by the isolation operation. First, the external coordinator obtains a list of all image files stored on the abnormal OST for which the exclusive lock has been acquired. This list is obtained by sending a query request to the MDS node, which stores information about all files stored on each OST. The external coordinator sends a file list query request to the MDS node, containing the unique identifier of the abnormal OST. Upon receiving the request, the MDS node retrieves the file information of all configured FLR image replicas stored on that OST, generates a list of image files, and returns it to the external coordinator. The list records information such as the file identifier, number of replicas, and replica distribution location of all image files stored on that OST.

[0071] It should be noted that after the external coordinator obtains the list of image files, it will perform a verification operation on the number of remaining healthy replicas for each image file in the list. The verification operation is achieved by querying the file's replica status information from the MDS node. The external coordinator sends a replica status query request for each image file to the MDS node in sequence. The MDS node returns the distribution location and running status information of all image replicas of the corresponding file. After isolating the abnormal OST, the external coordinator counts the number of remaining healthy replicas of the file. The number of remaining healthy replicas is the total number of healthy replicas of the file minus the number of replicas stored on the abnormal OST.

[0072] For example, a certain image file has three image copies, stored on three different OSTs. One copy is stored on the abnormal OST to be isolated, and the other two copies are stored on normally functioning OSTs. After isolating the abnormal OST, the number of remaining healthy copies of the file is two, which meets the operational requirements. If a certain image file has only one image copy, stored on the abnormal OST to be isolated, after isolating the abnormal OST, the number of remaining healthy copies of the file is zero, which does not meet the operational requirements.

[0073] Specifically, after the external coordinator completes the verification of the number of remaining healthy copies of all image files in the list, if the verification result shows that the number of remaining healthy copies of all image files is greater than or equal to one, it means that after isolating the abnormal OST, all files in the cluster retain at least one healthy image copy and can perform normal read and write operations. There will be no situation where no file has a usable copy. The external coordinator will then execute the subsequent inactive marking instruction issuance operation to complete the isolation process of the OST.

[0074] In this embodiment of the application, if the verification result shows that the number of remaining healthy copies of a mirror file is less than one, that is, the number of remaining healthy copies is zero, it means that after isolating the abnormal OST, the file will have no available healthy copies and cannot be read or written, which will cause the read and write service of the file to be interrupted. The external coordinator will not issue an inactive marking instruction, but will send a lock release instruction to the MGS node to release the DLM exclusive lock of the isolated slot file corresponding to the abnormal OST, mark the slot file as available, and trigger the temporary mirror copy creation process of the file with zero remaining healthy copies.

[0075] It should be noted that the temporary image replica creation process is implemented by calling Lustre's native image replica creation tool. The external coordinator sends a temporary replica creation command to the MDS node. The command contains the identifier of the corresponding file and the target OST information. The target OST is an OST that is running normally in the cluster and has available storage space. After receiving the command, the MDS node will create a new temporary image replica for the file on the target OST and synchronize the file data on the healthy replica to the temporary replica. After the temporary replica is created, the number of healthy replicas of the file is restored to the required value. The external coordinator will re-execute the replica redundancy pre-verification operation. After the verification passes, the subsequent isolation operation will be performed.

[0076] For example, a file is stored on an abnormal OST to be isolated. This file has only one mirror copy on that OST. The pre-verification result shows that the number of remaining healthy copies after isolation is zero. The external coordinator releases the exclusive lock on the isolated slot file and triggers a temporary copy creation process to create a temporary mirror copy for the file on another normally operating OST. After synchronization is complete, the file has two healthy copies. The external coordinator re-requests the exclusive lock and performs the pre-verification again. The verification result shows that the number of remaining healthy copies after isolation is one, which meets the requirements. The external coordinator issues an inactive flag instruction and performs the isolation operation.

[0077] Those skilled in the art will understand that by performing a replica redundancy pre-verification operation, the impact of the isolation operation on the file replica redundancy within the cluster can be verified in advance before the OST isolation operation is executed. This avoids the situation where multiple OSTs containing replicas of the same file simultaneously request isolation operations, leading to a zero file replica count, in extreme scenarios. This provides additional protection for cluster replica redundancy security and can prevent file read / write service interruptions caused by isolation operations. Furthermore, the pre-verification operation is implemented based on Lustre's native metadata query tools, requiring no additional development or deployment, resulting in high execution efficiency and no additional performance burden on the normal operation of the cluster.

[0078] After marking the target OST as inactive, the following steps are performed: Write the inactive status information and corresponding slot number of the target OST into the corresponding slot file on the MGS; The inactive status information of the target OST is synchronized to all MDS, OST and client nodes in the cluster through the MGS native configuration synchronization mechanism. Once feedback is received that the status synchronization of more than two-thirds of the nodes in the cluster is complete, the isolation operation is confirmed to be closed-loop, and the exclusive lock of the corresponding slot file is released.

[0079] In this embodiment, after the Lustre kernel marks the target OST as inactive, the external coordinator performs a cluster state synchronization operation. This operation synchronizes the inactive state information of the target OST to all nodes within the cluster, ensuring that all components in the cluster have a consistent understanding of the OST's operational status. This avoids discrepancies where some nodes have detected the OST's anomaly while others are still issuing read / write requests to it. First, the external coordinator writes the inactive state information of the target OST and the corresponding isolation slot number into the corresponding isolation slot file on the MGS node. The inactive state information includes the target OST's unique identifier, anomaly type, inactive marking time, and the node where the operation was executed. The slot number is the unique identifier of the isolation slot file, used to distinguish different isolation slot files. The write operation is performed through the MGS node's file read / write module. After the write is complete, the slot file will record all relevant information about this isolation operation for easy subsequent querying and auditing.

[0080] It's important to note that MGS nodes possess a native configuration synchronization mechanism. This mechanism is a built-in configuration information synchronization function of the Lustre file system, eliminating the need for additional synchronization components and enabling full cluster synchronization of cluster configuration information. When the content of the isolated slot file on the MGS node is updated, the native configuration synchronization mechanism is triggered. The MGS node will then use the cluster's internal communication network to broadcast the inactive status information of the target OST to all MDS nodes, OST nodes, and client nodes within the cluster, ensuring that all components within the cluster can obtain the latest operational status information of the OST.

[0081] Specifically, after receiving the inactive state information synchronized by the MGS node, each node in the cluster updates the node state information stored locally, marks the target OST as inactive, and returns feedback information to the MGS node indicating that the state synchronization is complete. The feedback information includes the node's unique identifier, the synchronization completion time, and other information. The MGS node collects the feedback information returned by each node, counts the number of nodes that have completed state synchronization, and synchronizes the statistical results to the external coordinator in real time.

[0082] In this embodiment, the external coordinator receives the statistical results of MGS node synchronization. When the number of nodes that have completed state synchronization exceeds two-thirds of the total number of nodes in the cluster, it confirms that the isolation operation loop is complete. The completion of the isolation operation loop indicates that the inactive state of the target OST has taken effect on the vast majority of nodes in the cluster, and almost no nodes in the cluster will issue new read / write requests to the target OST. The core process of the isolation operation has been completed. After confirming the completion of the isolation operation loop, the external coordinator sends a lock release command to the MGS node, releasing the DLM exclusive lock on the corresponding isolation slot file and marking the isolation slot file as available, so that it can be used by new isolation operation requests.

[0083] For example, there are 100 nodes in the cluster. After the MGS node synchronizes the inactive state information of the target OST to all nodes, it receives status synchronization completion feedback from 70 nodes. The number of nodes that have completed synchronization exceeds two-thirds of the total number of nodes. The external coordinator confirms that the isolation operation loop is complete, releases the DLM exclusive lock of the corresponding isolation slot file, and the slot file is restored to an available state.

[0084] It's important to note that the exclusive lock on the isolation slot file is only released after the isolation operation loop is complete and the vast majority of nodes in the cluster have synchronized their states. This prevents the next isolation operation from starting before the previous one has finished synchronizing its state, thus preventing cluster state chaos caused by multiple concurrent isolation operations. This ensures strict serial execution of isolation operations and further avoids over-isolation. Simultaneously, the cluster-wide state information synchronization is achieved through the native configuration synchronization mechanism of the MGS nodes. No additional synchronization components need to be developed or deployed; this is achieved based on Lustre's native functionality. This ensures consistent state information across all nodes in the cluster, preventing read / write request anomalies caused by state differences and improving cluster stability.

[0085] Those skilled in the art will understand that when more than two-thirds of the nodes in the cluster have completed state synchronization, it can be ensured that the vast majority of business access nodes in the cluster have updated the state information of the target OST and will no longer send read and write requests to the OST. This can guarantee the effectiveness of the isolation operation. At the same time, it does not need to wait for all nodes to complete synchronization, which can avoid the closed-loop delay of the isolation operation caused by some offline nodes, and balance the execution efficiency of the isolation operation with the coverage of state synchronization.

[0086] After the isolation operation is confirmed to be closed, the following steps are performed: After the exclusive lock on a slot file is released, the slot file is marked as available. According to the order of the isolation queue, apply for an exclusive lock on the slot file of the first abnormal OST in the queue that has not been isolated; Repeat the steps of replica redundancy pre-verification, inactive marking, and cluster state synchronization until all abnormal OSTs in the isolation queue have been isolated.

[0087] In this embodiment, after the external coordinator confirms the completion of the isolation operation loop and releases the DLM exclusive lock on the corresponding isolation slot file, it executes a serial isolation scheduling operation. This serial isolation scheduling operation sequentially completes the isolation operations for all abnormal OSTs in the isolation queue, gradually completing the isolation of all faulty nodes while limiting the number of concurrent isolations. First, after the DLM exclusive lock on the isolation slot file is released, the external coordinator marks the isolation slot file as available. This available status indicates that the slot file can be used by new isolation operations. The status marking operation is synchronously updated in the slot files of the MGS nodes, and all components within the cluster can query the latest status of the slot file.

[0088] It should be noted that the external coordinator will traverse the isolation queue according to the order of the previously generated isolation queue, locate the first abnormal OST in the queue that has not yet performed an isolation operation. This OST is the abnormal OST that is at the front of the queue and has not yet requested an exclusive lock. The external coordinator will send an exclusive lock request command to the MGS node to request an exclusive lock on an available isolation slot file for this OST. After receiving the request command, the MGS node will allocate the exclusive lock on the available isolation slot file to the OST and return a feedback message to the external coordinator that the lock request was successful.

[0089] Specifically, after the external coordinator successfully acquires an exclusive lock for the abnormal OST, it will repeat the complete steps of replica redundancy pre-verification, inactive marking, and cluster state synchronization. First, it will perform a replica redundancy pre-verification operation to check the number of remaining healthy replicas of all files after isolating the OST. After the verification passes, it will issue an inactive marking instruction to the Lustre kernel to mark the OST as inactive. Then, it will perform a cluster state synchronization operation to synchronize the inactive status information of the OST to all nodes in the cluster. After receiving synchronization completion feedback from more than two-thirds of the nodes, it will confirm that the isolation operation is complete, release the exclusive lock of the corresponding isolated slot file, and remark the slot file as available.

[0090] In this embodiment, the external coordinator continuously repeats the above process, performing lock acquisition, pre-verification, isolation marking, state synchronization, and lock release operations for each abnormal OST in the isolation queue according to the order of the isolation queue, until all abnormal OSTs in the isolation queue have completed the isolation operation and there are no remaining unprocessed abnormal OSTs in the queue. For example, there are eight abnormal OSTs in the isolation queue, and two isolation slot files are configured in the cluster. The external coordinator first performs isolation operations for the first two high-priority OSTs in the queue. After the isolation operation of the two OSTs is completed, the two slot files are released, and then the isolation operation is performed for the next two OSTs in the queue. This process is repeated, and the isolation operation of all eight abnormal OSTs in the queue is completed in four steps. Throughout the entire process, the number of OSTs in the isolation execution state at the same time never exceeds two, and the maximum number of concurrent isolations will not be exceeded.

[0091] It should be noted that the serial isolation scheduling operation can gradually complete the isolation processing of all abnormal OSTs under the premise of strictly limiting the number of concurrent isolations. It takes into account both the safety of cluster replica redundancy and the efficiency of fault handling, and avoids the problems of cluster state chaos and insufficient number of replicas caused by concurrent isolation operations. At the same time, it can complete the processing of all faulty nodes in the order of business priority, ensuring that high-priority services are restored first and low-priority services are processed in an orderly manner.

[0092] Those skilled in the art will understand that the execution flow of the serial isolation scheduling operation is fully integrated with operations such as isolation permission control, priority scheduling, pre-verification, and state synchronization, forming a complete multi-OST concurrent fault handling process. Regardless of the number of abnormal OSTs occurring simultaneously within the cluster, the isolation of all faulty nodes can be completed in an orderly manner while ensuring cluster security, preventing cluster read / write service interruptions caused by excessive isolation. Furthermore, the entire scheduling process requires no manual intervention and can automatically complete the isolation of all abnormal OSTs, improving the automation level of cluster fault handling and reducing the workload of operations and maintenance personnel.

[0093] After remarking OST as active, the following steps are performed: Synchronize the active status information of OST to the slot file of MGS and update the full-process audit log of the corresponding isolation operation; The active status information of OST is synchronized to all MDS, OST and client nodes in the cluster through the MGS native configuration synchronization mechanism. Verify the consistency of status information of all nodes in the cluster to complete the self-healing process closed loop.

[0094] In this embodiment, after the Lustre kernel marks the re-uploaded OST as active, the external coordinator performs state update and audit operations to complete the closed loop of the entire fault self-healing process. First, the external coordinator synchronizes the active status information of the OST to the corresponding isolation slot file on the MGS node. The active status information includes the OST's unique identifier, active marking time, incremental synchronization completion status, replica verification results, etc. The synchronization operation is performed through the file read / write module of the MGS node, which updates the status information of the OST recorded in the slot file and updates the full-process audit log of the corresponding isolation operation. The audit log records the full-process operation information of the OST from anomaly detection, isolation operation, incremental synchronization to re-upload, including the execution time, execution node, execution result, and related parameters of each operation. The audit log is persistently stored in the audit storage area of ​​the MGS node and can be used for subsequent cluster operation and maintenance, fault backtracking, operation auditing, and other scenarios.

[0095] It should be noted that after the external coordinator completes the information update of the slot file, it will use the native configuration synchronization mechanism of the MGS node to synchronize the active status information of the OST to all MDS nodes, OST nodes and client nodes in the cluster. The synchronization process is the same as the synchronization process of inactive status information. The MGS node sends the active status information to all nodes in the cluster in a broadcast manner. After receiving the status information, each node will update the node status information stored locally, re-mark the OST as active, restore the read and write request scheduling permissions of the OST, and return feedback information that the status synchronization is completed to the MGS node.

[0096] Specifically, after all nodes complete state synchronization, the external coordinator performs a state information consistency check operation on all nodes in the cluster. The check operation is achieved by comparing the state information of the OST stored on each node with the global state information stored on the MGS node. The external coordinator obtains the global state information from the MGS node and the state information of the OST stored locally on each node in the cluster, and compares them one by one. If the state information of all nodes is consistent with the global state information of the MGS node, the check passes. If the state information of any node is inconsistent with the global state information, the external coordinator will trigger the MGS node to resynchronize, resend the state information to the node with inconsistent state until the node completes the state update and the state information is consistent with the global information.

[0097] In this embodiment, after the state information consistency verification is passed, the external coordinator will confirm that the fault self-healing process of the OST is completed. The OST can participate in the cluster's read and write request scheduling normally. The request scheduling module of the Lustre kernel will restore the read and write task allocation permission of the OST and can issue new read and write operation instructions to the OST. The image copy stored on the OST can participate in file read and write operations and redundancy backup again, and the cluster will be restored to normal operation.

[0098] For example, after the OST completes incremental data synchronization, it is re-marked as active. The external coordinator updates the active status information of the OST to the slot file and audit log of the MGS node. The status information is synchronized to all nodes in the cluster through the MGS node. After verifying that the status information of all nodes is consistent, the self-healing process is confirmed to be closed loop. The OST is then rejoined to the cluster's read and write scheduling and can normally receive and execute read and write requests issued by the client.

[0099] It's important to note that by synchronizing and verifying status information, all nodes in the cluster can be assured of a consistent understanding of the OST's operational status. This prevents situations where some nodes have marked the OST as active while others still mark it as inactive, thus preventing read / write request scheduling anomalies caused by status differences. It ensures that the OST can participate normally in cluster read / write operations after being brought back online. Simultaneously, the updating of the full-process audit logs completely records all operational information related to fault self-healing, providing comprehensive data for subsequent cluster maintenance, fault analysis, and compliance audits, thereby improving the standardization of cluster operations.

[0100] Those skilled in the art will understand that the status update and auditing operations complete the final work of the fault self-healing process. They are interconnected with previous processes such as anomaly detection, replica switching, node isolation, and incremental synchronization, forming a complete closed loop from fault occurrence, service recovery, fault isolation to data recovery and node re-launch. The entire process can be completed automatically without manual intervention, improving the automated operation and maintenance capabilities and service availability of Lustre file system FLR image read and write scenarios.

[0101] This application also provides a replica consistency self-healing system to improve the high availability of Lustre images, such as... Figure 2 As shown, it includes: 01 Enhanced Client Module, 02 External Coordinator Module, 03 Incremental Repair Service Module, and 04 MGS Global Management Node; The enhanced client module is used to receive read and write requests initiated by business applications for the primary replica of Lustre files. After the request triggers a timeout or I / O exception and the retry fails, it queries the storage layout information of the file from MDS, selects healthy replicas as the new primary replicas, retryes the request, and returns the execution result to the business application. The external coordinator module is used to perform online status detection of OSTs according to a preset period, issue status marking instructions to the Lustre kernel for abnormal OSTs, apply for exclusive locks of isolation slot files from the MGS global management node, and perform business priority scheduling and pre-verification of replica redundancy before isolation of abnormal OSTs. The incremental repair service module is used to extract change log entries corresponding to the fault range from MDS after an OST marked as inactive is brought back online, parse and generate a list of change files, and call the Lustre native image recovery tool to perform incremental data synchronization. The MGS global management node is used to store isolated slot files and cluster configuration information, and synchronizes OST status information to all nodes in the cluster through the native configuration synchronization mechanism.

[0102] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for improving replica consistency and self-healing in high-availability scenarios of Lustre images, applied to Lustre file system FLR image scenarios, wherein FLR is Lustre's built-in file-level redundancy feature used to create synchronous image replicas of a single file across multiple OSTs, wherein OST is Lustre's object storage target, characterized in that... include: The client receives read / write requests from business applications and sends them to the corresponding OST of the primary replica. If the request triggers a timeout or I / O exception and fails to retry, the client determines that the OST is abnormal, queries the MDS for the file storage layout, selects a healthy replica as the new primary replica, and retryes the request. The MDS is Lustre's metadata service node. The external coordinator checks the online status of the OST at a preset period. When an anomaly is detected, it sends an inactive flag instruction to the kernel through Lustre native tools. The kernel then terminates the pending I / O requests of the OST and triggers client replica switching. After detecting that an inactive OST has come back online, extract the change log of the faulty area to generate a file list, call the Lustre native image recovery tool to perform incremental synchronization, and mark the OST as active after completion.

2. The method according to claim 1, characterized in that, Before the external coordinator issues the inactive flag instruction to the Lustre kernel, it performs the following steps: The external coordinator receives I / O exception events reported by the client and performs online status supplementary checks on the corresponding OST based on the exception events; By combining the periodic detection results with the supplementary detection results, the abnormal state of the target OST is confirmed.

3. The method according to claim 1, characterized in that, During the process of extracting the change log of the fault range and generating the file list, the change log entries are deduplicated based on the file identifier. Only the latest entry is retained among multiple change entries in the same file. The file list is generated based on the deduplicated entries. After performing incremental synchronization, a copy verification and comparison is performed on the synchronized files to confirm that the copy on the re-uploaded OST is consistent with the healthy copy data.

4. The method according to claim 2, characterized in that, Before the external coordinator issues the inactive flag instruction, the following steps are performed: Based on the minimum replica redundancy of the Lustre cluster FLR image, calculate the maximum number of concurrent isolations N, where N is the minimum replica redundancy minus one. Create a fixed number of N isolation slot files on the MGS node of the Lustre cluster, and configure a DLM exclusive lock for each slot file. The MGS is the global management node of the Lustre cluster, and the DLM is the native distributed lock manager of the Lustre kernel, which is used to ensure that a slot file can only be held by one isolation operation at the same time. For OSTs that are confirmed to be abnormal, request an exclusive lock on the slot file from MGS. Only after the exclusive lock is successfully requested will an inactive flag command be issued.

5. The method according to claim 4, characterized in that, Before requesting an exclusive lock for the slot file from MGS, the following steps are performed: Based on the directory-level FLR strategy of the Lustre cluster, a service priority is marked for each abnormal OST; Create isolated queues for abnormal OSTs in descending order of business priority; In the order of the isolation queue, the abnormal OST requests an exclusive lock on the slot file in turn; If all slot files are occupied by low-priority abnormal OSTs, and there are high-priority abnormal OSTs in the queue, then release the slot files occupied by the low-priority OSTs and allocate slot files for the high-priority OSTs.

6. The method according to claim 4, characterized in that, After the exclusive lock application is successful but before the inactive flag instruction is issued, the following steps are performed: Retrieve a list of all image files stored on the exception OST that acquired the exclusive lock; For each image file in the list, verify the number of remaining healthy copies of the file after isolating the abnormal OST; If the number of remaining healthy copies of all image files is greater than or equal to one, then an inactive marking instruction is issued. If the number of remaining healthy copies of an image file is less than one, the exclusive lock on the corresponding slot file is released, triggering the creation process of a temporary image copy of that file.

7. The method according to claim 2, characterized in that, After marking the target OST as inactive, perform the following steps: Write the inactive status information and corresponding slot number of the target OST into the corresponding slot file on the MGS; The inactive status information of the target OST is synchronized to all MDS, OST and client nodes in the cluster through the MGS native configuration synchronization mechanism. Once feedback is received that the status synchronization of more than two-thirds of the nodes in the cluster is complete, the isolation operation is confirmed to be closed-loop, and the exclusive lock of the corresponding slot file is released.

8. The method according to claim 7, characterized in that, After confirming that the isolation operation has been completed, perform the following steps: After the exclusive lock on a slot file is released, the slot file is marked as available. According to the order of the isolation queue, apply for an exclusive lock on the slot file of the first abnormal OST in the queue that has not been isolated; Repeat the steps of replica redundancy pre-verification, inactive marking, and cluster state synchronization until all abnormal OSTs in the isolation queue have been isolated.

9. The method according to claim 3, characterized in that, After remarking OST as active, perform the following steps: Synchronize the active status information of OST to the slot file of MGS and update the full-process audit log of the corresponding isolation operation; The active status information of OST is synchronized to all MDS, OST and client nodes in the cluster through the MGS native configuration synchronization mechanism. Verify the consistency of status information of all nodes in the cluster to complete the self-healing process closed loop.

10. A replica consistency self-healing system for improving the high availability of Lustre images, characterized in that, include: Enhance the client module, external coordinator module, incremental repair service module, and MGS global management node; The enhanced client module is used to receive read and write requests initiated by business applications for the primary replica of Lustre files. After the request triggers a timeout or I / O exception and the retry fails, it queries the storage layout information of the file from MDS, selects healthy replicas as the new primary replicas, retryes the request, and returns the execution result to the business application. The external coordinator module is used to perform online status detection of OSTs according to a preset period, issue status marking instructions to the Lustre kernel for abnormal OSTs, apply for exclusive locks of isolation slot files from the MGS global management node, and perform business priority scheduling and pre-verification of replica redundancy before isolation of abnormal OSTs. The incremental repair service module is used to extract change log entries corresponding to the fault range from MDS after an OST marked as inactive is brought back online, parse and generate a list of change files, and call the Lustre native image recovery tool to perform incremental data synchronization. The MGS global management node is used to store isolated slot files and cluster configuration information, and synchronizes OST status information to all nodes in the cluster through the native configuration synchronization mechanism.

Images