Data reading method and system for distributed parallel file system mirror copy

Abstract

The application provides a data reading method and system for a mirror copy of a distributed parallel file system. When a primary copy fails, the application first intercepts errors to avoid direct business interruption, converts the failure signal into a trigger condition for automatic recovery, and then interacts with a metadata server to complete the degradation of the failed copy and the promotion of the healthy copy, update the global metadata view of the file, avoid subsequent requests from continuously accessing the failed node, seamlessly redirect the read request to the healthy copy to complete data reading, the entire process is not perceived by the upper layer application, millisecond-level self-healing of the failure is achieved, the access capability of the redundant copy is released, and the business interruption problem caused by the single-point failure of the primary copy is solved.

Description

Technical Field

[0001] This application belongs to the field of data reading and writing, and in particular relates to a method and system for reading data from a distributed parallel file system mirror copy. Background Technology

[0002] The existing Lustre file system (distributed parallel file system) uses an asynchronous replication mechanism for file-level redundancy. Write operations only update the primary replica, leaving a vulnerable window for data access before the standby replica is synchronized. When the primary replica fails, client read requests will directly return I / O errors to the upper-layer application, lacking built-in automatic failover capabilities. It can only rely on external tools to complete replica synchronization and recovery, resulting in long recovery cycles and low automation. At the same time, the standby replica cannot be accessed due to status marking restrictions, leaving redundant storage resources idle and making it more prone to business interruptions, which violates the high availability design goal of data redundancy configuration. Summary of the Invention

[0003] The technical problem to be solved is: how to solve the problem of upper-layer business read interruption caused by a single point of failure of the primary replica while adhering to the Lustre native architecture specification and meeting the basic data persistence requirements of the Mirror (replica) file-level redundancy feature, and realize automatic and imperceptible switching of read requests and system self-healing in failure scenarios, so that the redundant replica can truly play a high availability role.

[0004] This application provides a method for reading data from a distributed parallel file system mirror copy, applied to a node client, the method comprising: The node client receives read requests from the upper-layer application, initiates a read operation on the primary replica of the Mirror file with file-level redundancy enabled, intercepts error messages returned when the read operation fails, and triggers an automatic recovery process. The node client interacts with Lustre's metadata server to complete the state degradation of the failed primary replica and the state promotion of the healthy standby replica; The node client obtains the latest file layout information returned by the metadata server, redirects the read request to the healthy replica with the status of init, and completes the data read.

[0005] Optionally, the interception of error messages returned during failed read operations and the triggering of an automatic recovery process include: The node client determines the failure type of the read operation and filters out error messages that are not replica failure types; The node client uses the filtered replica failure error messages as a trigger signal for the automatic recovery process, locking the layout information of the target Mirror file; The node client suspends initiating new read operations to the failed primary replica.

[0006] Optionally, the node client determines the failure type of the read operation and filters error messages that are not replica failures, including: The node client extracts the error codes and fault context information returned by the read operation; The node client will match the extracted information with a pre-defined database of replica fault characteristics; The node client will directly return the non-replica failure error information that failed to match to the upper layer application, terminating the automatic recovery process.

[0007] Optionally, the method is applied to a distributed network cluster composed of multiple node clients. After the metadata server completes the replica state change, it generates a write-ahead log event and broadcasts the global ID bitmap of the event only to all node clients in the cluster. The node clients complete the P2P synchronization of the event content based on the minimum reachable closed loop maintained by themselves. The write-ahead log event is an immutable event unit generated by the metadata server after completing the Mirror replica state change, containing a globally unique event ID, a file identifier, and a snapshot of the replica state change. The global ID bitmap is a lightweight message that only records the global ID and hash value of the write-ahead log event. The minimum reachable closed loop is a closed link formed by a single node client as the source node client, based on the P2P direct connection link between node clients, which can return to the source node client from the source node client through the minimum number of intermediate node clients.

[0008] Optionally, the node client completes P2P synchronization of event content based on its own maintained minimum reachable closed loop, including: The node client searches for active node clients within the same network segment that have opened the same Mirror file and establishes a P2P direct connection link. The node client completes the self-organized construction and two-way confirmation binding of the minimum reachable closed loop based on the P2P direct connection link; The node client completes the complementary synchronization of write-ahead log event content and the maintenance of the aggregation list within the minimum reachable closed loop.

[0009] Optionally, the node client completes the self-organized construction and two-way confirmation binding of the minimum reachable closed loop based on the P2P direct connection link, including: The node client, acting as the source node client, sends cycloprobe messages with hop count limits to all directly connected neighbor node clients; After receiving the probe message, other node clients acting as intermediate node clients update the list of path node clients and forward it to their directly connected neighbors that are not the source node clients. The source node client selects the link with the fewest node clients from the returned closed links as the minimum reachable closed loop, sends a loop binding request to the node clients within the minimum reachable closed loop, and completes bidirectional confirmation.

[0010] Optionally, the node client completes the complementary synchronization and aggregation list maintenance of the pre-written log event content within the minimum reachable closed loop, including: the node client receives the global ID bitmap, compares it with the local event ID, and generates an independent event missing bitmap; The node client sends the missing event bitmap to all node clients within its smallest reachable closed loop. After receiving the missing bitmap, the node client within the minimum reachable closed loop compares it with its own event list and returns the corresponding missing write-ahead log event content in batches to the requesting node client. After requesting the node client to verify the event content, update the local event information and synchronously update the event aggregation list within the circle.

[0011] Optionally, the node client detects the liveness status of the node client within the minimum reachable closed loop through Lustre's native Lnet heartbeat mechanism. When a node client failure is detected, it triggers automatic reconstruction of the standard closed loop and synchronization of the event list, including: The remaining node clients within the circle pause cross-circle event synchronization and re-execute the minimum reachable closed-loop circle detection process based on the remaining node clients. The remaining node clients negotiate to add new node clients, restoring the standard closed-loop size to the preset number of nodes; After the restored standard closed loop completes two-way confirmation binding, all node clients within the minimum reachable closed loop broadcast their own event lists, and generate a new clustered event list within the loop through complementary synchronization.

[0012] Optionally, if a node client fails to find the target write-ahead log event within its minimum reachable closed loop, it triggers a cross-loop query process, including: The node client determines the node client with the fewest overlapping nodes with the minimum reachable closed loop of the queried node client from among the node clients contained in its own minimum reachable closed loop, and uses this node client as the cross-loop jump node client; The node client sends a query request for the target event and a missing bitmap to the cross-circle jump node client; The jump node client completes the event query within the minimum reachable closed loop, returns the matching event content to the requesting node client, or continues to forward the query request to the next jump node client according to the same rules.

[0013] This application also provides a data reading system for a distributed parallel file system mirror copy, including a node client and a metadata server; The node client receives read requests from the upper-layer application, initiates a read operation on the primary replica of the Mirror file with file-level redundancy enabled, intercepts error messages returned when the read operation fails and triggers an automatic recovery process; interacts with Lustre's metadata server to complete the state degradation of the failed primary replica and the state promotion of the healthy standby replica; obtains the latest file layout information returned by the metadata server, redirects the read request to the healthy replica with the state of init and completes the data read.

[0014] As can be seen from the above technical solution, when the primary replica fails, this application first intercepts the error to avoid direct business interruption, transforms the fault signal into a trigger condition for automatic recovery, and then completes the degradation of the faulty replica and the promotion of the healthy replica through interaction with the metadata server, updates the global metadata view of the file, avoids subsequent requests from continuously accessing the faulty node, and seamlessly redirects read requests to the healthy replica to complete data reading. The whole process is imperceptible to the upper layer application, realizes millisecond-level self-healing of faults, releases the access capability of redundant replicas, and solves the problem of business interruption caused by single point of failure of the primary replica. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This paper illustrates a flowchart of a data reading method for a distributed parallel file system image copy according to an embodiment of this application. Figure 2 A flowchart illustrating one specific example of an embodiment of this application is shown; Figure 3 The second flowchart of a specific example in an embodiment of this application is shown; Figure 4 A schematic diagram illustrating the principle of the minimum reachable closed loop in a specific example of an embodiment of this application is shown. Detailed Implementation

[0017] In this application embodiment, the implementation scenario of the prior art is first described in detail to clarify the actual scenario source of the technical problem to be solved by this application. For example, in the application scenario of the existing Lustre file system, an enterprise-level big data analysis cluster is configured with 2 metadata servers, 8 object storage servers, and 60 client computing nodes. All user call detail record (CDR) data files within the cluster have file-level redundancy enabled, configured with 1 primary and 2 backup copies (3 replicas in total). During file writing, only the object storage server where the primary replica resides receives and updates the written data; the two backup replicas are marked as invalid, and data synchronization is performed via external tools after the writing is completed. Within the time window before the data writing is completed and the backup replicas are synchronized, the object storage server where the primary replica resides experiences a disk failure. Read operations initiated by the client to the primary replica directly return input / output errors, and the upper-layer data analysis task is directly interrupted. Maintenance personnel need to manually modify the replica status and synchronize the data using external tools. The entire recovery process takes tens of minutes, during which all read operations on the file cannot be performed normally. Simultaneously, the object storage servers where the two backup replicas reside are running normally but cannot be accessed by clients, rendering the redundant storage resources within the cluster ineffective.

[0018] The existing technologies described above expose several problems with the file-level redundancy features of the Lustre file system. Firstly, a single point of failure in the primary replica directly interrupts upper-layer business read operations, lacking a built-in automatic fault handling process. Secondly, the state limitations of the backup replica prevent the effective utilization of redundant storage resources. Thirdly, the recovery process, reliant on external tools, suffers from significant delays and low automation. Fourthly, in large-scale cluster scenarios with multiple concurrent clients, the metadata server needs to broadcast replica state change information to all clients, easily leading to performance bottlenecks on the metadata server and inconsistent replica state views among multiple clients, resulting in duplicate and invalid access operations. Based on these problems, this application provides corresponding technical solutions, the details of which are described below.

[0019] This application provides a method for reading data from a distributed parallel file system mirror copy. This method is applied to a node client, which is a computing node in the Lustre file system that mounts the cluster and initiates file access operations. The node client runs the Lustre file system client kernel module and can interact with a metadata server for metadata and with an object storage server for file data read / write interactions. Specifically, the method includes three main execution steps, such as... Figure 1 As shown: S1: The node client receives the read request from the upper layer application, initiates a read operation on the primary replica of the Mirror file with file-level redundancy enabled, intercepts the error message returned when the read operation fails, and triggers the automatic recovery process.

[0020] S2: The node client interacts with Lustre's metadata server to complete the state degradation of the failed primary replica and the state promotion of the healthy standby replica.

[0021] S3: The node client obtains the latest file layout information returned by the metadata server, redirects the read request to the healthy replica with the status of init, and completes the data read.

[0022] When the primary replica fails, this application first intercepts the error to prevent direct business interruption, transforms the failure signal into a trigger condition for automatic recovery, and then completes the degradation of the failed replica and the promotion of the healthy replica through interaction with the metadata server. It updates the global metadata view of the file to prevent subsequent requests from continuously accessing the failed node, and seamlessly redirects read requests to the healthy replica to complete data reading. The entire process is imperceptible to the upper-layer application, achieving millisecond-level self-healing of the failure, releasing the access capabilities of redundant replicas, and solving the problem of business interruption caused by a single point of failure of the primary replica.

[0023] The following is combined with Figures 2 to 4 The embodiments of this application will be described in detail.

[0024] It should be noted that the Mirror file with enabled file-level redundancy in this embodiment refers to a file configured with multiple data copies in the Lustre file system using file-level redundancy. Each data copy of the file is stored in a different object storage target on a different object storage server, and each copy corresponds to an independent storage location and status identifier. The primary copy is the default target copy for file write operations. During file write operations, only the primary copy updates the data content in real time; the other copies are backup copies. Backup copies are marked as stale before data synchronization is complete, and as init after data synchronization is complete. Replicas in the init state are data-complete and accessible, while replicas in the stale state are data-incomplete and inaccessible by default.

[0025] For example, a Mirror file is configured with three replicas: replica 0, replica 1, and replica 2. Replica 0 is set as the primary replica. Initially, all three replicas are synchronized and in the init state. When an upper-layer application initiates a write operation to this file, the node client only sends the write data to the object storage server where replica 0 resides. After replica 0 completes the data update, replicas 1 and 2 are automatically marked as stale. Once the background synchronization tool completes the data synchronization between replicas 1, replica 2, and replica 0, the states of replicas 1 and 2 are restored to init.

[0026] Specifically, the node client receives read requests from upper-layer applications, including but not limited to big data analysis tasks, high-concurrency data reading services, file archiving services, and other business programs running on the node client. The read requests generated by the upper-layer applications contain information such as the target file path, read offset, and read length. After receiving the read request, the Lustre kernel module of the node client retrieves the layout information of the target file from the local cache based on the file identifier of the target file. The layout information contains the object storage target, network address, replica status, replica sequence number, and other information corresponding to all replicas of the file. The node client determines the primary replica of the Mirror file based on the layout information and initiates a read operation to the object storage server where the primary replica is located.

[0027] It should be noted that when a node client initiates a read operation to the primary replica, it sends a read request message to the corresponding object storage server through the PTLRPC communication framework of the Lustre file system. Upon receiving the read request, the object storage server performs the corresponding disk read operation and returns a response message to the node client. There are two types of response messages: one is a successful read response message, which carries the read file data. Upon receiving this message, the node client returns the data to the upper-layer application, completing the read process. The other is a failed read response message, which carries the corresponding error code and fault-related context information. Upon receiving this message, the node client executes the error interception and automatic recovery process in this embodiment of the application, rather than directly returning the error information to the upper-layer application.

[0028] Specifically, the node client intercepts the error information returned by the read operation failure and triggers the automatic recovery process. This process includes three sub-steps. The first sub-step is for the node client to determine the failure type of the read operation and filter out error information that is not a replica failure. The second sub-step is for the node client to use the filtered replica failure error information as the trigger signal for the automatic recovery process and lock the layout information of the target Mirror file. The third sub-step is for the node client to pause initiating new read operations to the failed primary replica.

[0029] It should be noted that read operation failures are divided into two categories. One category is replica failure errors, which are caused by failures in the object storage server where the primary replica resides or the object storage target. These errors include, but are not limited to, disk failures on the object storage server, network unavailability, offline status of the object storage target, and data read verification failures. These errors will prevent the primary replica from providing normal read services and require triggering an automatic recovery process. The other category is non-replica failure errors, which are unrelated to the running status of the primary replica. These errors include, but are not limited to, invalid read request parameters from the upper-layer application, insufficient file permissions, failure to allocate local memory on the node client, and disconnection of the mount session between the client and the cluster. These errors cannot be resolved by replica switching. Therefore, these errors need to be filtered out, and the automatic recovery process should not be triggered. Instead, error information should be returned directly to the upper-layer application.

[0030] For example, after receiving a read failure response message, the node client extracts the error code from the message. If the object storage target corresponding to the error code is inaccessible, the error is determined to be a replica failure error, and the automatic recovery process is initiated. If the file permissions corresponding to the error code are insufficient, the error is determined to be a non-replica failure error, and the error information is directly returned to the upper-layer application without triggering the automatic recovery process.

[0031] Specifically, the node client uses the filtered replica failure error messages as the trigger signal for the automatic recovery process, locking the layout information of the target mirror file. It's important to note that locking the target mirror file's layout information means the node client performs a write-protection operation on the file's layout information locally. Before the automatic recovery process is complete, other local processes are prohibited from modifying the file's layout information. Simultaneously, new read operations based on the existing layout information are suspended, preventing duplicate and invalid read requests to the failed primary replica based on invalid layout information, thus reducing unnecessary network interactions and I / O overhead.

[0032] Specifically, the node client suspends initiating new read operations to the failed primary replica. It's important to note that the node client generates a marker for the failed primary replica locally. This marker includes the object storage target identifier, failure time, and error type. Before the automatic recovery process is complete, when the node client receives a new read request for this file, it will first match this fault marker and will not initiate new read requests to the primary replica already marked as failed, thus avoiding invalid network requests consuming network bandwidth and node processing resources.

[0033] In this embodiment, the node client determines the failure type of the read operation and filters out error information of non-replica fault types. This process includes three sub-steps: the first sub-step is for the node client to extract the error code and fault context information returned by the read operation; the second sub-step is for the node client to match the extracted information with a preset replica fault feature library; and the third sub-step is for the node client to directly return the non-replica fault type error information that failed to match to the upper layer application and terminate the automatic recovery process.

[0034] It should be noted that the error codes returned by the read operation are standard error codes defined in the Lustre file system. Each error code corresponds to a fixed error type. The fault context information includes, but is not limited to, the object storage target identifier corresponding to the read request, the object storage server network address, the request sending time, the timeout period, and additional fault description information returned by the server. When the node client extracts the error codes and fault context information, it parses the fields from the read failure response message according to a preset message format, extracts the corresponding field content, and temporarily stores it locally for subsequent fault type matching.

[0035] Specifically, the node client matches the extracted information against a pre-configured replica fault feature library. It's important to note that the replica fault feature library is a pre-configured feature set on the node client's local machine. This feature set contains error code values ​​and fault context feature rules for all corresponding replica fault classes. The feature library can be updated uniformly through cluster configuration tools or manually modified locally on the node client. During the matching process, the node client first compares the extracted error code with the error code set in the feature library. If the error code is not in the feature library, the match is considered a failure. If the error code is in the feature library, the fault context information is further matched against the corresponding feature rules in the feature library to verify whether the error was caused by a replica fault. If the verification passes, the match is considered successful; otherwise, the match is considered a failure.

[0036] For example, the replica fault feature library contains error codes corresponding to the object storage server being unreachable from the network. After the node client extracts the error code, it further extracts the network address of the object storage server from the fault context information and performs a network connectivity test. If the test result is that the network is unreachable, the verification passes and the match is determined to be successful; if the test result is that the network is reachable, the verification fails and the match is determined to be unsuccessful.

[0037] Specifically, the node client directly returns non-replica failure error messages that failed to match to the upper-layer application, terminating the automatic recovery process. It's important to note that a failed match indicates the read failure was not caused by a replica failure and cannot be resolved through replica switching. Therefore, the node client terminates the automatic recovery process, encapsulates the original error message in a standard format, and returns it to the upper-layer application that initiated the read request for further error handling. If the match is successful, it means the read failure was caused by a replica failure, and the node client continues with the subsequent steps of the automatic recovery process.

[0038] In this embodiment, after the node client triggers the error interception and automatic recovery process, it interacts with the Lustre metadata server to complete the state degradation of the failed primary replica and the state promotion of the healthy standby replica. It should be noted that the metadata server is the service node in the Lustre file system responsible for managing file metadata. It stores metadata information for all files in the cluster, including file layout information, replica status, access permissions, file size, modification time, etc. The metadata server can receive metadata modification requests sent by the node client, complete the atomic update of the corresponding file metadata, and return the updated metadata information to the node client.

[0039] Specifically, when a node client completes the state degradation of a failed primary replica, it sends a replica state degradation request to the metadata server. The request includes the file identifier of the target file, the replica identifier of the failed primary replica, and the target state "stale." Upon receiving the request, the metadata server verifies the node client's permissions. If the verification is successful, the server changes the state of the failed primary replica in the file's metadata from "init" to "stale" and persists the modified metadata information to the local disk, completing the state degradation operation. It's worth noting that downgrading a failed primary replica to "stale" marks it as no longer providing services at the metadata level, preventing other node clients in the cluster from continuing to initiate read requests to the failed replica and reducing unnecessary network requests within the cluster.

[0040] Specifically, the process of a node client promoting a healthy standby replica to the desired state involves the node client sending a replica state promotion request to the metadata server. This request includes the file identifier of the target file, the replica identifier of the healthy standby replica to be promoted, and the target state "init". Upon receiving the request, the metadata server verifies the node client's permissions. If the verification is successful, the server changes the state of the corresponding standby replica in the file's metadata from "stale" to "init" and persistently stores the modified metadata information on the local disk, thus completing the healthy standby replica state promotion operation. It's important to note that promoting a healthy standby replica to the "init" state removes access restrictions on that replica, allowing node clients to initiate read operations normally, ensuring that redundant storage resources within the cluster can be utilized effectively, and providing data read services to upper-layer applications.

[0041] It should be noted that the node client can determine the healthy standby replica to be promoted from all standby replicas of the target file according to a preset selection strategy. The preset selection strategy includes, but is not limited to, selecting the standby replica with the smallest replica identifier, selecting the standby replica with the lowest load on the corresponding object storage server, selecting the standby replica with the lowest network latency with the node client, and selecting the standby replica with the highest data synchronization completion rate. For example, after obtaining information on all standby replicas of the target file, the node client sends network probe packets to the object storage server where each standby replica resides, calculates the network round-trip latency of each object storage server, selects the standby replica with the lowest network round-trip latency as the healthy standby replica to be promoted, and sends a status promotion request to the metadata server.

[0042] In this embodiment, after the node client completes the state downgrade of the failed primary replica and the state promotion of the healthy standby replica, it obtains the latest file layout information returned by the metadata server, redirects the read request to the healthy replica with the state "init," and completes the data read. It should be noted that after modifying the replica state, the metadata server encapsulates the latest layout information of the target file in a response message and returns it to the requesting node client. The latest layout information includes the latest state of all replicas of the file, the corresponding object storage target information, network address, etc. Upon receiving the latest layout information, the node client updates the file layout information in its local cache, releases the previous lock on the layout information, and, based on the latest layout information, determines all healthy replicas with the state "init." It then redirects previously failed read requests, as well as subsequent new read requests for the file, to the object storage server where the healthy replica with the state "init" resides, initiates a new read operation, obtains the corresponding file data, and returns it to the upper-layer application.

[0043] For example, if a node client's read operation to the primary replica fails, it interacts with the metadata server to downgrade the failed primary replica to a stale state and promote the standby replica with the least network latency to an init state. The metadata server returns the latest layout information, and the node client, based on the latest layout information, redirects the read request to the object storage server where the newly promoted standby replica in the init state resides, initiates a new read operation, and after receiving the returned file data, returns the data to the upper-layer big data analysis task. Throughout the entire process, the upper-layer business only perceives the brief latency of the read operation, does not receive any error information, and the business operation is not interrupted.

[0044] Furthermore, in large-scale cluster scenarios with multiple concurrent clients, the automatic failover solution based on LustreMirror read scenarios requires the metadata server to broadcast the write-ahead log event content to all clients in the cluster after completing the replica state change. This easily leads to saturation of the metadata server's egress bandwidth, a surge in CPU utilization, and delays or even freezes in the metadata service response. Simultaneously, client event synchronization relies on centralized broadcasting, which is prone to interruption when node network jitter occurs, causing inconsistencies in the replica state views across multiple clients, resulting in duplicate or invalid failovers. The solution suffers from severe scalability and reliability limitations in large-scale clusters. To address this issue, while meeting the core functional requirements of automatic failover in LustreMirror read scenarios and ensuring global consistency of the replica state views across all clients in the cluster, the solution aims to completely offload the metadata server from the event content broadcasting processing burden, resolve the scalability bottleneck in large-scale multi-client cluster scenarios, and improve the reliability of event synchronization and overall system stability in a distributed cluster environment.

[0045] In the preferred scheme, this approach utilizes the collaboration between the metadata server broadcasting only a global ID bitmap and the client-side P2P (Peer-to-Peer) event synchronization based on a minimum reachable closed loop. This significantly reduces the amount of broadcast data on the metadata server, offloading its event content distribution processing pressure. Then, through the minimum reachable closed loop self-organized by the client, event content synchronization is transformed from centralized broadcasting to distributed neighborhood complementary synchronization. The high connectivity of the closed link achieves low latency and high reliability sharing of events, while ensuring the consistency of the state view of all client replicas across the cluster, avoiding invalid failover issues. This scheme has stronger horizontal scalability and operational stability in large-scale clusters.

[0046] The preferred embodiments of this application will be described in detail below.

[0047] In this embodiment, the data reading method for the distributed parallel file system mirror replica described above can be applied to a distributed network cluster composed of multiple node clients. The cluster includes multiple node clients, at least one metadata server, and multiple object storage servers. All node clients mount the same Lustre file system and can access the mirror files within the cluster. In this distributed network cluster application scenario, after the metadata server completes the replica state change, it generates a write-ahead log event and broadcasts the global ID bitmap of the event only to all node clients in the cluster. The node clients complete the P2P synchronization of the event content based on the minimum reachable closed loop they maintain.

[0048] It's important to note that write-ahead log events are immutable event units generated by the metadata server after completing a Mirror replica state change. These units contain a globally unique event ID, a file identifier, and a snapshot of the replica state change. The file identifier is a globally unique file identifier for each file in the Lustre file system. The replica state change snapshot includes the replica identifier, state information, and corresponding object storage target information for all replicas before and after the state change. The globally unique event ID is a globally monotonically increasing numeric identifier generated by the metadata server for each replica state change. Each write-ahead log event corresponds to a unique globally unique event ID, which cannot be repeated or modified. The metadata server generates a corresponding write-ahead log event each time it completes a Mirror file replica state change operation. This event records complete information about the replica state change and serves as the baseline information for all node clients in the cluster to update their local file layout cache.

[0049] It's important to note that the Global ID Bitmap is a lightweight message that only records the global ID and hash value of write-ahead log events. The Global ID Bitmap uses a bitmap structure to record the globally unique ID of each generated write-ahead log event. Simultaneously, it stores the hash value corresponding to each event ID. The hash value is a fixed-length value generated by the metadata server using a hash algorithm based on the complete content of the write-ahead log event, used by subsequent node clients to verify the integrity of the event content. Because the Global ID Bitmap only records the event ID and hash value, and does not contain the complete content of the write-ahead log event, the message size is small. The message length of the Global ID Bitmap corresponding to 1000 write-ahead log events does not exceed 150 bytes. When the metadata server broadcasts this bitmap to the entire cluster, it does not generate a large amount of network traffic, nor does it place a significant processing burden on the metadata server.

[0050] It should be noted that the minimum reachable closed loop is a closed link formed by a single node client as the source node client, based on the P2P direct connection between node clients, that returns to the source node client from the source node client through the minimum number of intermediate node clients. The minimum reachable closed loop is the core unit for synchronizing write-ahead log events among node clients within the cluster. Each node client maintains its own corresponding minimum reachable closed loop. The standard closed loop size is 3 to 4 node clients, which ensures the reachability and fault tolerance of the link, while avoiding the problem of reduced event synchronization efficiency caused by an excessively large standard closed loop size.

[0051] In this embodiment, the node client completes P2P synchronization of event content based on its own minimum reachable closed loop. This process includes three sub-steps: the first sub-step is for the node client to find active node clients in the same network segment that have opened the same Mirror file and establish a P2P direct connection link; the second sub-step is for the node client to complete the self-organized construction and bidirectional confirmation binding of the minimum reachable closed loop based on the P2P direct connection link; and the third sub-step is for the node client to complete the complementary synchronization of pre-written log event content and the maintenance of the aggregation list within the minimum reachable closed loop.

[0052] Specifically, the node client searches for active node clients within the same network segment that have opened the same Mirror file and establishes a P2P direct connection. It's important to note that when a node client opens a target Mirror file, it sends a request to the metadata server to obtain the network address information of all node clients in the current cluster that have opened the same Mirror file. The metadata server maintains a list of open sessions for each file, recording the identifier, network address, session status, and other information of all node clients that have opened the file. The node client can obtain this list through a standard metadata request. After obtaining the list, the node client selects node clients within the same network segment as its neighboring node clients to connect to. Based on the native Lnet (Lustre Network) network layer of the Lustre file system, a P2P direct connection is established with the selected neighboring node clients. This direct connection reuses Lustre's native PTLRPC communication framework, requiring no new network connections and is fully compatible with the native communication mechanism of the Lustre file system, avoiding compatibility issues.

[0053] For example, after node client A opens the target Mirror file, it requests a list of open sessions for the file from the metadata server. It finds that a total of 80 node clients in the cluster have opened the file. Node client A selects 12 node clients that are in the same rack network segment as itself from the list and establishes P2P direct connection links with these 12 node clients through the Lnet network layer for subsequent standard closed loop construction and event synchronization interaction.

[0054] Specifically, the node client completes the self-organized construction and bidirectional confirmation binding of the minimum reachable closed loop based on the P2P direct link. This process includes three sub-steps. The first sub-step is that the node client acting as the source node client sends a loop probe message with hop count limit to all directly connected neighbor node clients. The second sub-step is that other node clients acting as intermediate node clients receive the probe message, update the path node client list, and forward it to the directly connected neighbors of non-source node clients. The third sub-step is that the source node client selects the link with the fewest node clients from the returned closed links as the minimum reachable closed loop, sends a loop binding request to the node clients within the minimum reachable closed loop, and completes bidirectional confirmation.

[0055] Specifically, the source node client sends a circle probe message with a hop count limit to all directly connected neighbor node clients. It should be noted that the circle probe message carries a list of path node clients, a hop count limit, a source node client identifier, and a unique sequence number. The hop count limit is set to 3 hops to control the forwarding range of the probe message and prevent infinite forwarding within the cluster, which could cause a broadcast storm. The path node client list initially only contains the source node client identifier when the message is sent, used to record the paths of all node clients traversed by the probe message. The source node client then sends circle probe messages to all its neighbor node clients with established P2P direct connections, initiating the standard closed-loop probe process.

[0056] Specifically, after receiving the probe packet, other node clients acting as intermediate nodes update their path node client lists and forward it to their directly connected neighbors (excluding the source node client). It's important to note that upon receiving a circle probe packet, an intermediate node client first adds its own node client identifier to the path node client list in the packet and decrements the remaining hop count. It then checks if the remaining hop count is greater than 0. If it is not, it terminates the forwarding of the packet; otherwise, it forwards the updated circle probe packet to its directly connected neighbor node clients (excluding the source node client). Furthermore, upon receiving a circle probe packet, the intermediate node client checks if its own identifier already exists in the path node client list. If it does, it indicates a loop, and the forwarding of the packet is terminated to avoid network resource consumption caused by repeated forwarding.

[0057] It should be noted that if the last identifier in the path node client list of the circle probe message received by the intermediate node client is the identifier of the source node client, it means that the message has formed a closed link. The intermediate node client will no longer forward the message, but will return the message directly to the source node client for the source node client to filter the closed link.

[0058] Specifically, the source node client selects the link with the fewest node clients from the returned closed links as its minimum reachable closed loop, sends a loop binding request to the node clients within the minimum reachable closed loop, and completes bidirectional confirmation. It should be noted that the source node client receives all returned messages containing closed links, extracts the number of node clients in each closed link, sorts and compares them, and selects the closed link with the fewest node clients as its own minimum reachable closed loop. If the source node client does not receive any closed link return messages, it actively negotiates with its two directly connected neighbor node clients to construct a standard closed loop with three node clients, ensuring that each node client in the cluster has a corresponding belonging standard closed loop and that there are no isolated nodes.

[0059] It should be noted that the standard closed loop referred to in this application refers to a closed link formed by direct connections between node clients; the minimum reachable closed loop is a standard closed loop and is the standard closed loop with the fewest number of nodes, which will not be repeated in this application.

[0060] like Figure 4As shown, the topology of the node clients includes node client 1, node client 2, node client 3, node client 4, node client 5, node client 6, and other node clients not labeled in the diagram. The lines in the diagram represent direct connections. Taking node client 1 as an example, node client 1 includes two standard closed loops. Node client 1, node client 2, and node client 3 form one standard closed loop, and node client 1, node client 2, node client 4, node client 5, and node client 6 form another standard closed loop. For simplicity, if node client 1 only has the two standard closed loops shown in the diagram, then node client 1, node client 2, and node client 3 constitute the minimum reachable closed loop. Figure 4 This application is merely an example and does not impose any limitations or elaborate on it.

[0061] After determining the minimum reachable closed loop, the source node client generates a loop binding request. This request includes a list of node clients within the standard closed loop, a unique loop identifier, and the source node client's identifier. This request is then sent to all other node clients within the standard closed loop. Upon receiving a loop binding request, the receiving node client records the standard closed loop in its local storage, returns a binding confirmation message to the source node client, and adds the source node client to its own standard closed loop. This ensures that all node clients within the standard closed loop have a completely consistent view of the standard closed loop, completing the two-way confirmation binding process.

[0062] For example, node client A, acting as the source node client, sends a circle probe message with a 3-hop limit to its directly connected neighbor nodes. It eventually receives return messages from multiple closed links. One of these closed links consists of node client A, node client B, and node client C—a total of three node clients. This is the link with the fewest nodes among all closed links. Node client A selects this link as its minimum reachable closed loop and sends a circle binding request to node clients B and C. Upon receiving the request, node clients B and C record the standard closed loop information, return a binding confirmation message to node client A, and simultaneously use this standard closed loop as their own minimum reachable closed loop, completing the bidirectional confirmation binding.

[0063] In this embodiment, the node client completes the complementary synchronization and aggregation list maintenance of the write-ahead log event content within the minimum reachable closed loop. This process includes four sub-steps: the first sub-step is that the node client receives the global ID bitmap and compares it with the local event ID to generate an independent event missing bitmap; the second sub-step is that the node client sends the event missing bitmap to all node clients within its minimum reachable closed loop; the third sub-step is that the node clients within the minimum reachable closed loop receive the missing bitmap, compare it with their own event list, and return the corresponding missing write-ahead log event content in batches to the requesting node client; the fourth sub-step is that the requesting node client verifies the event content, updates its local event information, and synchronously updates the event aggregation list within the loop.

[0064] Specifically, after receiving the global ID bitmap, the node client compares it with the local event IDs to generate an independent event missing bitmap. It's important to note that after the metadata server generates a new write-ahead log event, it broadcasts the updated global ID bitmap to all node clients in the cluster. Upon receiving the global ID bitmap, the node client extracts the globally unique IDs of all write-ahead log events recorded in the bitmap, compares them one by one with the locally stored, synchronized event IDs, filters out the event IDs not stored locally, and generates an independent event missing bitmap based on the filtered missing event IDs. This event missing bitmap only contains the IDs of the write-ahead log events missing locally on that node client and their corresponding hash values, used for subsequent requests for the corresponding event content from other node clients.

[0065] Specifically, each node client sends an event missing bitmap to all node clients within its minimum reachable closed loop. It's important to note that after generating the event missing bitmap, the node client prioritizes initiating event synchronization requests within its own minimum reachable closed loop, rather than sending requests to the entire cluster or the metadata server. This limits event synchronization to the minimum standard closed loop, reducing network traffic within the cluster and leveraging low-latency links from neighboring nodes on the same network segment to improve synchronization efficiency. The node client then sends the event missing bitmap and event synchronization requests to all other node clients within the standard closed loop via the established P2P direct connection.

[0066] Specifically, after receiving the missing bitmap, the node client within the minimum reachable closed loop compares it with its own event list and returns the corresponding missing write-ahead log (WAL) event content in batches to the requesting node client. It should be noted that after receiving the event synchronization request and the missing event bitmap, the node client within the standard closed loop extracts the missing event ID from the bitmap, compares it with its own locally stored WAL event list, selects the complete WAL event content corresponding to the missing event ID from its own stored list, packages multiple event contents in batches, and returns them to the requesting node client via a P2P direct connection. If the node client does not have the corresponding missing event content, it returns a response message indicating no corresponding event to the requesting node client, without carrying the event content.

[0067] Specifically, after verifying the event content, the requesting node client updates its local event information and synchronously updates the event aggregation list within the loop. It should be noted that after receiving write-ahead log event content from other node clients within the loop, the requesting node client first performs integrity verification on the received event content based on the corresponding event hash value in the global ID bitmap. If the verification passes, the event content is stored in the local event list, and the locally synchronized event ID information is updated, the missing event bitmap is updated, and synchronized event IDs are removed. Furthermore, the node client synchronously updates the verified write-ahead log event content to the locally maintained event aggregation list within the loop. This list records all write-ahead log event content stored by all node clients within the minimum reachable closed loop. All node clients within the standard closed loop synchronize this aggregation list in real time, achieving redundant multi-replica storage of write-ahead log events within the standard closed loop. This ensures that any node client within the standard closed loop can obtain the complete write-ahead log event content through the event aggregation list within the loop.

[0068] In this embodiment, the node client detects the liveness status of the node client within the minimum reachable closed loop through the Lustre native Lnet heartbeat mechanism. When a node client failure is detected, the standard closed loop is automatically reconstructed and the event list is synchronized. This process includes three sub-steps: the first sub-step is that the remaining node clients in the loop suspend cross-loop event synchronization and re-execute the minimum reachable closed loop detection process based on the remaining node clients; the second sub-step is that the remaining node clients negotiate to add new node clients to restore the standard closed loop size to the preset number of nodes; and the third sub-step is that after the restored standard closed loop completes two-way confirmation binding, all node clients in the minimum reachable closed loop broadcast their own event lists and generate a new intra-loop event aggregation list through complementary synchronization.

[0069] Specifically, node clients use Lustre's native Lnet heartbeat mechanism to detect the liveness of other node clients within the minimum reachable closed loop. It's important to note that the Lnet heartbeat mechanism is a native node liveness detection mechanism of the Lustre file system, requiring no additional keep-alive logic. Within the standard closed loop, node clients send heartbeat detection messages every 100 milliseconds via the Lnet network layer. Upon receiving a heartbeat message, the receiver immediately returns a heartbeat response message. If a node client fails to receive a heartbeat response message from another node client within the loop three consecutive times, that node client is considered to have failed, triggering the standard closed loop automatic reconstruction and event list synchronization process.

[0070] Specifically, the remaining node clients within the loop suspend cross-loop event synchronization and re-execute the minimum reachable closed-loop detection process based on the remaining node clients. It should be noted that after a node client failure is detected, the remaining normal node clients within the standard closed-loop will first suspend their currently executing cross-loop event synchronization operations to avoid initiating invalid cross-loop interactions based on the failed standard closed-loop view. Simultaneously, based on the remaining normal node clients, they will re-execute the minimum reachable closed-loop detection process. This detection process is consistent with the aforementioned standard closed-loop self-organizing construction process, searching for new closed links by sending loop detection messages.

[0071] Specifically, the remaining client nodes negotiate to add new client nodes, restoring the standard closed loop size to the preset number of nodes. It should be noted that the preset number of nodes is 3 to 4 for the standard closed loop. If, after re-probing, the number of nodes in the standard closed loop formed by the remaining client nodes is less than 3, the remaining client nodes will negotiate to select new client nodes from their directly connected neighbor client nodes to join the standard closed loop, restoring the number of nodes in the new standard closed loop to the preset 3 to 4 nodes. This ensures that the new standard closed loop has sufficient fault tolerance and event synchronization efficiency.

[0072] Specifically, after the restored standard closed loop completes two-way confirmation binding, all node clients within the minimum reachable closed loop broadcast their own event lists, generating a new intra-loop event aggregation list through complementary synchronization. It should be noted that after the remaining node clients determine the new standard closed loop, they will complete two-way confirmation binding with the newly joined node clients, ensuring that the standard closed loop views of all node clients within the minimum reachable closed loop are completely consistent. Afterwards, all node clients within the restored standard closed loop will broadcast their locally stored write-ahead log event lists to other node clients within the loop. Through intra-loop complementary synchronization, the event content of all node clients is merged, generating a new intra-loop event aggregation list, restoring the multi-replica redundant storage of write-ahead log events within the standard closed loop. The entire process requires no participation from a metadata server or other central nodes and is entirely self-organized by the node clients within the standard closed loop.

[0073] In this embodiment, when a node client fails to find the target write-ahead log event within its own minimum reachable closed loop, it triggers a cross-loop query process. This process includes three sub-steps: the first sub-step is for the node client to determine the node client with the fewest overlapping nodes with the minimum reachable closed loop of the querying node client from among the node clients included in its own minimum reachable closed loop, and to use this node client as the cross-loop jump node client; the second sub-step is for the node client to send a query request for the target event and a missing bitmap to the cross-loop jump node client; and the third sub-step is for the jump node client to complete the event query within its minimum reachable closed loop, return the matching event content to the requesting node client, or continue to forward the query request to the next jump node client according to the same rules.

[0074] Specifically, the node client selects the node client with the fewest overlapping nodes with the minimum reachable closed loop of the query node client from among the node clients included in its own minimum reachable closed loop, and uses this node client as the cross-loop jump node client. It should be noted that if a node client does not find the target write-ahead log event in the event aggregation list within its minimum reachable closed loop, it triggers a cross-loop query process. First, it retrieves the standard closed loop information for each node client within its minimum reachable closed loop. Then, it calculates the number of overlapping nodes between each node client's standard closed loop and its minimum reachable closed loop. The number of overlapping nodes represents the number of identical node clients in the two standard closed loops. The fewer the overlapping nodes, the lower the overlap between the two standard closed loops, and the larger the range of new node clients that can be covered in a single cross-loop jump, allowing for a faster search of the node client storing the target write-ahead log event within the cluster. The node client will select the node client with the fewest overlapping nodes as the cross-loop jump node client for this cross-loop query.

[0075] For example, the standard closed loop of node client A includes node clients A, B, and C; the standard closed loop of node client B includes node clients B, D, and E; the number of overlapping nodes with the standard closed loop of node client A is 1; the standard closed loop of node client C includes node clients C, A, and F; the number of overlapping nodes with the standard closed loop of node client A is 2. Therefore, node client A selects node client B as the cross-loop jump node client.

[0076] Specifically, the node client sends a query request for the target event and a missing bitmap to the cross-circle jump node client. It should be noted that the query request carries the globally unique ID of the target write-ahead log event and the identifier of the requesting node client. The missing bitmap contains the IDs and hash values ​​of all write-ahead log events that the requesting node client needs to obtain. The node client sends the query request and missing bitmap to the selected cross-circle jump node client via a P2P direct connection.

[0077] Specifically, the jump node client completes the event query within the minimum reachable closed loop, returns the matching event content to the requesting node client, or continues to forward the query request to the next jump node client according to the same rules. It should be noted that after receiving the query request and missing bitmap, the jump node client first queries its own local event list and the event aggregation list of the minimum reachable closed loop. If the corresponding write-ahead log event content is found, it is validated and then batch-packaged and returned to the requesting node client. If no corresponding event content is found, the same filtering rules are applied, and the next cross-loop jump node client with the fewest overlapping nodes is selected from the node clients of the minimum reachable closed loop to continue forwarding the query request and missing bitmap. It should be noted that the maximum number of forwarding hops for cross-loop queries is set to 3 hops. If the corresponding write-ahead log event content is still not found after 3 forwardings, the query process is terminated, and the requesting node client initiates a single event retrieval request to the metadata server to obtain the corresponding write-ahead log event content. This eliminates the need to retrieve the entire file layout information, reducing the processing pressure on the metadata server.

[0078] In addition, after the requesting node client obtains the target write-ahead log event content through cross-loop query, it will perform integrity verification on the event content. If the verification passes, the event content will be stored in the local event list and broadcast to all node clients within the minimum reachable closed loop, updating the event aggregation list within the loop. This enables a single query for an event to be shared across the entire loop, preventing other node clients within the standard closed loop from repeatedly initiating the same cross-loop query request and reducing cross-loop network interaction traffic within the cluster.

[0079] This application embodiment also provides a data reading system for a distributed parallel file system mirror copy, which includes a node client and a metadata server. The node client receives read requests from upper-layer applications, initiates a read operation to the primary copy of the Mirror file with file-level redundancy enabled, intercepts error messages returned during failed read operations and triggers an automatic recovery process, interacts with the metadata server to complete the state degradation of the failed primary copy and the state promotion of the healthy standby copy, and after obtaining the latest file layout information returned by the metadata server, redirects the read request to the healthy copy with the state "init" and completes the data reading.

[0080] Those skilled in the art will understand that the system provided in this application embodiment is implemented in the same way as the aforementioned data reading method for distributed parallel file system image copies. All technical details in the aforementioned method embodiment are applicable to this system embodiment and will not be repeated here.

[0081] For example, the following complete cluster application scenario embodiment further illustrates the complete execution flow of the technical solution of this application. In a service Lustre cluster used to carry user call detail record (CDR) data analysis services, the cluster includes 2 metadata servers, 10 object storage servers, and 80 node clients. All CDR data files have the Lustre Mirror feature enabled, and are configured with 1 primary and 2 backup copies, for a total of 3 copies. Using the technical solution provided in the embodiment of this application, automatic switching and seamless service recovery are achieved after the primary copy fails. At the same time, P2P synchronization of copy state change events is completed through the minimum reachable closed loop.

[0082] After the cluster starts, all 80 node clients are mounted to the Lustre cluster. When a node client opens the target call detail record (CDR) mirror file, it requests the list of open sessions for that file from the metadata server, obtaining the network address information of all node clients that have opened the file. Each node client then selects active node clients on the same rack segment and establishes a P2P direct link. Each node client acts as a source node client, sending a loop probe message with a 3-hop limit to its directly connected neighbor node clients. Intermediate node clients receive the message, update their path list, and forward it. The source node client selects the 3-node link with the fewest nodes from the returned closed links as its minimum reachable closed loop. The 80 node clients form 27 independent minimum reachable closed loops. Each standard closed loop completes bidirectional confirmation binding, ensuring that the standard closed loop views of all node clients within the minimum reachable closed loop are completely consistent.

[0083] The object storage server where the primary copy of the target call detail record (CDR) file resides experiences a read error due to a disk failure. Node client A initiates a read operation to the primary copy and receives a read failure response message. Node client A intercepts the error message, extracts the error code and fault context information from the message, and matches it with the locally preset copy fault feature database. The matching result is a copy fault type error. Node client A uses this error message as a trigger signal for the automatic recovery process, locks the layout information of the target CDR file, and suspends new read operations to the faulty primary copy.

[0084] Node client A interacts with the cluster's metadata server, sending a replica status downgrade request to downgrade the failed primary replica from init to stale. After the metadata server completes the status modification, node client A selects a healthy backup replica 1 from the backup replicas according to a preset selection strategy that minimizes network latency, and sends a replica status promotion request to the metadata server to promote backup replica 1 from stale to init. After the metadata server completes this replica status change, it generates a corresponding write-ahead log event, updates the global ID bitmap, and broadcasts the updated global ID bitmap only to all 80 node clients in the cluster. It does not push the complete content of the write-ahead log event; the broadcast message length is only 125 bytes, and the metadata server's CPU usage shows no significant fluctuation.

[0085] Node client A receives the latest layout information of the target call detail record (CDR) file returned by the metadata server, updates the layout information in its local cache, releases the layout lock, and redirects previously failed read requests, as well as subsequent new read requests for the file, to the object storage server where the newly promoted standby replica 1 resides. After successfully obtaining the CDR file data, it returns it to the upper-layer data analysis business. The upper-layer business only perceives a brief read delay and there is no interruption.

[0086] After receiving the global ID bitmap broadcast by the metadata server, all node clients in the cluster compare it with their own locally stored event IDs to generate independent event missing bitmaps. Each node client sends its event missing bitmap to other node clients within its minimum reachable loop. Upon receiving the request, each node client compares it with its own event list and returns the corresponding missing write-ahead log event content in batches to the requesting node client. After verifying the event content, the requesting node client updates its local event information and the event aggregation list within its loop. The entire event synchronization process is completed within 5 milliseconds, and the total time for all 80 node clients to complete event synchronization does not exceed 10 milliseconds. The local replica state views of all node clients are updated, and no node clients experience view inconsistencies.

[0087] During event synchronization, node client B within a standard closed loop went offline due to a business process crash. The remaining two node clients within the loop, through the Lnet heartbeat mechanism, failed to receive a heartbeat response from node client B three times consecutively, determining that node client B had failed and triggering the standard closed loop automatic reconstruction process. The remaining two node clients within the loop first paused cross-loop event synchronization, re-executed the minimum reachable closed loop detection process, and negotiated to add node client C from the same rack to the new standard closed loop, restoring the standard closed loop size to the preset number of 3 nodes. After the restored standard closed loop completed two-way confirmation binding, the 3 node clients within the loop broadcast their own event lists, generating a new intra-loop event aggregation list through complementary synchronization. The entire reconstruction and synchronization process was completed within 800 milliseconds, with no event content loss and no interruption in the event synchronization process.

[0088] A newly added node client C, lacking historical write-ahead log event data for the target file locally, first queries within its own new standard closed loop. Finding only the latest event content and missing historical events, it triggers a cross-loop query process. Node client C selects node client D from the nodes of the minimum reachable closed loop, choosing it as the cross-loop jump node. Node client C sends a query request and event missing bitmap to node client D. Node client D retrieves the complete historical write-ahead log event content within its own standard closed loop and returns the event content in batches to node client C. After successful verification, node client C updates its local event list and broadcasts the historical event content to other nodes within the minimum reachable closed loop, updating the event aggregation list within the loop. The entire cross-loop query process is completed within two hops, with a total time not exceeding 8 milliseconds.

[0089] Throughout the primary replica failure handling process, the metadata server only performed replica state changes and lightweight broadcasts of the global ID bitmap, without participating in the distribution and synchronization of any event content. Broadcast traffic was reduced by over 99%, and the response latency of the metadata service remained stable within 1 millisecond, with no performance bottlenecks observed. All 80 node clients completed replica state event synchronization within 10 milliseconds, obtaining the latest file layout information. No node clients initiated invalid read requests based on outdated layout information. The total failover time remained stable within 15 milliseconds, and data analysis services were completely unaware of the failure, with no performance degradation.

[0090] Those skilled in the art will understand that the operations related to the Lustre file system not described in detail in the embodiments of this application are all performed in accordance with the native standard specifications of the Lustre file system. All technical solutions in the embodiments of this application are fully compatible with the native architecture of the Lustre file system and can be deployed in the Lustre cluster as optional functional modules. They are completely transparent to files without the Mirror feature enabled and traditional node clients without the deployed functional module, enabling smooth upgrades and deployments of the cluster.

[0091] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for reading data from a distributed parallel file system mirror copy, applied to a node client, characterized in that, The method includes: The node client receives read requests from the upper-layer application, initiates a read operation on the primary replica of the Mirror file with file-level redundancy enabled, intercepts error messages returned when the read operation fails, and triggers an automatic recovery process. The node client interacts with Lustre's metadata server to complete the state degradation of the failed primary replica and the state promotion of the healthy standby replica; The node client obtains the latest file layout information returned by the metadata server, redirects the read request to the healthy replica with the status of init, and completes the data read.

2. The data reading method according to claim 1, characterized in that, The interception of error messages returned during failed read operations and the triggering of an automatic recovery process include: The node client determines the failure type of the read operation and filters out error messages that are not replica failure types; The node client uses the filtered replica failure error messages as a trigger signal for the automatic recovery process, locking the layout information of the target Mirror file; The node client suspends initiating new read operations to the failed primary replica.

3. The data reading method according to claim 2, characterized in that, The node client determines the failure type of the read operation and filters out error messages that are not replica failures, including: The node client extracts the error codes and fault context information returned by the read operation; The node client will match the extracted information with a pre-defined database of replica fault characteristics; The node client will directly return the non-replica failure error information that failed to match to the upper layer application, terminating the automatic recovery process.

4. The data reading method according to claim 1, characterized in that, The method is applied to a distributed network cluster composed of multiple node clients. After the metadata server completes the replica state change, it generates a write-ahead log event and broadcasts the global ID bitmap of the event only to all node clients in the cluster. The node clients complete the P2P synchronization of the event content based on the minimum reachable closed loop maintained by themselves. The write-ahead log event is an immutable event unit generated by the metadata server after completing the Mirror replica state change, containing a globally unique event ID, a file identifier, and a snapshot of the replica state change. The global ID bitmap is a lightweight message that only records the global ID and hash value of the write-ahead log event. The minimum reachable closed loop is a closed link formed by a single node client as the source node client, based on the P2P direct connection link between node clients, which can return to the source node client from the source node client through the minimum number of intermediate node clients.

5. The data reading method according to claim 4, characterized in that, The node client completes P2P synchronization of event content based on its own maintained minimum reachable closed loop, including: The node client searches for active node clients within the same network segment that have opened the same Mirror file and establishes a P2P direct connection link. The node client completes the self-organized construction and two-way confirmation binding of the minimum reachable closed loop based on the P2P direct connection link; The node client completes the complementary synchronization of write-ahead log event content and the maintenance of the aggregation list within the minimum reachable closed loop.

6. The data reading method according to claim 5, characterized in that, The node client completes the self-organized construction and two-way confirmation binding of the minimum reachable closed loop based on the P2P direct connection link, including: The node client, acting as the source node client, sends cycloprobe messages with hop count limits to all directly connected neighbor node clients; After receiving the probe message, other node clients acting as intermediate node clients update the list of path node clients and forward it to their directly connected neighbors that are not the source node clients. The source node client selects the link with the fewest node clients from the returned closed links as the minimum reachable closed loop, sends a loop binding request to the node clients within the minimum reachable closed loop, and completes bidirectional confirmation.

7. The data reading method according to claim 5, characterized in that, The node client completes the complementary synchronization and aggregation list maintenance of the pre-written log event content within the minimum reachable closed loop, including: the node client receives the global ID bitmap, compares it with the local event ID, and generates an independent event missing bitmap; The node client sends the missing event bitmap to all node clients within its smallest reachable closed loop. After receiving the missing bitmap, the node client within the minimum reachable closed loop compares it with its own event list and returns the corresponding missing write-ahead log event content in batches to the requesting node client. After requesting the node client to verify the event content, update the local event information and synchronously update the event aggregation list within the circle.

8. The data reading method according to claim 4, characterized in that, Node clients use Lustre's native Lnet heartbeat mechanism to detect the liveness status of node clients within the minimum reachable closed loop. When a node client failure is detected, it triggers automatic reconstruction of the standard closed loop and synchronization of the event list, including: The remaining node clients within the circle pause cross-circle event synchronization and re-execute the minimum reachable closed-loop circle detection process based on the remaining node clients. The remaining node clients negotiate to add new node clients, restoring the standard closed-loop size to the preset number of nodes; After the restored standard closed loop completes two-way confirmation binding, all node clients within the minimum reachable closed loop broadcast their own event lists, and generate a new clustered event list within the loop through complementary synchronization.

9. The data reading method according to claim 4, characterized in that, When a node client fails to find the target write-ahead log event within its minimum reachable closed loop, it triggers a cross-loop query process, including: The node client determines the node client with the fewest overlapping nodes with the minimum reachable closed loop of the queried node client from among the node clients contained in its own minimum reachable closed loop, and uses this node client as the cross-loop jump node client; The node client sends a query request for the target event and a missing bitmap to the cross-circle jump node client; The jump node client completes the event query within the minimum reachable closed loop, returns the matching event content to the requesting node client, or continues to forward the query request to the next jump node client according to the same rules.

10. A data reading system for a distributed parallel file system mirror copy, characterized in that, Includes node clients and metadata servers; The node client receives read requests from the upper-layer application, initiates a read operation on the primary replica of the Mirror file with file-level redundancy enabled, intercepts error messages returned when the read operation fails, and triggers an automatic recovery process. Interact with Lustre's metadata server to complete the state degradation of the failed primary replica and the state promotion of the healthy standby replica; obtain the latest file layout information returned by the metadata server, redirect read requests to the healthy replica with the state of init, and complete the data read.

Images