A fault handling method and computing device

By disconnecting communication and configuring the status of other nodes in the event of a shared file system failure, fault isolation is achieved, solving the problem of the wide impact of node failures and ensuring business continuity.

CN118170571BActive Publication Date: 2026-11-17XFUSION DIGITAL TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410132852.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-30
Publication Date
2026-11-17
Estimated Expiration
2044-01-30

AI Technical Summary

Technical Problem

When a node fails, the existing shared file system directly resets the entire node, which will affect the business of other types of data on the node, resulting in a large impact on the fault handling.

Method used

When a node fails, communication with other nodes is disconnected, and other nodes are configured to be in the first state in the target configuration file, so that they simulate leaving the shared cluster and stop running the shared file system, thereby achieving fault isolation without restarting the entire node.

Benefits of technology

This effectively reduced the scope of the failure, avoided interference with the normal operation of other services, and ensured that the shared file system could still be unloaded normally in the event of a failure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118170571B_ABST
    Figure CN118170571B_ABST
Patent Text Reader

Abstract

This application provides a fault handling method and computing device, relating to the field of computing device technology. It enables fault isolation of a shared file system without resetting the node, preventing disruption to the normal operation of other services on the node. The method can be applied to the first node in a shared cluster, where each node is configured with a shared file system used by the nodes to access shared data. The method includes: when the shared file system of the first node fails, the first node disconnects its communication connection with other nodes; the node status of other nodes configured on the first node is set to a first state; these other nodes are nodes in the shared cluster other than the first node; and fault isolation of the shared file system occurs while all other nodes in the shared cluster are in the first state. This application embodiment can be used during computing device maintenance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computing device technology, and in particular to a fault handling method and a computing device. Background Technology

[0002] A shared file system is a file system used in multi-node environments. Multiple nodes read and write the same data (such as shared data) in parallel through the shared file system. Each piece of data in this shared data is stored on different nodes in a striped or other format.

[0003] The current fault handling mechanism for shared file systems involves resetting the node when a shared file system on a specific node fails, thus stopping the shared file system's operation and ensuring that other nodes can access the shared resources used by that node for accessing shared data. However, a node contains not only shared data but also distributed resource data and local resource data. Resetting the entire node directly would affect the services corresponding to the distributed and local resource data on that node, resulting in a wider impact from fault handling. Summary of the Invention

[0004] This application provides a fault handling method and computing device that can isolate a shared file system from failure without resetting the node, thus avoiding affecting the normal operation of other services on the node.

[0005] In a first aspect, embodiments of this application provide a fault handling method applied to a first node in a shared cluster. Each node in the shared cluster is configured with a shared file system, which is used by the nodes to access shared data corresponding to the shared cluster. The method includes: when the shared file system of the first node fails, the first node disconnects its communication connection with other nodes; the node status of other nodes configured on the first node is set to a first state; the other nodes are nodes in the shared cluster other than the first node; the first state is used to indicate that the shared file system on the node stops running; and when all other nodes in the shared cluster are in the first state, the shared file system is fault isolated.

[0006] The fault handling method provided in this application, when the shared file system of the first node fails, configures the node status of other nodes configured on the first node to a first state, which indicates that the shared file system on the node has stopped running. By disconnecting the communication connection between the first node and other nodes, and configuring other nodes to go offline (i.e., the shared file system stops running) on ​​the first node, the first node can confirm that it is the only node in the shared cluster. In this way, when the shared file system fails, the first node does not need to negotiate with other nodes or restart the entire node to achieve fault isolation of the shared file system. This effectively solves the problem in current solutions where a failure would reset the entire node, causing other services to be affected, and reduces the impact of shared file system failures.

[0007] One possible implementation involves configuring the node states of other nodes on the first node to the first state. This includes configuring the node states of other nodes to the first state in the target configuration file. The target configuration file is a file that configures the running parameters of the shared file system, including the node states of each node in the shared cluster. It should be understood that by modifying the target configuration file on the first node, it is possible to simulate other nodes exiting the shared cluster on the first node, thereby ensuring that the first node can properly unload the shared file system without affecting the normal use of other nodes.

[0008] In another possible implementation, the shared file system runs in kernel mode, and the first node also includes a process manager running in user mode; the process manager is used to manage all processes in the node; when other nodes in the shared cluster are all in the first state, fault isolation is performed on the shared file system, including: when other nodes in the shared cluster are all in the first state, the shared file system sends a first instruction message to the process manager; the first instruction message is used to instruct the unloading of the shared file system; in response to the first instruction message, the shared file system is unloaded through the process manager to isolate the shared file system from failure.

[0009] Another possible implementation involves unloading the shared file system through a process management program. This includes: identifying the target process in the first node; ensuring the target process is a process in the first node that depends on the shared file system; stopping the target process through the process management program; and unloading the shared file system through the process management program after the target process has stopped running. It should be understood that the shared file system is a kernel-mode program and cannot be forcibly stopped. This solution achieves normal unloading of the shared file system by terminating processes that depend on it.

[0010] In another possible implementation, when the shared file system on the first node fails, the method further includes: configuring the read / write status parameter of the shared file system on the first node to a non-writable state; the non-writable state is used to indicate that the first node is prohibited from writing data to the shared data. It should be understood that configuring it to a non-writable state can prevent erroneous data caused by the failure from having a greater impact on the shared file system and shared data.

[0011] In another possible implementation, the method further includes: after the shared file system of the first node recovers from a failure, the first node establishes a communication connection with other nodes and deisolates the shared file system; the node status of other nodes configured on the first node is configured to a second state; the second state is used to indicate that the shared file system on the node is operating normally.

[0012] In another possible implementation, the method further includes: when it is determined that the communication connection between the first node and the second node among other nodes is broken, accessing shared data based on a first shared resource; the first shared resource includes a second shared resource, which is the resource occupied by the second node to access the shared data. It should be understood that by occupying the shared resources of the failed node, the impact of a node failure on the services of the shared cluster can be reduced.

[0013] In another possible implementation, the method further includes: receiving an instruction when the shared file system on the first node is functioning correctly; the instruction instructing the unloading of the shared file system on the first node; in response to the instruction, releasing a second shared resource through interaction with other nodes; the second shared resource being the resource occupied by the first node for accessing shared data; and unloading the shared file system on the first node after the first node has released the second shared resource. It should be understood that, under normal circumstances, when unloading the shared file system, interaction allows other nodes to promptly occupy the shared resources, ensuring business continuity.

[0014] In another possible implementation, each node is configured with a disk, and the disks on different nodes are used to store different parts of each piece of data in the shared data.

[0015] Secondly, embodiments of this application provide a fault handling apparatus, which includes: a configuration module and an isolation module; the configuration module is used to disconnect communication connections with other nodes; the configuration module is also used to configure the node status of other nodes configured on the first node to a first state when the shared file system of the first node fails; the other nodes are nodes in the shared cluster other than the first node; the first state is used to indicate that the shared file system on the node stops running; the isolation module is used to isolate the shared file system from failure when all other nodes in the shared cluster are in the first state.

[0016] In one possible implementation, the configuration module is specifically used to configure the node status of the other nodes to the first state in the target configuration file; wherein the target configuration file is a file that configures the running parameters of the shared file system, and the running parameters include the node status of each node in the shared cluster.

[0017] In another possible implementation, the shared file system runs in kernel mode, and the first node also includes a process management program running in user mode; the process management program is used to manage all processes in the node; the unloading module is specifically used to send a first instruction message to the process management program when all other nodes in the shared cluster are in the first state; the first instruction message is used to instruct the unloading of the shared file system; in response to the first instruction message, the shared file system is unloaded through the process management program.

[0018] In another possible implementation, the unloading module is specifically used to: identify the target process in the first node; the target process is the process in the first node that depends on the shared file system; terminate the target process through the process management program; and unload the shared file system through the process management program when the target process has stopped running.

[0019] In another possible implementation, the configuration module is also used to configure the read / write status parameter of the shared file system in the first node to a non-writable state; the non-writable state is used to indicate that the first node is prohibited from writing data to the shared data.

[0020] In another possible implementation, the configuration module is also used to establish communication connections with other nodes and deisolate the shared file system; configure the node status of other nodes configured on the first node to a second state; the second state is used to indicate that the shared file system on the node is operating normally.

[0021] In another possible implementation, the above apparatus further includes an access module. The access module is used to access shared data based on a first shared resource when it is determined that the communication connection between the first node and the second node among the other nodes is broken; the first shared resource includes the resources occupied by the second node for accessing the shared data.

[0022] In another possible implementation, the above apparatus further includes: a receiving module and a sending module; the receiving module is configured to receive an instruction when the shared file system of the first node is not faulty; the instruction is configured to instruct the unloading of the shared file system on the first node; the sending module is configured to release a second shared resource by interacting with other nodes in response to the instruction; the second shared resource is the resource occupied by the first node for accessing the shared data; the unloading module is configured to unload the shared file system on the first node when the resource occupied by the first node for currently accessing the shared data has been released.

[0023] In another possible implementation, each node is configured with a disk, and the disks on different nodes are used to store different parts of each piece of data in the shared data.

[0024] Thirdly, embodiments of this application provide a computing device including a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the method described in the first aspect.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium comprising: computer software instructions; when the computer software instructions are executed in a computing device, they cause the computing device to implement the method described in the first aspect.

[0026] Fifthly, embodiments of this application provide a computer program product that, when run on a computing device, causes the computing device to execute the steps of the relevant method described in the first aspect above, so as to implement the method of the first aspect above.

[0027] The beneficial effects of the second to fifth aspects mentioned above can be referred to the corresponding description of the first aspect, and will not be repeated here. Attached Figure Description

[0028] Figure 1 A schematic diagram of the architecture of a local file system provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of a shared cluster architecture provided for an embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating the resource distribution in a node, provided as an embodiment of this application.

[0031] Figure 4 This application provides a schematic diagram of the architecture of a computing device.

[0032] Figure 5 A schematic diagram of the architecture of software modules involved in a fault handling process provided in an embodiment of this application;

[0033] Figure 6 A flowchart illustrating a fault handling method provided in an embodiment of this application;

[0034] Figure 7 A flowchart illustrating another fault handling method provided in an embodiment of this application;

[0035] Figure 8 A flowchart illustrating another fault handling method provided in an embodiment of this application;

[0036] Figure 9 A flowchart illustrating another fault handling method provided in an embodiment of this application;

[0037] Figure 10 A flowchart illustrating another fault handling method provided in an embodiment of this application;

[0038] Figure 11 This application provides a schematic flowchart of a fault handling process.

[0039] Figure 12 This is a schematic diagram of the composition of a fault handling device provided in an embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the composition of a computing device provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0044] The following is a brief introduction to the technical terms used in the embodiments of this application:

[0045] 1. Node: In the field of computing devices, a node is a single device in a network, and each node has a unique identifier to identify its identity in the network. In the embodiments of this application, in a shared cluster composed of multiple computing devices, a computing device can be referred to as a node.

[0046] 2. Shared Cluster: A cluster composed of multiple computing devices. These devices integrate their physical storage (such as disks) through a shared file system, forming a single directory space for data storage. Any computing device can access data from this directory space without needing to know whether the data is stored on its own node or another node. This allows any node to access data in the physical storage of other computing devices as if it were accessing data in its own physical storage. The data in the integrated physical storage can be called shared data. The shared file system can be OCFS2, GFS, VMFS, etc.

[0047] The implementation methods of file management in computing devices generally include implementation methods based on local file systems and implementation methods based on shared file systems.

[0048] In implementations based on a local file system, the physical storage (such as a disk) used to store data within a node is connected to the node's processor. The processor in the node accesses the data in the physical storage by running the local file system. For example, the local file system could be FAT16 / 32, NTFS, or EXT2 / 3 / 4 extended filesystems. Figure 1 This application provides a schematic diagram of the architecture of a local file system, as shown in the embodiments. Figure 1 As shown, each node's processor connects to its corresponding disk through its own file system. Different nodes cannot access the contents of the disks corresponding to other nodes (for example, processor 1 of node 1 cannot access disk 2 corresponding to node 2), nor can they operate on the data in the disks corresponding to other nodes.

[0049] In the implementation based on a shared file system, multiple nodes can read and write their shared data in parallel through the shared file system. Figure 2 This is a schematic diagram of a shared cluster architecture provided in an embodiment of this application, such as... Figure 2As shown, in multiple nodes, the processors of each node read and write the same shared data in parallel through the same shared file system (each node is configured with its own shared file system, each shared file system reads the disk corresponding to its own node, and multiple shared file systems integrate the data of each node's disk through a network to form a single directory space. For each node, the data accessed through the shared file system is in the same directory space, therefore, the shared file systems configured in multiple nodes can be regarded as the same shared file system). That is, the processor of each node can access all disks (such as disks) through its own input / output (I / O) interface. Figure 2 The data in disks 1-3 can be read and written.

[0050] In this shared data structure, each piece of data is stored on different I / O disks (i.e., disks on different nodes) using a striping method. Striping refers to splitting a complete piece of data into multiple parts, with each part distributed across the disks of different nodes. This allows each node to create multiple processes to simultaneously access the disks of different nodes when reading data, thereby improving data access efficiency.

[0051] Shared file systems have the following characteristics:

[0052] 1. Parallel reading and writing of shared data among multiple nodes.

[0053] 2. To protect data, nodes need to implement mutual exclusion protection for shared data. This means that for a given piece of data in the shared data, only one node's process is allowed to access that data at a time. This mutual exclusion protection can be achieved using protection locks, a mechanism in the shared file system used to protect data from accidental modification or deletion by other nodes. When a node accesses a piece of data in the shared file system, that node configures parameters such as the access subject and access time for that data. This instructs that only the node specified by the access subject can access the file within that access time, thus implementing a protection lock. Due to the existence of the protection lock, when other nodes request access to the data through the shared file system, the shared file system determines that the protection lock exists and refuses access to other nodes, thereby preventing data conflicts caused by simultaneous modification by multiple nodes.

[0054] 3. If a node fails (e.g., the shared file system within the node becomes unusable), other nodes capable of accessing shared data will preempt the shared resources of the failed node. These shared resources refer to the resources used by nodes when accessing shared data (e.g., protection locks, data copies). For example, a normal node can identify protection locks belonging to the failed node through the shared file system and change the access subject specified by these locks from the failed node to a normal node, thus taking over the access to the files protected by the protection locks and achieving preemption of shared resources. Another example is that when a failed node accesses shared data, it may generate some temporary data (i.e., the aforementioned data copies) waiting to be written to disk. Normal nodes can read this temporary data through the shared file system and perform the write operation to disk on behalf of the failed node, thus achieving preemption of shared resources.

[0055] 4. If a node, such as the first node, experiences a failure in its shared file system (e.g., network failure, read / write failure), it may be unable to continue accessing shared data. In this case, the first node needs to stop the shared file system to exit the shared cluster. Since the shared resources occupied by the first node at this time will affect other nodes' access to shared data, the first node needs to release the shared resources it is using before it can stop the shared file system. During this process, the first node needs to negotiate with other nodes to allow them to preempt the shared resources that need to be released. However, because the first node's shared file system has failed, such as due to a network failure, the negotiation process cannot proceed, meaning the first node cannot stop the shared file system. Therefore, in the event of a failure in the first node's shared file system, the first node's processor will instruct the node to reset (i.e., restart) to forcibly stop the shared file system. Additionally, since multiple nodes form a shared cluster based on the shared file system, each node continuously sends heartbeat signals to other nodes to inform them of its online status. Therefore, when the shared file system of a certain node stops running, it will no longer continuously send heartbeat signals to other nodes, so that other nodes can detect that the node has stopped running and can take over the shared resources occupied by the node.

[0056] As mentioned earlier, the current approach to handling a shared file system failure within a node is to reset the entire node. However, in typical scenarios, a node not only runs services that depend on shared data but also services that depend on other types of data. If a node reset is caused by a shared file system issue, it will affect the services running on that node that rely on other types of data. Figure 3 This is a schematic diagram illustrating the resource distribution within a node, as provided in an embodiment of this application. Figure 3As shown, the system comprises three nodes, each deploying services that rely on distributed resource data, services that rely on local resource data, and services that rely on shared data. If the shared file system on node 1 fails, node 1 will automatically reset based on the current shared file system fault handling mechanism. However, the services that are running normally on node 1 and those that rely on distributed resource data, as well as those that rely on local resource data, will also be affected by the reset of node 1. Therefore, the current shared file system has a large scope of impact from fault handling, which is not reasonable.

[0057] In conclusion, ensuring that the fault handling mechanism of the shared file system does not affect the normal operation of other services on the node is an urgent problem to be solved.

[0058] This application provides a fault handling method. In this method, when the shared file system of a first node fails, the node status of other nodes configured on the first node is set to a first state, which indicates that the shared file system on the node has stopped running. By disconnecting the communication connection between the first node and other nodes, and configuring other nodes to go offline (i.e., the shared file system stops running) on ​​the first node, the first node can confirm that it is the only node in the shared file system. Thus, when the shared file system fails, the first node does not need to negotiate with other nodes or restart the entire node to achieve fault isolation of the shared file system. Therefore, this effectively solves the problem in current solutions where a fault resets the entire node, affecting other services and reducing the impact of shared file system failures.

[0059] The solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0060] The technical solutions provided in this application can be applied to computing devices, which may specifically be blade servers, high-density servers, rack servers, or high-performance servers, or general-purpose servers, graphics processing unit (GPU) servers, data processing unit (DPU) servers, etc. Figure 4 A schematic diagram of the system architecture of a computing device, such as Figure 4 As shown, the hardware of this computing device includes a processor, an out-of-band controller, memory, and RAM. The software includes an out-of-band management module and an operating system (OS).

[0061] The out-of-band management module runs within the out-of-band controller, while the OS runs on the processor (e.g., ...). Figure 4 (As shown).

[0062] The out-of-band management module can be a management unit for non-business modules. For example, the out-of-band management module can remotely maintain and manage the computing device through a dedicated data channel. This out-of-band management module is completely independent of the computing device's operating system and can communicate with the basic input / output system (BIOS) and the operating system through the computing device's out-of-band management interface.

[0063] For example, an out-of-band management module may include a management unit for the operating status of a computing device, a management system in a management chip outside the processor, a baseboard management controller (BMC), a system management mode (SMM), etc. It should be noted that the specific form of the out-of-band management module is not limited in the embodiments of this application; the above is merely an illustrative example.

[0064] Memory, also known as internal memory or main memory, is installed in memory slots on the motherboard of a computing device. Memory communicates with the memory controller via memory channels. Memory has at least one memory rank, each memory rank located on one face of the memory. Each memory rank includes at least one subrank. Each memory rank or subrank comprises multiple memory chips. Each memory chip is divided into multiple bankgroups. Each bankgroup comprises multiple banks. Each bankgroup is divided into multiple cells. Each cell has a row address and a column address, and each cell includes one or more bits. In one partitioning method, memory can be divided from top to bottom into memory chips, bankgroups, bankgroups, rows / columns, cells, and bits. The addresses of memory chips, bankgroups, bankgroups, rows, columns, cells, and bits on memory are actual physical addresses. In another partitioning method, the CPU divides the memory chip into multiple memory pages based on a paging mechanism. The address of the memory page is a virtual address, which needs to be translated into a real physical address.

[0065] Storage devices can be disks, smart media cards (SMCs), secure digital cards (SDs), flash cards, etc., that are equipped on computing devices, or they can be external storage devices, such as USB flash drives.

[0066] In this embodiment, the memory stores a driver program corresponding to the shared file system, and the processor can run the shared file system to read shared data from the memory.

[0067] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0068] Figure 5 This is a schematic diagram of the software module architecture involved in a fault handling process provided in an embodiment of this application. For example... Figure 5 As shown in the diagram, there are two levels: kernel mode and user mode. Kernel mode and user mode are two operating states of the operating system, and their main difference lies in their access permissions.

[0069] Kernel mode, also known as privileged mode, refers to the execution mode with the highest privileges in a computing device. Kernel mode is mainly used to handle system-level tasks, such as device drivers and file management.

[0070] The kernel mode includes a shared function module 501, a fault handling module 502, and a fault notification module 503. The functional modules shown in the kernel mode are those within the shared file system. Specifically, the shared function module 501 executes some functions currently available in the shared file system, such as read / write functionality, resource mutual exclusion, and fault detection. The fault handling module 502, when the shared function module 501 detects a shared file system fault, disconnects communication with other nodes and simulates the cessation of shared file system operation on the target node (the node running the fault handling module), thus exiting the shared cluster. The fault notification module 503 notifies the user-mode fault isolation module 504 to unload the shared file system to isolate it from other nodes (i.e., the faulty node cannot access shared resources or interact with other nodes through the shared file system).

[0071] User mode, also known as non-privileged mode, refers to a lower-privilege execution mode on a computing device, typically used by applications. User mode is primarily used to execute applications such as browsers, editors, and games. Due to the lower privileges associated with user mode, applications running in user mode can only execute certain specific instructions and access limited data to prevent malicious applications from harming the computing device.

[0072] The user space includes a fault isolation module 504, which is a functional module in the process management program running on the node. The fault isolation module 504 is used to receive messages sent by the fault notification module 503 in the kernel space, and unload the shared file system so that the faulty node cannot access the shared resources. This can be regarded as the node having left the shared cluster, thus achieving fault isolation between the faulty node and other normal nodes.

[0073] Figure 6 This is a flowchart illustrating a fault handling method provided in an embodiment of this application. For example, the fault handling method provided in this embodiment can be applied to the first node in a shared cluster. Each node in the shared cluster is configured with a shared file system, which is used by the nodes to access shared data corresponding to the shared cluster. For example... Figure 2 Nodes 1 through 3 form a shared cluster. This method can be applied to... Figure 2 In any node.

[0074] like Figure 6 As shown, the fault handling method provided in this application embodiment may specifically include the following steps:

[0075] S601. When the shared file system of the first node fails, the first node disconnects its communication connection with other nodes.

[0076] If the shared file system on the first node fails, erroneous instructions may be generated during the first node's operation. Due to the interactive nature of the nodes in the shared cluster, these erroneous instructions may be sent to other nodes in the shared cluster, affecting their normal operation. Therefore, when the shared file system on the first node fails, a message indicating that the communication connection should be disconnected can be sent to other nodes to sever the communication connection between the first node and other nodes.

[0077] It should be understood that when a shared file system fails, the first node promptly disconnects from other nodes. This can prevent the first node from generating incorrect instructions that could affect other nodes, and it also allows other nodes to promptly detect that the first node is offline and preempt the shared resources used by the first node to ensure the operation of the business.

[0078] S602. Configure the node status of other nodes configured on the first node to the first state.

[0079] Among them, the other nodes are the nodes in the shared cluster other than the first node, and the first state is used to indicate that the shared file system on the node has stopped running.

[0080] S603. When the first node is in the first state while all other nodes in the shared cluster are in the first state, the shared file system is fault isolated.

[0081] Furthermore, as mentioned earlier, when the shared function module of the shared file system detects a fault, it needs to negotiate with other nodes to release the shared resources it occupies, allowing the first node to normally stop the operation of the shared file system and achieve fault isolation. However, the existence of a fault prevents this negotiation process from taking place. Therefore, the current fault handling mechanism is to reset the entire node to stop the operation of the shared file system.

[0082] Therefore, in this embodiment, when the shared file system of the first node fails, the first node can configure the node status of other nodes configured on the first node to the first state, i.e., the state where the shared file system is stopped. In this way, the first node will perceive itself as the only node in the shared cluster at the current moment, thus eliminating the need for negotiation with other nodes when the shared file system stops running.

[0083] Furthermore, once the first node confirms that all other nodes in the shared cluster are in the first state, the first node can unload its own shared file system to achieve fault isolation of the shared file system.

[0084] It should be understood that configuring other nodes to the first state on the first node here refers to simulating other nodes leaving the shared cluster. Since the communication connection between the first node and other nodes has been broken, modifying the node's state will not affect other nodes. In other words, other nodes have not actually left the shared cluster. The advantage of this is that the first node can normally unload the shared file system without affecting the normal operation of other nodes, ensuring that other nodes can preempt the shared resources of the first node.

[0085] In the embodiments of this application, such as Figure 7 As shown, in S602 above, configuring the node state of other nodes configured on the first node to the first state can be implemented as follows:

[0086] S6021. In the target configuration file, configure the node status of other nodes as the first state.

[0087] The target configuration file is a file that configures the running parameters of the shared file system, including the node status of each node in the shared cluster.

[0088] Taking OCFS2 as an example of a shared file system, the target configuration file is typically located in the node's configuration file directory. The target configuration file primarily defines information about each node in the shared cluster, including at least one of the following: the node's address, node identifier, number of nodes, and node status. In other words, the target configuration file allows for adding or deleting nodes in the shared cluster, or modifying the address, identifier, or status of a specific node.

[0089] The shared file system on the first node relies on the configuration information in the target configuration file to operate; in other words, the target configuration file stores some parameters required for the shared file system to run. For example, the shared file system accesses the node based on the addresses of other nodes in the target configuration file to read or write data from the node's disk. The size of the shared resources is determined based on the number of nodes in the target configuration file for management. The node status in the target configuration file determines whether each node is online and running.

[0090] The first node can determine the location of the target configuration file and modify the node status of all nodes in the target configuration file except the first node to the first state (which can also be described as offline state), so as to simulate the exit of other nodes from the shared cluster on the first node.

[0091] For example, the shared function module requires node information configured in the target configuration file to run, and the shared function module has the location information of the target configuration file configured on it. The fault handling module requests the location information of the target configuration file from the shared function module, or determines the location of the target configuration file based on its own pre-configured location information (such as the index of the configuration file directory mentioned above), and then determines the target configuration file based on that address, and modifies the node status of other nodes in the target configuration file.

[0092] In this embodiment, the shared file system can run in kernel mode, and the first node also includes a process management program running in user mode, which manages the processes in the node. Figure 8 As shown, the above S603 can also be implemented as follows:

[0093] S6031. When all other nodes in the shared cluster are in the first state, the first node runs the shared file system and sends the first instruction message to the process management program.

[0094] The first instruction message is used to instruct the unloading of the shared file system.

[0095] S6032. In response to the first instruction message, the first node's running process management program unloads the shared file system to isolate the shared file system from failure.

[0096] Regarding S6031 and S6032 above, after confirming that all other nodes are in the first state, the first node can run the shared file system in kernel mode and send a first instruction message to the process manager in user mode to instruct it to unload the shared file system. Furthermore, in response to this message, the first node unloads the shared file system through the process manager (the first node can execute the umount command, which instructs the process manager to unload the shared file system), thereby enabling the first node to exit the shared cluster and achieving fault isolation between the first node's shared file system and other nodes.

[0097] For example, the kernel-mode fault notification module, based on feedback from the fault handling module and after confirming that all other nodes are in the first state, sends a first indication message to the user-mode fault isolation module. Correspondingly, the fault isolation module unloads the shared file system, thus achieving fault isolation.

[0098] Optionally, the first instruction message mentioned above can be a netlink message. Netlink is a communication mechanism used between kernel mode and user mode. Netlink messages can be sent by kernel modules or applications and can be multicast or unicast to one or more receivers. In this embodiment, the shared file management system in kernel mode is the sender of the netlink message, and the process management program in user mode is the receiver of the netlink message. This embodiment does not limit the specific content and form of the netlink message, as long as it can instruct the unloading of the shared file system.

[0099] It should be understood that the shared file system belongs to kernel mode and can directly access data on the disk. Therefore, the shared file system is a kernel module with the highest access rights to the computing device. Unlike other user-mode programs (such as browsers, games, etc.), the shared file system cannot be stopped by forcibly terminating it, because other user-mode processes may rely on the shared file system to access data on the disk. To enable the shared file system to be unloaded normally, the technical solution provided in this application embodiment also includes the following:

[0100] In one implementation, unloading the shared file system through the process management program in S6022 above can be achieved as follows: steps a-c:

[0101] Step a: The first node identifies the target process within the first node. The target process is the process within the first node that depends on the shared file system.

[0102] Step b: The first node's process management program stops the target process;

[0103] Step c: If the target process has stopped running, the first node runs the process management program to unload the shared file system.

[0104] The process manager can manage each process within a node, meaning it can determine the data accessed by each process based on its information. Therefore, the process manager can identify processes within its management whose target parameters point to shared data; these are processes that depend on the shared file system. The target parameters can be disk I / O, file descriptors, etc. Disk I / O refers to the instructions sent by the processor to the disk when a process accesses it, indicating the location of the data to be accessed. A file descriptor is an index returned by the processor to the process when it accesses a file, indicating the file being accessed. After identifying the target process, the process manager stops its execution. Once all target processes have stopped, the process manager can proceed with the unloading of the shared file system.

[0105] In this embodiment of the application, when the shared file system of the first node fails, the fault handling method provided in this embodiment further includes: configuring the read / write status parameter of the shared file system in the first node to a non-writable state. The non-writable state is used to indicate that the first node is prohibited from writing data to the shared data.

[0106] It should be understood that the purpose of a shared file system is to support the first node in reading and writing shared data. When the shared file system fails, the first node may generate erroneous data during the writing process, contaminating the shared data. Other nodes may also access this contaminated shared data, meaning that a failure of the first node can affect other nodes in the shared cluster. Therefore, upon detecting a failure, the first node configures the shared file system's read / write status parameter to a non-writable state, prohibiting the first node from writing data through the shared file system, thereby preventing further impact on the shared file system and shared data.

[0107] In some scenarios, after S603 mentioned above, if the shared file system recovers from a failure, such as Figure 9 As shown, the first node can also perform the following:

[0108] S604. After the shared file system of the first node recovers from a failure, the first node establishes communication connections with other nodes and releases the isolated shared file system.

[0109] S605. Configure the node status of other nodes configured on the first node to the second state.

[0110] The second state indicates that the shared file system on the node is functioning normally.

[0111] It should be understood that after the shared file system recovers from a failure, the first node can establish a communication connection by sending a message to other nodes to establish a connection, and unisolate the shared file system. It can also restore the node status of other nodes to the second state by configuring the target configuration file, ensuring that the node status of each node in the shared cluster is normal after the failure is recovered, and that the nodes can interact normally, thus ensuring the normal operation of the shared cluster.

[0112] Furthermore, regarding shared file system failure recovery, recovery can generally be achieved by restarting the file system, or by having operations and maintenance personnel perform repairs (e.g., modifying parameter configurations) and then restarting the file system. This allows the first node to rejoin the shared cluster after the failure is resolved, ensuring the performance of the shared cluster in processing business. For example, the first node can use the process management program to mount the shared file system (i.e., the first node executes the `mount` command, which instructs the process management program to mount the shared file system) to achieve a restart.

[0113] The above description illustrates the actions performed by the first node when the shared file system fails. The following describes the actions that the first node can perform when the shared file system is functioning correctly.

[0114] The technical solution provided in this application further includes: when it is determined that the communication connection between the first node and the second node among other nodes is broken, accessing shared data based on a first shared resource. The first shared resource includes the resources currently occupied by the second node for accessing the shared data.

[0115] As mentioned earlier, due to the characteristics of a shared file system, the shared resources used by a node are shared by all nodes in the shared cluster. Therefore, before a node's shared file system is notified to run, other nodes in the shared cluster will interact with that node to determine how the shared resources used by that node should be allocated (the shared file system can allocate resources based on each node's network conditions, memory usage, etc.). Therefore, once the first node determines that the communication connection with the second node in the shared cluster has been lost, it can preempt the shared resources occupied by the second node. It should be understood that by occupying the shared resources of a failed node, the impact of a node failure on the shared cluster's services can be reduced.

[0116] For example, taking a shared resource as a lock resource, when a second node accesses file A in the shared data, it configures a protective lock on file A, preventing other nodes from accessing file A. If the second node suddenly disconnects due to a failure and fails to release the protective lock on file A in time, the first node can acquire the protective lock and configure it to allow itself to continue accessing file A, thus achieving preemption of the shared resource. Alternatively, the first node can acquire the protective lock and delete it when it no longer needs to access file A, allowing other nodes to access file A normally. This avoids the situation where the second node goes offline abnormally, causing the protective lock to remain unreleased and preventing other nodes from accessing the data corresponding to the protective lock.

[0117] In some scenarios, when the shared file system is functioning correctly, such as... Figure 10 As shown, the shared file system can be terminated by following these steps.

[0118] S1001, When the shared file system of the first node is not faulty, an instruction is received.

[0119] This instruction is used to instruct the unloading of the shared file system on the first node.

[0120] S1002, in response to the instruction, releases the second shared resource by interacting with other nodes.

[0121] The second shared resource is the resource used by the first node to access the shared data.

[0122] S1003. If the second shared resource has been released on the first node, unload the shared file system on the first node.

[0123] Regarding S1001-S1003 above, when the shared file system on the first node is running normally, the first node can receive an instruction to unload the shared file system. This instruction can be a configuration instruction between maintenance personnel on the first node, or it can be received by the first node from other devices (such as the user-side devices of maintenance personnel) via the network, or it can be an instruction generated internally by the first node (such as stopping the operation of the shared file system within a pre-configured preset time period). This application embodiment does not limit the specific content of the instruction or the specific implementation method of the first node receiving the instruction.

[0124] Furthermore, the first node can interact with other nodes, allowing them to preempt the second shared resource. Other nodes can allocate the first node's shared resources based on the shared file system's functionality, thereby releasing the shared resources occupied by the first node. This enables the first node to properly unload the shared file system, thus taking the first node offline from the shared cluster.

[0125] Figure 11 This is a schematic diagram illustrating a fault handling process provided in an embodiment of this application. Figure 11 As shown, 1. During the operation of the shared file system, 2. a fault is detected. 3. Fault handling is performed (i.e., configuring other nodes to go offline in the first node as described in S602 above), 4. the fault notification module notifies the user-space fault isolation module. 5. The fault isolation module isolates the fault.

[0126] The fault handling method provided in this application, when a shared file system on a first node fails, configures the node status of other nodes configured on the first node to a first state, which indicates that the shared file system on the node has stopped running. By disconnecting the communication connection between the first node and other nodes, and configuring other nodes to go offline (i.e., the shared file system stops running) on ​​the first node, the first node can confirm that it is the only node in the shared file system. In this way, when the shared file system fails, the first node does not need to negotiate with other nodes or restart the entire node to achieve fault isolation of the shared file system. This effectively solves the problem in current solutions where a failure would reset the entire node, causing other services to be affected, and reduces the impact of shared file system failures.

[0127] Furthermore, when a shared file system malfunctions, promptly disconnecting the communication connection between the malfunctioning node and other nodes can prevent the erroneous data generated by the malfunctioning node from affecting other nodes. It also allows other nodes to detect the node malfunction in a timely manner, enabling them to seize shared resources and ensure business continuity. This approach can be applied to most scenarios.

[0128] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] In an exemplary embodiment, this application also provides a fault handling apparatus. This fault handling apparatus may be the aforementioned computing device or a processor within the computing device. The fault handling apparatus may include one or more functional modules for implementing the fault handling methods of the above method embodiments.

[0130] For example, Figure 12 This is a schematic diagram illustrating the composition of a fault handling device provided in an embodiment of this application. Figure 12 As shown, the fault handling device includes a configuration module 1201 and an isolation module 1202. The configuration module 1201 and the isolation module 1202 are interconnected.

[0131] Configuration module 1201 is used to disconnect communication with other nodes when the shared file system of the first node fails.

[0132] Configuration module 1201 is also used to configure the node status of other nodes configured on the first node to a first state; other nodes are nodes in the shared cluster other than the first node; the first state is used to indicate that the shared file system on the node has stopped running.

[0133] The isolation module 1202 is used to isolate the shared file system from failure when all other nodes in the shared cluster are in the first state.

[0134] In one possible implementation, the configuration module 1201 is specifically used to configure the node status of other nodes to the first state in the target configuration file; wherein, the target configuration file is a file that configures the running parameters of the shared file system, and the running parameters include the node status of each node in the shared cluster.

[0135] In another possible implementation, the shared file system runs in kernel mode, and the first node also includes a process management program running in user mode; the process management program is used to manage all processes in the node; the isolation module 1202 is specifically used to send a first instruction message to the process management program when all other nodes in the shared cluster are in the first state; the first instruction message is used to instruct the unloading of the shared file system; in response to the first instruction message, the shared file system is unloaded through the process management program.

[0136] In another possible implementation, the isolation module 1202 is specifically used to: determine the target process in the first node; the target process is a process in the first node that depends on the shared file system; terminate the target process through the process management program; and, if the target process has stopped running, unload the shared file system through the process management program.

[0137] In another possible implementation, the configuration module 1201 is further configured to set the read / write status parameter of the shared file system in the first node to a non-writable state; the non-writable state is used to indicate that the first node is prohibited from writing data to the shared data.

[0138] In another possible implementation, the configuration module 1201 is also used to establish communication connections with other nodes; configure the node status of other nodes configured on the first node to a second state; the second state is used to indicate that the shared file system on the node is operating normally.

[0139] In another possible implementation, the above-mentioned device further includes an access module 1203. The access module 1203 is used to access shared data based on a first shared resource when it is determined that the communication connection between the first node and the second node among other nodes is broken; the first shared resource includes the resources occupied by the second node currently accessing the shared data.

[0140] In another possible implementation, the above-mentioned device further includes: a receiving module 1204 and a sending module 1205; the receiving module 1204 is configured to receive an instruction when the shared file system of the first node is not faulty; the instruction is configured to instruct the unloading of the shared file system on the first node; the sending module 1205 is configured to release a second shared resource by interacting with other nodes in response to the instruction; the second shared resource is the resource occupied by the first node for accessing shared data; the unloading module is configured to unload the shared file system on the first node when the resource occupied by the first node for currently accessing shared data has been released.

[0141] In another possible implementation, each node is configured with a disk, and the disks on different nodes are used to store different parts of each piece of data in the shared data.

[0142] In an exemplary embodiment, this application also provides a computing device. Figure 13 This is a schematic diagram illustrating the composition of a computing device provided in an embodiment of this application. Figure 13 As shown, the computing device may include a processor 1301 and a memory 1302; the memory 1302 stores instructions executable by the processor 1301; when the processor 1301 is configured to execute instructions, the computing device implements the method described in the foregoing method embodiments.

[0143] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The computer-readable storage medium can also be an external storage device of the recovery device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the recovery device. Further, the computer-readable storage medium can include both internal storage units of the recovery device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the recovery device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0144] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to perform any of the fault handling methods described in the above embodiments.

[0145] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0146] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0147] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fault handling method, characterized in that, The first node is applied in a shared cluster, wherein each node in the shared cluster is configured with a shared file system, which is used by the node to access the shared data corresponding to the shared cluster; the shared file system runs in kernel mode, and the first node also includes a process management program running in user mode; The process management program is used to manage all processes in the node; the method includes: When the shared file system of the first node fails, the first node disconnects its communication connection with other nodes. Configuring the node status of other nodes configured on the first node to a first state includes: configuring the node status of the other nodes to the first state in the target configuration file; wherein, the target configuration file is a file that configures the running parameters of the shared file system, and the running parameters include the node status of each node in the shared cluster; the other nodes are nodes in the shared cluster other than the first node; the first state is used to indicate that the shared file system on the node stops running; When all other nodes in the shared cluster are in the first state, fault isolation of the shared file system includes: when all other nodes in the shared cluster are in the first state, running the shared file system to send a first instruction message to the process management program; the first instruction message is used to instruct the unloading of the shared file system; In response to the first instruction message, the process management program is run to unload the shared file system to isolate the shared file system from failure; wherein, running the process management program to unload the shared file system includes: Identify the target process in the first node; the target process is the process in the first node that depends on the shared file system. The target process is stopped by the process management program; If the target process has stopped running, run the process management program to unload the shared file system.

2. The method according to claim 1, characterized in that, When the shared file system of the first node fails, the method further includes: Configure the read / write status parameter of the shared file system in the first node to a non-writable state; the non-writable state is used to indicate that the first node is prohibited from writing data to the shared data.

3. The method according to claim 1, characterized in that, The method further includes: After the shared file system of the first node recovers from a failure, the first node establishes communication connections with other nodes. Configure the node status of other nodes configured on the first node to the second state; the second state is used to indicate that the shared file system on the node is operating normally.

4. The method according to claim 1, characterized in that, The method further includes: When it is determined that the communication connection between the first node and the second node among the other nodes is broken, the shared data is accessed based on the first shared resource; the first shared resource includes the resources occupied by the second node to access the shared data.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: When the shared file system on the first node is not faulty, an instruction is received; the instruction is used to instruct the unloading of the shared file system on the first node; In response to the instruction, the second shared resource is released by interacting with other nodes; the second shared resource is the resource occupied by the first node when accessing the shared data. If the first node has released the two shared resources, the shared file system on the first node is unloaded.

6. The method according to any one of claims 1-4, characterized in that, Each node is equipped with a disk, and the disks on different nodes are used to store different parts of each piece of data in the shared data.

7. A computing device, characterized in that, The computing device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the fault handling method as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Fencing shared cluster resources

    CA2719673A1

  • Shared disk based high availability cluster isolation method and system

    CN104077199A