dual control device

Abstract

The application discloses a double-control device, comprising a first control node, a second control node and an arbitration disk, wherein, in the case of network isolation between the first control node and the second control node, the arbitration disk receives a first reservation acquisition command and a second reservation acquisition command, the first reservation acquisition command is sent by the first control node, and the second reservation acquisition command is sent by the second control node; and a target node is determined through the arbitration disk. The problem that two control nodes cannot determine a master node in the case of network isolation is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a dual-control device. Background Technology

[0002] With the rise of big data, cloud computing, and high-performance applications, distributed storage systems have become the de facto back-end storage. In a dual-controller high-availability storage system, two control nodes work collaboratively to manage the back-end disk array. However, after network isolation, both control nodes believe themselves to be the master node and simultaneously provide read and write services, leading to data inconsistency.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a dual-control device system to at least solve the problem that the master node cannot be determined when two control nodes are isolated in a network.

[0005] This application provides a dual-control device, including: a first control node, a second control node, and an arbitration disk, wherein, when the first control node and the second control node are network isolated, the arbitration disk receives a first reservation acquisition command and a second reservation acquisition command, wherein the first reservation acquisition command is sent by the first control node, and the second reservation acquisition command is sent by the second control node; and a target node is determined through the arbitration disk.

[0006] In an exemplary embodiment, determining the target node through the arbitration disk includes: if the first control node successfully acquires the lock and the second control node fails to acquire the lock, the first control node and the second control node determine that the second control node is the target node; wherein, the first reservation acquisition command and the second reservation acquisition command are used to instruct the lock acquisition.

[0007] In one exemplary embodiment, after determining the target node through the arbitration disk, the method further includes: if the second control node determines that the second control node is the target node, the second control node releases the second storage resource.

[0008] In one exemplary embodiment, after the second control node releases the second storage resource, the method further includes: the second control node sending a release completion instruction to the first control node, wherein the release completion instruction is used to indicate that the second control node has released the second storage resource.

[0009] In one exemplary embodiment, after the second control node sends a release completion instruction to the first control node, the method further includes: the first control node creating a second storage resource service; and the first control node managing the second storage resource through the second storage resource service.

[0010] In one exemplary embodiment, the first control node further includes a first storage resource service, which is used to manage the first storage resources of the first control node.

[0011] In an exemplary embodiment, after the first control node manages the second storage resource through the second storage resource service, the method further includes: after the first control node determines that the second control node has recovered from the fault, the first control node sends a reservation release command to the arbitration disk, wherein the reservation release command is used to request the release of the second storage resource.

[0012] In one exemplary embodiment, after the first control node manages the second storage resource through the second storage resource service, the method further includes: after the second control node recovers from a fault, the second control node sends a third reservation command acquisition command to the arbitration disk, wherein the third reservation command is used to indicate lock-grabbing; if the second control node successfully grabs the lock, the second control node creates a third storage resource service and manages the second storage resource through the third storage resource service.

[0013] In an exemplary embodiment, before the arbitration disk receives the first reservation acquisition command and the second reservation acquisition command, the method further includes: the first control node periodically sending a first heartbeat message to the second control node through a first thread; and the second control node periodically sending a second heartbeat message to the first control node through a second thread.

[0014] In one exemplary embodiment, the method further includes: in the event of an abnormal heartbeat message between the first control node and the second control node, the first control node sends the first reservation acquisition command, and the second control node sends the second reservation acquisition command.

[0015] This application provides a dual-control device. When the first control node and the second control node are network isolated, the arbitration disk receives a first reservation acquisition command and a second reservation acquisition command. The first reservation acquisition command is sent by the first control node, and the second reservation acquisition command is sent by the second control node. The target node is determined through the arbitration disk. This achieves the goal of determining the master node by the arbitration disk when the first control node and the second control node are network isolated. Therefore, it solves the technical problem in related technologies where the master node cannot be determined when two control nodes are network isolated, achieving the effect of the master node providing write services after network isolation, ensuring data consistency. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This application provides a schematic diagram of a dual-control device process.

[0018] Figure 2 Network diagram provided for embodiments of this application Figure 1 ;

[0019] Figure 3 Network diagram provided for embodiments of this application Figure 2 ;

[0020] Figure 4 This is an interactive schematic diagram of heartbeat detection provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the data flow of a dual-control device provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of resource switching provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the fault switching process provided in an embodiment of this application;

[0024] Figure 8 A flowchart of fault reversal provided for embodiments of this application;

[0025] Figure 9 This is a schematic diagram of resource switching and reverting provided in an embodiment of this application;

[0026] Figure 10 A schematic diagram illustrating architectural changes provided for embodiments of this application. Detailed Implementation

[0027] The keywords are explained below:

[0028] Distributed storage systems distribute data across multiple independent storage devices, enabling data sharing and access via network connections, thereby improving the reliability, scalability, and performance of data storage.

[0029] OSS: Object Storage Service, is a storage device used in distributed storage systems. It is responsible for the final storage of data and also provides functions such as data replication, recovery, and rebalancing.

[0030] Dual Controller Node: A dual controller node is a storage device design that includes two independent controllers, each capable of independently controlling the operation of the entire storage device. Dual controller nodes are typically used in high-availability storage systems to provide higher data reliability and availability.

[0031] Node1: The first control node within the dual-controller frame.

[0032] Node1: The second control node within the dual-control unit frame.

[0033] Hard drive sharing: Both Node1 and Node2 can access all 100 hard drives within the enclosure.

[0034] RESA: Divide the hard drives into 50 partitions based on the slot number (SLOT number), which are used by Node1 by default.

[0035] RESB: Based on the SLOT number, allocate the remaining 50 hard drives, which are used by Node2 by default.

[0036] Data reconstruction: In a distributed storage system, when some data is lost or becomes inaccessible due to node or disk failure, the system uses redundant data (replicas or erasure code check blocks) to regenerate and restore the lost data on other healthy nodes.

[0037] NVMe Reservations: NVMe 1.3 and above protocols define comprehensive reservation functionality, including atomic operations such as registration, acquisition, and release, providing a technical foundation for building a lightweight, low-latency arbitration solution that does not require third-party arbitration devices.

[0038] Arbitration disk: A shared disk that uses Small Computer System Interface (SCSI) reservation (SCSI-3 PR).

[0039] With the increasing prevalence of Non-Volatile Memory Express (NVMe) storage arrays, there is a need for a native, efficient, and reliable arbitration solution based on the NVMe protocol. The NVMe protocol itself defines powerful reservation functions that can be used to control access permissions of multiple hosts to a namespace, which provides a foundation for building a lightweight, low-latency arbitration solution that does not require a third-party arbitration device.

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0041] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0042] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] This application provides a dual-control device, including: a first control node, a second control node, and an arbitration panel, such as... Figure 1 As shown, it includes the following steps:

[0044] Step S102: When the first control node and the second control node are network isolated, the arbitration disk receives a first reservation acquisition command and a second reservation acquisition command, wherein the first reservation acquisition command is sent by the first control node and the second reservation acquisition command is sent by the second control node.

[0045] In an exemplary embodiment, before the arbitration disk receives the first reservation acquisition command and the second reservation acquisition command, the method further includes: the first control node periodically sending a first heartbeat message to the second control node through a first thread; and the second control node periodically sending a second heartbeat message to the first control node through a second thread.

[0046] The first and second heartbeat messages mentioned above can be User Datagram Protocol (UDP) messages.

[0047] In one exemplary embodiment, the method further includes: in the event of an abnormal heartbeat message between the first control node and the second control node, the first control node sends the first reservation acquisition command, and the second control node sends the second reservation acquisition command.

[0048] like Figure 2 As shown, the first control node Node1 and the second control node Node2 are two redundant storage controllers, interconnected via a network heartbeat line, and both connected to a shared backend NVMe disk array via a PCIe switch.

[0049] Shared NVMe arbitration disk: A specific NVMe namespace in the backend NVMe disk array, pre-configured as an arbitration disk, used to store system status information and serve as a carrier for arbitration locks.

[0050] The heartbeat management module is deployed on two control nodes and is used to periodically send and detect heartbeat signals through the heartbeat network to determine the liveness status of the other controller.

[0051] The NVMe reservation management module is deployed on two controllers and is used to initiate NVMe reservation commands to the shared NVMe arbitration disk to acquire, hold, or release the arbitration lock.

[0052] like Figure 3 As shown, there is a direct Ethernet link between the two motherboards within the box. During deployment, the IP address of Node1 is configured as 192.168.10.1, and the IP address of Node2 is configured as 192.168.10.2.

[0053] For the Ethernet direct connection between motherboards, UDP packets are sent and received. When a disconnection is detected, a failover is triggered; when the link is restored, a switchback is triggered. Detection can be achieved within seconds.

[0054] like Figure 4 The diagram shows the interaction between the first control node and the second control node for heartbeat detection.

[0055] After the high availability information distributed file system (idfs-ha) module starts, it listens on the UDP port. The first and second control nodes start threads specifically for sending heartbeat messages (the first control node sends the first heartbeat message, and the second control node sends the second heartbeat message). The message content contains resource information and node information, and the sending interval is 200 milliseconds (configurable).

[0056] The first and second control nodes start threads to receive heartbeat messages sent by the peer (the first control node's thread receives the second heartbeat message, and the second control node receives the first heartbeat message). If a heartbeat message is received from the peer, the time of receipt is recorded and marked as network link restored. If no heartbeat message is received from the peer after a preset interval, it is marked as network link broken. Abnormal received heartbeat messages, including incorrect format (not two parts) or incorrect content (incorrect node name and resource name), are ignored and logged.

[0057] When two nodes detect a network link failure, they both assume the other is faulty and attempt to take over the other's OSS. To prevent split-brain issues, an arbitration mechanism is needed to determine which node survives. The surviving node must wait until the other node stops its OSS before it can start the other node's OSS.

[0058] NVMe disk arbitration is used to prevent two controllers from using the same hard drive group resources simultaneously.

[0059] Step S102: Determine the target node using the arbitration disk.

[0060] The characteristic of dual-controller nodes is that each control node can access the disks of the other control node, using an NVMe disk arbitration mechanism. When one control node fails, the OSS service switches to the other controller. Figure 5 As shown, the solid line represents the data flow when the dual control is working normally, and the dashed line represents the data flow when a certain control node fails.

[0061] Although both control nodes can see all the hard drives, the same drive letter does not correspond to the same hard drive. Therefore, it is necessary to divide the shared disks into two groups based on the disk SLOT number to ensure disk group exclusivity. Figure 6 As shown, under normal conditions, each of the two control nodes uses one set of resources without conflict. When one control node fails, the other control node automatically takes over both sets of resources.

[0062] like Figure 7The diagram shows the failover process. Node1 and Node2 detect node failures at the same time. Node1 and Node2 simultaneously send NVME_RESERVATION_ACQUIRE reservation commands to the shared NVMe arbitration disk (including the first reservation acquisition command and the second reservation acquisition command mentioned above).

[0063] In an exemplary embodiment, determining the target node through the arbitration disk includes: if the first control node successfully acquires the lock and the second control node fails to acquire the lock, the first control node and the second control node determine that the second control node is the target node; wherein, the first reservation acquisition command and the second reservation acquisition command are used to instruct the lock acquisition.

[0064] The above lock is an arbitration lock provided by the arbitration disk. The node that successfully acquires the lock is the live node (also known as the master node), and the node that fails is the non-live node (also known as the backup node or the faulty node, corresponding to the target node mentioned above).

[0065] Assume the surviving node is Node1 (the first control node), and the non-surviving node is Node2 (the second control node). For the non-surviving node Node2, the storage resource OSS is released, and the surviving node Node1 takes over the storage resource released by the non-surviving node.

[0066] In one exemplary embodiment, after determining the target node through the arbitration disk, the method further includes: if the second control node determines that the second control node is the target node, the second control node releases the second storage resource.

[0067] In one exemplary embodiment, after the second control node releases the second storage resource, the method further includes: the second control node sending a release completion instruction to the first control node, wherein the release completion instruction is used to indicate that the second control node has released the second storage resource.

[0068] In one exemplary embodiment, after the second control node sends a release completion instruction to the first control node, the method further includes: the first control node creating a second storage resource service; and the first control node managing the second storage resource through the second storage resource service.

[0069] In one exemplary embodiment, the first control node further includes a first storage resource service, which is used to manage the first storage resources of the first control node.

[0070] In an exemplary embodiment, after the first control node manages the second storage resource through the second storage resource service, the method further includes: after the first control node determines that the second control node has recovered from the fault, the first control node sends a reservation release command to the arbitration disk, wherein the reservation release command is used to request the release of the second storage resource.

[0071] In one exemplary embodiment, after the first control node manages the second storage resource through the second storage resource service, the method further includes: after the second control node recovers from a fault, the second control node sends a third reservation command acquisition command to the arbitration disk, wherein the third reservation command is used to indicate lock-grabbing; if the second control node successfully grabs the lock, the second control node creates a third storage resource service and manages the second storage resource through the third storage resource service.

[0072] like Figure 8 The diagram shows the fault handover flowchart. When the faulty node is detected to have recovered, Node1 sends the NVME_RESERVATION_RELEASE command (corresponding to the reserved release command mentioned above) to the NVME arbitration disk, and then preempts the lock after the faulty node recovers. The process of the recovered node Node2 taking over the resources: Node2 sends the NVME_RESERVATION_ACQUIRE command (corresponding to the third reserved acquisition command) to the NVME disk, Node2 successfully acquires the lock, and Node2 takes over the storage resources.

[0073] like Figure 9 As shown, when both control nodes are operating normally, the first control node uses hard disk group A, and the second control node uses hard disk group B. When the first control node fails, the second control node takes over hard disk group A. When the first control node recovers, the second control node releases hard disk group A, and the first control node takes over hard disk group A again. When the second control node fails, the first control node takes over hard disk group B. When the second control node recovers, the first control node releases hard disk group B, and the second control node takes over hard disk group B again.

[0074] The architecture has changed from share-nothing to share-disk, such as Figure 10As shown, OSS (Operational Security Service) is a disk management service. Due to changes in hardware architecture, it requires failover and rollback based on node status changes. When nodes are working normally, the two controllers each manage their own disks and provide data storage services to the cluster as two different OSS sets. When one controller fails, the other controller creates a new OSS process and quickly takes over the disks of the failed controller. At this time, both OSS processes run on the same controller. When the failed controller recovers, one OSS set is then switched back to the recovered node. For dual-controller hardware platforms, disk resources are grouped and allocated. This allows OSS processes to be created on the non-failed controller when a single controller fails, taking over the disk groups on the failed controller. When the failed controller recovers, the taken-over disk groups are released, and OSS processes are created on the recovered controller. This achieves seamless failover and rollback without data reconstruction during single-controller failures, and second-level failover / failback (failover / failback IO drop-to-zero time less than 10 seconds), avoiding data migration and reducing the impact of failures on business performance.

[0075] This application proposes a method and system for failover and recovery based on NVME disk arbitration on dual-controller nodes. Both controllers of the dual-controller storage node can access each other's disks. By utilizing NVME disk arbitration, when one controller fails, the partner controller in that node will automatically start the unavailable OSS process of the failed node and quickly take over the fault. This achieves no data reconstruction in the event of a single controller failure and second-level failover in the event of a single controller failure (failover / failback IO drop to zero time is less than 10 seconds).

[0076] It enables controller fault switching and controller fault recovery switching functions, avoiding data migration due to controller failure and data re-migration after fault recovery, reducing the performance impact caused by data migration competing for resources, and improving the reliability and security of the product.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0078] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described server management system embodiments.

[0079] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described server management system embodiments when it is run.

[0080] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0081] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described server management system embodiments.

[0082] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described server management system embodiments.

[0083] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0084] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] The server management system provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A dual-control device, characterized in that, include: The system comprises a first control node, a second control node, and an arbitration disk. When the first control node and the second control node are network isolated, the arbitration disk receives a first reservation acquisition command and a second reservation acquisition command, wherein the first reservation acquisition command is sent by the first control node and the second reservation acquisition command is sent by the second control node; The target node is determined by the arbitration disk.

2. The device according to claim 1, characterized in that, Determining the target node through the arbitration disk includes: If the first control node successfully acquires the lock while the second control node fails to acquire the lock, the first control node and the second control node determine that the second control node is the target node. The first reserved acquisition command and the second reserved acquisition command are used to instruct the lock grabbing.

3. The device according to claim 1 or 2, characterized in that, After determining the target node through the arbitration disk, the method further includes: If the second control node determines that the second control node is the target node, the second control node releases the second storage resource.

4. The device according to claim 3, characterized in that, After the second control node releases the second storage resource, the process also includes: The second control node sends a release completion instruction to the first control node, wherein the release completion instruction is used to indicate that the second control node has released the second storage resource.

5. The device according to claim 3, characterized in that, After the second control node sends a release completion command to the first control node, the process further includes: The first control node creates a second storage resource service; The first control node manages the second storage resource through the second storage resource service.

6. The device according to claim 5, characterized in that, The first control node further includes a first storage resource service, which is used to manage the first storage resources of the first control node.

7. The device according to claim 5, characterized in that, After the first control node manages the second storage resource through the second storage resource service, the process further includes: After the first control node determines that the second control node has recovered from the fault, the first control node sends a reserved release command to the arbitration disk, wherein the reserved release command is used to request the release of the second storage resource.

8. The device according to claim 3, characterized in that, After the first control node manages the second storage resource through the second storage resource service, the process further includes: After the second control node recovers from the fault, the second control node sends a third reserved command acquisition command to the arbitration disk, wherein the third reserved command is used to indicate lock grabbing; If the second control node successfully acquires the lock, the second control node creates a third storage resource service and manages the second storage resource through the third storage resource service.

9. The device according to claim 1, characterized in that, Before the arbitration disk receives the first reservation acquisition command and the second reservation acquisition command, the following is also included: The first control node periodically sends a first heartbeat message to the second control node through a first thread; The second control node periodically sends a second heartbeat message to the first control node through a second thread.

10. The device according to claim 7, characterized in that, Also includes: In the event of an abnormal heartbeat message between the first control node and the second control node, the first control node sends the first reservation acquisition command, and the second control node sends the second reservation acquisition command.

Images