Method and system for implementing high-availability cluster heartbeat link based on serial communication

Abstract

The application relates to a high-availability cluster heartbeat link implementation method and system based on serial communication, and belongs to the technical field of high-availability cluster management. The method solves the problems of fault node misjudgment, half-duplex bus unable to simultaneously transmit and receive and unable to perform one-way fault positioning in the existing Ethernet scheme. The method comprises the following steps: analyzing a cluster heartbeat general configuration file to obtain configuration information; configuring a network communication link based on network heartbeat link configuration information, separating and initializing a first serial port capable of receiving external data and a second serial port for externally transmitting data based on serial heartbeat link configuration information, establishing two full-duplex independent serial communication links, transmitting and receiving heartbeat messages with a peer node through the network / serial communication link, and judging the reachability of the node based on the connectivity of the network and serial communication links. The two serial links can simultaneously transmit and receive, one-way positioning of faults can be realized, and the misjudgment of fault nodes can be reduced.

Description

Technical Field

[0001] This invention relates to the field of high availability cluster management technology, and in particular to a method and system for implementing a high availability cluster heartbeat link based on serial communication. Background Technology

[0002] A high-availability (HA) cluster system consists of multiple servers, each of which is a node. In a cluster system, the heartbeat mechanism enables fault detection and node status monitoring through the periodic exchange of status information between nodes.

[0003] Existing heartbeat mechanisms include: Ethernet heartbeat: Based on the UDP protocol, lightweight messages are periodically sent. If no message is received within a timeout period, the node is considered faulty. When a switch failure or network congestion causes the heartbeat to be interrupted, the cluster may misjudge it as a node failure, triggering unnecessary resource switching, or even causing a split-brain scenario. Existing cluster software (such as Corosync and Pacemaker) supports multiple links, but only Ethernet links, and cannot achieve redundancy at the physical medium level. In addition, the heartbeat mechanism judges based on "overall connectivity," and cannot distinguish between "local transmission failure" and "peer reception failure," lacking fine-grained awareness of unidirectional link failures, making troubleshooting difficult.

[0004] Serial heartbeat: Two nodes' serial ports are directly connected via a half-duplex RS-485 bus, using time-division multiplexing for heartbeat communication. Node A periodically sends heartbeat messages to the serial port, while node B continuously listens for the heartbeat and resets its timeout timer. If no heartbeat is received within the timeout period, node B determines that node A has failed (possibly due to a fault in node B itself). This approach cannot distinguish between "local transmission failure" and "received data failure," lacking fine-grained awareness of unidirectional link failures, leading to difficulties in troubleshooting. Furthermore, while RS-485 serial ports offer physical independence, they suffer from severe isolation: their code is highly coupled to the serial port hardware, lacking a standardized communication protocol framework, making integration into modern cluster frameworks like Corosync difficult, resulting in poor scalability for multi-node clusters. Furthermore, serial port heartbeats typically operate as independent modules, failing to form a dynamic complementary "master / backup / concurrent" collaborative mechanism with network heartbeats. They cannot automatically take over when network heartbeats fail and lack automatic reconnection and redundancy capabilities. Heartbeats are permanently interrupted after a loose physical connection or device reset, requiring manual intervention. They cannot form a heterogeneous redundancy system with network heartbeats. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a method and system for implementing a high-availability cluster heartbeat link based on serial communication, in order to solve the problems of misjudgment of faulty nodes, inability of half-duplex buses to transmit and receive simultaneously, and inability to locate faults in one direction in existing Ethernet solutions.

[0006] On one hand, embodiments of the present invention provide a method for implementing a high-availability cluster heartbeat link based on serial communication, including the following steps: Obtain and parse the cluster heartbeat configuration file to get the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes; Based on the network heartbeat link configuration information, the network communication link of the node is configured, and based on the serial port heartbeat link configuration information, the two serial ports of the node are separately initialized and configured to establish two full-duplex independent serial communication links. The two serial ports include a first serial port for receiving external data and a second serial port for sending data to the outside. Choose to send and receive heartbeat messages with the peer node via network communication link and / or serial communication link; The reachability of a node is determined based on the connectivity of the network communication link and the serial communication link.

[0007] Based on the further improvement of the above method, the cluster heartbeat general configuration file is obtained and parsed to obtain the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes, including: Traverse the address configuration items in the general configuration file of the cluster heartbeat. If the address configuration item includes a preset prefix identifier, it is determined to be the serial port heartbeat link configuration information; if it does not include the preset prefix identifier, it is determined to be the network heartbeat link configuration information of the cluster node.

[0008] A further improvement to the above method involves performing separate initialization configurations for the two serial ports of a node based on the serial heartbeat link configuration information, including: Remove the preset prefix identifier from the address configuration item to obtain the remaining configuration information of the address configuration item; The first delimiter is determined from the remaining configuration information. The information before the first delimiter is parsed as the first serial port configuration information, and the information after the first delimiter is parsed as the second serial port configuration information. Obtain the parameters required for serial communication from the cluster heartbeat configuration file; The first serial port is configured based on the first serial port configuration information and the parameters required for serial communication, and the second serial port is configured based on the second serial port configuration information and the parameters required for serial communication.

[0009] Further improvements to the above method also include: When a first error identifier is captured during data transmission or reception via the serial heartbeat link, the current serial port is automatically closed and initialization is re-executed to rebuild the serial heartbeat link. If the rebuild fails, periodic retries are performed. The first error identifier indicates a serious error caused by the failure of underlying communication resources.

[0010] Further improvements to the above method also include: When a second error identifier is captured during data transmission or reception, the current frame data is discarded and the next frame is received. The second error identifier indicates a data error.

[0011] Further improvements to the above method include sending and receiving heartbeat messages with the peer node via a serial communication link, including: Create independent serial port transceiver threads. The serial port sending thread sends heartbeat messages to the first serial port of the peer node through the second serial port, and the serial port receiving thread receives heartbeat messages sent by the peer node through the second serial port through the first serial port.

[0012] Based on further improvements to the above method, the frame structure of the heartbeat message includes: a preamble field, a checksum field, a payload length field, and a payload field; wherein the preamble field adopts a 16-byte consecutive 0xAA pattern.

[0013] A further improvement to the above method involves determining the reachability of a node based on the connectivity of the network communication link and the serial communication link, including: The connectivity status of the network communication link and the serial communication link is statistically analyzed. When both the network communication link and the serial communication link are disconnected, the node is determined to be faulty; when at least one of the network communication link and the serial communication link is connected, the node is determined to be reachable.

[0014] Further improvements to the above method also include: Once the network communication link and serial heartbeat link are configured and running, a link status structure is created, which includes a link type identifier, a configuration completion identifier, link status information, and link parameters. Perform any of the following operations based on the link state structure: Choose to send and receive heartbeat messages with the peer node via network communication link and / or serial communication link; Node reachability determination; Determine the link where the fault is located.

[0015] On the other hand, embodiments of the present invention provide a high-availability cluster heartbeat link implementation system based on serial communication, including: The configuration parsing module is used to obtain and parse the general configuration file for cluster heartbeat, and to obtain the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes; The multi-link scheduling module is used to configure the network communication links of the node based on the network heartbeat link configuration information, and to perform separate initialization configuration of the two serial ports of the node based on the serial port heartbeat link configuration information, and to establish two full-duplex independent serial communication links, wherein the two serial ports include a first serial port for receiving external data and a second serial port for sending data to the outside. The serial communication module is used to select between sending and receiving heartbeat messages with the peer node via a network communication link and / or a serial communication link. The heartbeat timeout detection module is used to determine the reachability of a node based on the connectivity of the network communication link and the serial communication link.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: (1) Configure serial port links and network links to achieve redundancy at the physical medium level. Based on the connectivity of serial port links and network links, the node fault is judged comprehensively. The cluster will not misjudge the node fault due to network device failure, triggering unnecessary resource switching and preventing split-brain.

[0017] (2) By configuring the two serial ports of the node as a sending serial port and a receiving serial port respectively, two independent full-duplex serial communication links are established. When a fault occurs in both sending and receiving, it is possible to distinguish between "the sending end fails and the receiving end cannot receive" and "the sending end fails and the receiving end cannot receive", thus realizing unidirectional fault judgment.

[0018] (3) When a physical abnormality occurs in the serial port device itself, the mechanism automatically detects and rebuilds the connection of the underlying device without manual intervention.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0021] Figure 1 This is a flowchart of the high-availability cluster heartbeat link implementation method based on serial communication in Embodiment 1 of the present invention; Figure 2This is a schematic diagram showing the connection between nodes A and B via a crossover serial cable in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the system architecture for implementing a high-availability cluster heartbeat link based on serial communication in Embodiment 1 of the present invention. Detailed Implementation

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0023] Example 1, A specific embodiment of the present invention discloses a method for implementing a high-availability cluster heartbeat link based on serial communication, such as... Figure 1 As shown, it includes the following steps: S11: Obtain and parse the cluster heartbeat general configuration file to get the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes.

[0024] In some embodiments, the cluster heartbeat general configuration file is obtained and parsed to obtain the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes, including: Traverse the address configuration items in the general configuration file of the cluster heartbeat. If the address configuration item includes a preset prefix identifier, it is determined to be the serial port heartbeat link configuration information; if it does not include the preset prefix identifier, it is determined to be the network heartbeat link configuration information of the cluster node.

[0025] S12: Configure the network communication link of the node based on the network heartbeat link configuration information, and perform separate initialization configuration of the two serial ports of the node based on the serial port heartbeat link configuration information to establish two full-duplex independent serial communication links, wherein the two serial ports include a first serial port for receiving external data and a second serial port for sending data to the outside.

[0026] In some embodiments, the two serial ports of a node are separately initialized based on the serial heartbeat link configuration information, including: Remove the preset prefix identifier from the address configuration item to obtain the remaining configuration information of the address configuration item; The first delimiter is determined from the remaining configuration information. The information before the first delimiter is parsed as the first serial port configuration information, and the information after the first delimiter is parsed as the second serial port configuration information. Obtain the parameters required for serial communication from the cluster heartbeat configuration file; The first serial port is configured based on the first serial port configuration information and the parameters required for serial communication, and the second serial port is configured based on the second serial port configuration information and the parameters required for serial communication.

[0027] The implementation process of steps S11 and S12 is described in detail below.

[0028] Specifically, in the nodelist list of the cluster heartbeat configuration file, each node configures the address information of each link through the ringN_addr field (i.e., the address configuration item). For serial heartbeat links, the prefix "serial:" is used for identification, followed by the configuration information of the first and second serial ports. The first serial port acts as a read device, receiving data sent from outside, while the second serial port acts as a write device, sending data to outside. In this case, the configuration information of the first serial port is the read device path, and the configuration information of the second serial port is the write device path, separated by semicolons. For example, "serial: / dev / ttyS0; / dev / ttyS1;" indicates that this node uses / dev / ttyS0 as the read device and / dev / ttyS1 as the write device for serial heartbeat communication. Figure 2 As shown.

[0029] Configuration example: nodelist{ node{ ... name: hostA ring0_addr: 192.168.1.10 ring1_addr: serial: / dev / ttyS0; / dev / ttyS1; } } Node A's ring0_addr is configured with an IP address (e.g., 192.168.1.10) for network heartbeat, and ring1_addr is configured with a serial port device path (e.g., serial: / dev / ttyS0; / dev / ttyS1;) for serial port heartbeat.

[0030] The specific configuration file parsing and configuration are usually handled in the following steps: 1) Traverse address configuration items: Traverse all ringN_addr configuration items (N is 0, 1, 2...).

[0031] 2) Prefix matching identification: Examine the prefix of the addr string to identify the link type: For ring0_addr=192.168.1.10, since the string does not start with serial: and conforms to the IPv4 address format, it is identified as a network link and the IP address 192.168.1.10 is directly resolved. If the string starts with "serial:", it is identified as a serial port link. The content after the colon is extracted, and the remaining string is split according to the semicolon ";". The serial port read device path (such as / dev / ttyS0) and write device path (such as / dev / ttyS1) are parsed out respectively, and the parameters required for serial communication (including optional baud rate parameters, etc.) are extracted.

[0032] This logic, based on fixed prefix matching and delimiter segmentation, enables automated identification and parameter extraction of heterogeneous link configurations.

[0033] Specifically, the parameters required for serial communication may include enable receive, data bits, parity, stop bits, operating mode, read timeout, and baud rate. Among these, baud rate: if the baud rate is not explicitly specified in the configuration string (e.g., serial:115200:...), the default value (e.g., 115200) will be used. Data bits / stop bits / parity: use the global default configuration or the default values ​​defined in the configuration file.

[0034] Enable reception: Set the CLOCAL and CREAD flags to ensure that the program can receive data from the serial port.

[0035] Data bits: 8 data bits (CS8) Parity check: Do not perform parity check (clear the PARENB flag). Stop bit: 1 stop bit (clears the CSTOPB flag) Operating mode: RAW mode. In this mode, the receiving device directly delivers all bytes of data received from the serial port to the application layer without performing additional processing such as line editing or echoing. Read timeout: Setting VTIME to 0 and VMIN to 0 indicates non-blocking read; the read call returns immediately, returning 0. Baud rate: The default setting is 115200, but different baud rates such as 9600, 4800, 2400, and 460800 can be selected according to the actual scenario. Configuration example: serial_config { # 1. Serial port device file path device_path: / dev / ttyS0; / dev / ttyS1; # 2. Baud Rate # Optional values: 9600, 19200, 38400, 57600, 115200, 460800 baud_rate: 115200 # 3. Data Bits # Default is 8 data_bits: 8 # 4. Stop Bits # 1 or 2, usually 1 stop_bits: 1 # 5. Parity Check # none, even, odd # Recommendation: none (no checksum) to reduce overhead and rely on application-layer CRC checksum. parity: "none" # 6. Working Mode mode: "raw" # 7. Read Timeout Control (termios VTIME / VMIN) # Effect: read() returns immediately, or 0 if the buffer is empty, and does not block. read_timeout: vmin: 0 vtime: 0 # Optional: Advanced control flags flags: # CLOCAL: Ignore modem control line (auto-enabled) clocal: true # CREAD: Enable receiver (auto-enabled) cread: true } 3) Based on the identification results, the parsed specific parameters are passed to the corresponding type of link for initialization.

[0036] For example: For ring0_addr=192.168.1.10 Recognition: If the string does not contain the serial: prefix and conforms to the IPv4 format, it is identified as a network link.

[0037] deal with: IP address resolved: 192.168.1.10.

[0038] Bind / connect to this IP.

[0039] Create a UDP / TCP socket.

[0040] Register with epoll.

[0041] For ring1_addr=serial: / dev / ttyS0; / dev / ttyS1; Recognition: If the string starts with "serial:", it is identified as a serial port link.

[0042] deal with: Remove the prefix: remove serial:, the remaining part is / dev / ttyS0; / dev / ttyS1;.

[0043] Parsing path: Find the first semicolon;.

[0044] The first part, / dev / ttyS0, resolves to a read device (RX).

[0045] The latter part, / dev / ttyS1, resolves to a write device (TX).

[0046] Initialize serial port parameters: First, load the serial_config block and read each field sequentially: First, parse the device_path string by finding the semicolon; split it into the read device path (e.g., / dev / ttyS0) and the write device path (e.g., / dev / ttyS1); then parse the numeric parameters baud_rate (e.g., 115200), data_bits (e.g., 8), stop_bits (e.g., 1), and the enumeration parameter parity (e.g., "none"); then check if the string parameter mode is "raw" to determine whether to disable line editing and echoing functions; then read vmin (set to 0) and vtime (set to 0) in the nested structure read_timeout to configure non-blocking read mode; finally, check the clocal and cread boolean values ​​in the flags structure to confirm whether to automatically enable ignoring modem signals and enable the receive function. Finally, encapsulate all parsed and verified parameters into a unified serial port configuration structure for use when parsing serial port heartbeats.

[0047] During the specific configuration, when initializing the serial port link, the read device and write device should be opened separately: open(" / dev / ttyS0", O_RDWR | O_NOCTTY | O_NONBLOCK) -> Get the read FD (file descriptor of the read device).

[0048] open(" / dev / ttyS1", O_RDWR | O_NOCTTY | O_NONBLOCK) -> Get the write FD (file descriptor of the write device).

[0049] The above process assumes a successful call to the `open` function, which returns a file descriptor (FD) informing the kernel to configure the ttyS0 / ttyS1 device. This FD is then used to call the POSIX-based termios function to modify the parameters required for serial communication within the kernel.

[0050] After configuring the two serial communication links for nodes A and B as described above, connect the two physical serial ports of nodes A and B respectively using an RS-232 point-to-point crossover cable to establish an independent transmit / receive channel based on the full-duplex crossover cable. Figure 2 As shown in the diagram, the read and write devices are two serial ports on the same node. The read device only reads data, and the write device only writes data. They are connected to two serial ports on another node via serial cables. Correspondingly, the write device on this node is connected to the read device on the other node. That is, if this node wants to send data, it writes to the write device, and the other node reads data from the read device. From the perspective of one serial cable, the data flow is unidirectional; the other serial cable is the opposite. Therefore, two serial cables are used to achieve full-duplex communication for data transmission and reception.

[0051] In summary, to address the issue that existing Ethernet heartbeats with multiple links (such as UDP resource links) typically share the same physical transmit and receive channels, while half-duplex buses (such as RS485) cannot transmit and receive simultaneously, this invention employs RS-232 point-to-point full-duplex cross-connection to achieve complete isolation of the physical transmit and receive channels of each link.

[0052] This design brings a key capability that existing technologies cannot achieve—one-way fault location and degraded operation. When only the write device fails, the local end can still sense the survival of the peer through the read device (i.e., it can still read data from the read device, but cannot write data to the write device), avoiding the cluster from mistakenly judging node failure due to one-way communication interruption; similarly, when only the read device fails, the local end can still inform the peer that it is alive. This breaks the binary judgment limitation of traditional heartbeats: "if it works, all are connected; if it fails, all are disconnected."

[0053] S13: Select to send and receive heartbeat messages with the peer node via network communication link and / or serial communication link.

[0054] In some embodiments, sending and receiving heartbeat messages with a peer node via a serial communication link includes: Create independent serial port transceiver threads. The serial port sending thread sends heartbeat messages to the first serial port of the peer node through the second serial port, and the serial port receiving thread receives heartbeat messages sent by the peer node through the second serial port through the first serial port.

[0055] In practice, the strategy can be adjusted to either a priority-based single-transmission strategy or a concurrent strategy that sends messages through both links. Specifically, based on the different priorities of the network communication link and the serial communication link, the link with the higher priority can be selected for sending heartbeat messages, or both the network communication link and the serial communication link can be used. The network link is set to have a higher priority than the serial communication link.

[0056] The frame structure used in the heartbeat message includes: a preamble field, a checksum field, a payload length field, and a payload field; wherein the preamble field uses 16 consecutive bytes of 0xAA mode.

[0057] Specifically, to address the issue of data loss due to interference in serial communication, where traditional methods discard the entire batch of data and re-search if the preamble is lost, leading to slow frame synchronization recovery, heartbeat delays, or even timeouts, this invention designs a highly reliable serial frame protocol to ensure reliable data transmission over the serial link. This frame format is used to encapsulate and parse heartbeat messages. The frame structure design is shown in Table 1. Table 1

[0058] Frame synchronization mechanism: (1) Preamble design: The preamble uses a 16-byte consecutive 0xAA (i.e., binary 10101010) pattern. The reason for choosing 0xAA is that this pattern generates a regular square wave signal on the serial port line, which is convenient for the receiver to perform clock recovery and baud rate adaptation; the probability of 16 consecutive 0xAA bytes appearing in normal data is extremely low, which can effectively distinguish the frame boundary (each serial heartbeat frame and the random data in the middle are distinguished by the preamble) from random data. Random data refers to non-transmitted data, such as the fluctuating signal that may appear on the serial port line before and after sending a heartbeat frame. These are randomly generated. If they are not distinguished, it will be impossible to locate which data is desired. Therefore, the start of the serial heartbeat frame is determined by the preamble frame, and the end of the serial heartbeat frame is determined by the frame length field.

[0059] (2) Fault-tolerant design of the frame synchronization algorithm: The receiving end uses a byte-by-byte matching method to find the preamble. The frame synchronization method includes: reading byte data, counting the number of consecutive preamble characters, and determining that frame synchronization is successful when the number of consecutive preamble characters reaches a preset threshold. If it does not reach the threshold, breakpoint matching is performed. In view of the characteristics of serial communication being prone to packet loss, the breakpoint matching mechanism designed in this algorithm is as follows: read a batch of byte data, count the number of consecutive preamble characters at the end, if the end contains some consecutive 0xAA (e.g., 5), but not 16, the algorithm records the 5 matched bytes as an offset. When the next batch of data arrives, it continues to match the remaining preamble based on the offset from the 6th byte until 16 consecutive 0xAA are found to determine that synchronization is successful. This algorithm supports the rapid recovery of frame synchronization from any position in the data stream, even if some preamble is lost in the middle, which significantly reduces the impact of serial communication errors on heartbeat continuity.

[0060] (3) Verification Mechanism: The CRC16 checksum algorithm is used to verify the entire frame (after the checksum field is cleared). When constructing a frame, the sending end first sets the checksum field to zero, then calculates the CRC16 checksum value for the entire frame, and fills the result into the checksum field. After receiving the frame, the receiving end saves the value of the checksum field, clears it to zero, recalculates the CRC16 checksum value for the entire frame, and compares it with the saved original checksum value. If they are inconsistent, it is determined that an error has occurred during data transmission, and the current frame is discarded. The CRC16 checksum algorithm can detect all single-bit errors, double-bit errors, and most burst errors, and has good protection against common interferences in serial communication.

[0061] In summary, by designing the above frame structure, serial communication is susceptible to interference, which can lead to data loss. If the preamble is lost, traditional methods will discard the entire batch of data and search for it again, resulting in slow frame synchronization recovery, heartbeat delay, or even timeout.

[0062] In practical implementation, existing cluster frameworks (such as Corosync / Kronosnet) only support socket-based network event-driven operations. Serial ports, as character devices, cannot be directly integrated into their event loops, leading to integration difficulties and low efficiency in multi-link scheduling. Based on this, this invention designs a unified thread model: threads are divided into two types: independent transmit / receive threads for serial port links and a main receive thread.

[0063] The serial port link uses an independent send / receive thread model. The serial port receive thread continuously reads data from the reading device, and after receiving a complete frame, the send / receive thread model passes the data to the main receive thread through an internal Socketpair; the serial port send thread retrieves data from the message queue (described later) and sends it through the write device, realizing full-duplex communication with physical channel isolation.

[0064] The Main Receiver Thread is the "brain" and "unified scheduling center" of the entire heartbeat system. Its core responsibility is to shield the differences in the underlying physical media. During the initialization phase, it establishes a Socketpair channel between the receiving thread and the Main Receiver Thread, registers the internal Socketpair receiver file descriptor of the serial port link and the Socket descriptor of the network link into the epoll instance, and then uses the epoll event-driven model to uniformly listen for data arrival events from all heterogeneous links.

[0065] In summary, the unified thread model enables seamless integration of different physical media links at the event-driven layer, converting serial port data into Socket events. This allows the upper-layer cluster framework to transparently manage serial port heartbeats without modifying the original network event processing logic, eliminating the overhead of multi-threaded polling and achieving unified event processing with high concurrency and low latency.

[0066] Specifically, the communication mechanism is as follows: Create a socketpair.

[0067] Start a serial port receive thread: read data from ttyS0 -> parse the frame -> write to the socketpair write end.

[0068] The main receiving thread listens on the read end of the socketpair.

[0069] In practical implementation, serial port bandwidth is much lower than Ethernet (e.g., 115200bps vs 1Gbps). If serial port transmission is blocked or the queue is backed up, the heartbeat response thread will be suspended, which will affect the transmission of high-priority network heartbeats and cause a system-wide heartbeat avalanche. Based on this, this invention designs an independent message queue buffer mechanism between the serial port transmission thread and the upper-layer heartbeat producer to decouple the data production and consumption processes, thereby improving the system's throughput and stability.

[0070] Design principles: The message queue uses a ring buffer structure and maintains two core pointers: Write Tail: Points to the next free location in the buffer that can be written to.

[0071] Read Head: Points to the next data location to be read in the buffer.

[0072] Working mechanism: (1) Enqueue Operation: When data needs to be sent via the serial port link, the message enqueue interface is called. The enqueue operation first involves the producer acquiring a mutex lock, writing the data to the write pointer position of the circular buffer, updating the write pointer, and then notifying the consumer thread (serial port sending thread) that new data is available to send through a condition variable. Finally, the mutex lock is released. If the queue is full, the current message (i.e., data) is discarded according to the strategy and logged to avoid blocking the producer thread and affecting the normal operation of other links.

[0073] (2) Dequeue Operation: The serial port sending thread calls the message dequeue interface in a loop. The dequeue operation first involves the consumer acquiring a mutex lock, checking if the queue is empty, and if it is empty, waiting for new data to arrive using a condition variable; if the queue is not empty, it reads data from the read pointer position, updates the read pointer, and finally releases the mutex lock and returns the read data and its length. The sending thread then actually sends the data out through the serial port device.

[0074] In summary, designing a message queue buffering mechanism can achieve the following beneficial effects. (1) Non-blocking enqueue: When the queue is full, the message is discarded directly instead of blocking the producer, ensuring that the heartbeat processing flow will not be blocked due to the slow serial port sending speed, and ensuring the normal operation of the network and other links.

[0075] (2) Independent Queue Instances: Each serial port link has an independent message queue instance, and different links do not affect each other. When the queue of a certain link is backlogged, it will not affect the message processing of other links. Two nodes share one serial port link, and the links of different nodes have independent message queue instances. In addition, there can be multiple serial port links between two nodes (high redundancy), and these multiple serial port links are also independent. Note: A serial port link actually requires two serial cables for physical connection, and n serial port links require 2n serial cables for connection. (3) Size limit: The total capacity of the queue is preset to ensure sufficient buffer space to cope with sudden traffic and prevent unlimited memory growth. Each message record in the queue contains complete load data and its length information to ensure that the original message can be completely recovered when dequeuing.

[0076] (4) Thread safety: Mutex locks protect concurrent access to the head and tail pointers of the queue. Condition variables are used to make consumer threads enter a waiting state when the queue is empty, avoiding busy waiting and wasting CPU resources.

[0077] In summary, a message queue buffering mechanism decouples slow links from fast business logic. By employing a non-blocking "discard when full" strategy, reliable delivery of a single serial port heartbeat is sacrificed in exchange for the absolute smooth operation of the producer thread. This ensures that the core cluster heartbeat scheduling logic is not blocked or delayed due to low-speed peripherals, thus guaranteeing the stability of the entire cluster control plane.

[0078] In specific implementation, this invention defines two types of heartbeat messages: a heartbeat request message (PING), which is sent periodically by the heartbeat thread to detect the liveness status of the peer node; and a heartbeat response message (PONG), which is replied to immediately by the receiving end after receiving the PING message to confirm the reachability of the peer node.

[0079] Table 2 shows the general Ping / Pong message format (application layer payload): Table 2

[0080] Unified heartbeat processing procedure: When a node receives a PING heartbeat request message from the peer, it needs to select the appropriate sending method to reply with a PONG message based on the current link type (stored in the link state structure, which will be described in detail later): if the current link is a serial link, the PONG message is written to the serial link's message queue to await transmission; if the current link is a network link, it is sent directly via a socket. This unified processing flow allows the upper-layer heartbeat logic to be unaware of the underlying physical medium, achieving transparent processing for various link types.

[0081] S14: Determine the reachability of the node based on the connectivity of the network communication link and the serial communication link.

[0082] In some embodiments, determining the reachability of a node based on the connectivity of the network communication link and the serial communication link includes: The connectivity status of the network communication link and the serial communication link is statistically analyzed. When both the network communication link and the serial communication link are disconnected, the node is determined to be faulty; when at least one of the network communication link and the serial communication link is connected, the node is determined to be reachable.

[0083] In practice, the link status is first determined by heartbeat detection, and the connectivity status of the link is recorded based on the detection results.

[0084] Network link detection: Periodically send UDP heartbeat packets to the peer IP via the Socket interface and listen to the Socket receive buffer. Once a PONG response is received from the peer, immediately calculate the instantaneous round-trip time (RTT) and update the smoothed delay value of the link using the exponential moving average (EMA) algorithm. If the smoothed delay is below the threshold and a specified number of consecutive PONGs are received, the network link status is determined to be UP; if no response is received within consecutive timeouts, the network link status is determined to be DOWN.

[0085] Serial link detection: Bidirectional Detection: The system defaults to a bidirectional full-duplex detection mechanism. Node A periodically sends encapsulated heartbeat PING messages to Node B through the serial port's write device channel. Simultaneously, a separate serial port receiving thread continuously listens for the raw byte stream returned by Node B from Node A's read device channel. After frame synchronization and CRC verification, the parsed valid PONG messages are passed to the main receiving thread via an internal Socketpair. The main receiving thread calculates the round-trip delay of the current link and uses an exponential moving average algorithm to calculate the smoothed delay value of the link. If the smoothed delay value is lower than the preset heartbeat timeout threshold and a sufficient number of PONG responses have been received consecutively, the serial port link status is determined to be UP. If no PONG response is received within several consecutive heartbeat cycles (e.g., 3 times) or the smoothed delay value is higher than the preset heartbeat timeout threshold, the serial port link status is determined to be DOWN. This mechanism effectively prevents frequent link state switching due to occasional packet loss, ensuring the stability of link state determination.

[0086] Simultaneously, it switches to one-way detection mode: after consecutive failures of bidirectional detection to determine link failure, the system automatically downgrades to one-way detection mode to maintain basic connectivity awareness. At this time, only the serial port's read device channel is enabled for passive listening, and heartbeat requests are no longer actively sent. The receiving thread performs frame synchronization and CRC verification on the read raw byte stream. Once any valid peer heartbeat message (whether it is PING or PONG) is parsed and the message arrives within the recently set short timeout window (such as 3-5 seconds), the link status is immediately determined to be UP. This sacrifices some bidirectional handshake accuracy in exchange for the link survival awareness capability in communication-restricted or one-way failure scenarios.

[0087] The aforementioned heartbeat PING / PONG packet mechanism and format are existing open-source content, and the link status determination is also an existing technology, so they will not be described in detail.

[0088] Furthermore, because this invention employs a design that separates the read and write devices, the direction of the fault can be determined, wherein: Read direction failure: The heartbeat from the other end can be received, but the other end cannot receive the heartbeat from our side; Write direction failure: The other end can receive the heartbeat from our side, but our side cannot receive the heartbeat from the other end.

[0089] Bidirectional failure: Communication is impossible in both directions.

[0090] After determining the connectivity status of the network communication link and the serial communication link through the above tests, it is necessary to determine whether the node is reachable based on the connectivity status of the network communication link and the serial communication link. If at least one link is in the UP state, the node is considered reachable; if all links are in the DOWN state, the node is considered unreachable.

[0091] In some embodiments, it also includes: When a first error identifier is captured during data transmission or reception via the serial heartbeat link, the current serial port is automatically closed and initialization is re-executed to rebuild the serial heartbeat link. If the rebuild fails, periodic retries are performed. The first error identifier indicates a serious error caused by the failure of underlying communication resources.

[0092] When a second error identifier is captured during data transmission or reception, the current frame data is discarded and the next frame is received. The second error identifier indicates a data error.

[0093] In practical implementation, this invention designs an intelligent error handling mechanism that distinguishes different types of errors and takes corresponding measures.

[0094] Error classification and handling: (1) Critical Errors (Return -1): These include serial port device read / write failures, device file descriptor errors, etc. These errors usually indicate a problem with the serial port device itself, requiring the device to be closed and the link reinitialized to recover. Serial port device read / write failures refer to whether the open, receive, and write functions called for reading / writing the device return normal. "Device file descriptor error" means that the underlying operating system kernel reports irreversible damage or failure to the file handle state.

[0095] (2) Common errors (returns -2): These include CRC check failures, payload data length exceeding limits, etc. This type of error indicates that the current frame data is corrupted, but the serial port device itself is still working normally. Simply discard the current frame and continue receiving the next frame. In addition, this error code is also returned when it is detected that a thread needs to exit, to notify the upper-layer thread to terminate normally.

[0096] Automatic recovery mechanism: When a critical error (-1) is returned, the serial port transceiver thread enters an automatic recovery process: First, it shuts down the current serial port device (the local serial port read / write device shuts down the corresponding device based on read or write failure), and then attempts to reinitialize the serial port link (reopening the device and reconfiguring the serial port parameters). If reinitialization is successful, a recovery success log is recorded and normal packet reception continues; if reinitialization fails, it waits 1 second and retryes until the link is actively deleted by the user or reinitialized successfully. The entire recovery process requires no manual intervention, ensuring business continuity.

[0097] Existing technologies (such as link switching methods based on weighted calculations) focus on "selecting the best" among multiple links, but when a physical-level anomaly occurs in an underlying link, it often has to abandon that link. The intelligent error handling mechanism of this invention focuses on the self-healing of underlying devices. By distinguishing between critical errors (device-level anomalies) and ordinary errors (data-level interference), when encountering critical errors such as serial port device file descriptor failures, it does not switch links, but automatically performs a reconstruction process on the current link, involving device shutdown, reopening, and parameter reconfiguration. This restores link availability without manual intervention, fundamentally improving system availability rather than simply escaping faults between links.

[0098] In some embodiments, it also includes: Once the network communication link and serial heartbeat link are configured and running, a link status structure is created, which includes a link type identifier, a configuration completion identifier, link status information, and link parameters. Perform any of the following operations based on the link state structure: Choose to send and receive heartbeat messages with the peer node via network communication link and / or serial communication link; Node reachability determination; Determine the link where the fault is located.

[0099] In practical implementation, this invention designs a unified link abstraction mechanism that treats serial port links and network links equally, so that the upper-layer cluster software does not need to distinguish between link types.

[0100] Specifically, a unified link state structure was constructed: each link maintains a unified state structure, including link type identifier (whether it is a serial link, a storage link, or a network link), configuration completion identifier, link state information (including connectivity, latency, statistical count, etc.), and link parameters (including priority, heartbeat interval, heartbeat timeout, etc.).

[0101] The link type identifier and link parameters are initially configured in the configuration file. After configuration, a configuration completion identifier is generated (only for normal configuration loading), and then link status information (including connectivity, latency, statistical counts, etc.) is recorded according to the running status.

[0102] By using a link type identifier field, links from different physical media (including serial port devices (jacks) and network devices (NICs)) can be handled uniformly, with type-based processing only required when access to the physical device is needed. This design greatly reduces system complexity and improves maintainability and scalability.

[0103] A unified link-state structure plays a crucial role in system operation, enabling core routing decisions, dynamic fault management, and adaptive policy adjustment. Specifically, a unified link-state structure can be applied to the following scenarios: 1. Heartbeat packet sending decision and multimodal concurrent scheduling Priority and load balancing strategy: Active links can be dynamically selected based on the link priority and connectivity status recorded in the link status structure. For example, low-latency network links are preferred by default; when the network link connectivity is false or the latency exceeds the threshold, it is automatically downgraded and switched to a backup serial port link.

[0104] Concurrent transmission strategy: In the mode that supports multiple active links, heartbeat messages are sent simultaneously to multiple links with an up status based on the link parameters, thereby improving the arrival probability by utilizing the physical independence of different media.

[0105] Dynamic frequency adjustment: If a link is found to have large latency fluctuations or an increase in packet loss rate in the statistics, the heartbeat interval of the link can be temporarily increased to reduce congestion according to the preset strategy, or the default frequency can be restored after stabilization.

[0106] 2. Node survival determination and intelligent anti-shake mechanism Weighted voting / majority rule: Instead of relying solely on the timeout of a single link, it combines the connectivity status of all active links to determine node reachability. For example, if a network link times out but the serial port link receives packets N times consecutively, the system determines that the node is still alive (it may be a network failure rather than a node crash). This avoids single-point misjudgment and thus suppresses unnecessary resource switching.

[0107] Jitter prevention: Statistical counts (such as the number of state transitions in the past 10 seconds) are used to identify transient faults. If the link state frequently jumps between UP and DOWN (jitter), the current judgment result will be temporarily locked or a cooldown period will be triggered until the state stabilizes, preventing frequent split-brain or resource migration of the cluster due to network micro-flickers.

[0108] Differentiated timeout tolerance: Different heartbeat timeout times can be configured for different link types. For example, serial links are limited by physical baud rate, so the timeout time is set to be longer; network links have shorter timeout times. The corresponding timeout threshold is applied based on the source of the latest received packet, improving the accuracy of the judgment.

[0109] 3. Link health monitoring and automatic self-healing trigger Fault root cause localization: By comparing the status of network links and serial links, the system can assist maintenance personnel or automated scripts in quickly locating the fault domain. For example, if the network is down but the serial port is up, the fault is limited to the network device or network cable; if both are down, it is most likely that the node itself is faulty or the other end is completely down.

[0110] Serial port device self-healing monitoring: For serial port links, continuously monitor the configuration completion flag and the underlying driver status. Once a serial port device reset is detected (e.g., unexpected loss of connectivity but configuration still exists), and the configuration completion flag is true, it can automatically attempt to reopen the device handle and reset the baud rate configuration without manual intervention or restarting the entire cluster service.

[0111] Statistical early warning: Based on statistical counts (such as the number of CRC errors and the number of retransmissions), when the error rate exceeds the threshold, an early warning of link quality degradation is issued, prompting an inspection of the physical line or electromagnetic interference.

[0112] 4. Configure hot updates and dynamic policy adjustments Dynamic parameter activation: Link parameters in memory can be modified via the management interface (such as raising the priority of a backup link to the highest level or adjusting the heartbeat interval). Because a unified state structure is maintained in memory, these changes can be immediately reflected in the scheduling algorithm, enabling hot switching of strategies.

[0113] New link dynamic discovery: When a new physical interface (such as a newly inserted USB serial adapter) is detected, a new state structure can be scanned and initialized, automatically added to the heartbeat pool, and its participation in heartbeat transmission can be determined according to priority configuration, so as to achieve plug-and-play high availability expansion.

[0114] Example 2, Another embodiment of the present invention discloses a high-availability cluster heartbeat link implementation system based on serial communication, thereby implementing the high-availability cluster heartbeat link implementation method based on serial communication in Embodiment 1. The specific implementation methods of each module are as described in the corresponding description in Embodiment 1, including: The configuration parsing module is used to obtain and parse the general configuration file for cluster heartbeat, and to obtain the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes; The multi-link scheduling module is used to configure the network communication links of the node based on the network heartbeat link configuration information, and to perform separate initialization configuration of the two serial ports of the node based on the serial port heartbeat link configuration information, and to establish two full-duplex independent serial communication links, wherein the two serial ports include a first serial port for receiving external data and a second serial port for sending data to the outside. The serial communication module is used to select between sending and receiving heartbeat messages with the peer node via a network communication link and / or a serial communication link. The heartbeat timeout detection module is used to determine the reachability of a node based on the connectivity of the network communication link and the serial communication link.

[0115] like Figure 3The diagram shows the system architecture. The configuration parsing module and heartbeat timeout detection module are located in the application layer. The multi-link scheduling module is located in the transmission abstraction layer. The serial communication module is located in the driver abstraction layer. The serial port is located in the system's hardware layer.

[0116] The configuration parsing module parses and obtains network link information and serial port link information, which is then configured by the multi-link scheduling module of the transmission abstraction layer. The multi-link scheduling module then selects a network communication link and / or a serial communication link, generates a heartbeat message, and the serial communication module sends and receives the heartbeat message, returning the result to the heartbeat timeout detection module for detection.

[0117] The link state structure described above is constructed and maintained by the multi-link scheduling module.

[0118] Furthermore, the transmit / receive thread management is located in the transport abstraction layer and is used for network and serial port transmit / receive thread management. This layer also manages network and serial port links. The driver abstraction layer returns different error codes when encountering different types of errors, and the serial port transmit / receive thread management module of the transport abstraction layer automatically restores the serial port link.

[0119] Frame synchronization and verification after heartbeat message reception are implemented by the frame synchronization module and verification module of the driver abstraction layer.

[0120] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the corresponding computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0121] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for implementing a high-availability cluster heartbeat link based on serial communication, characterized in that, Includes the following steps: Obtain and parse the cluster heartbeat configuration file to get the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes; Based on the network heartbeat link configuration information, the network communication link of the node is configured, and based on the serial port heartbeat link configuration information, the two serial ports of the node are separately initialized and configured to establish two full-duplex independent serial communication links. The two serial ports include a first serial port for receiving external data and a second serial port for sending data to the outside. Choose to send and receive heartbeat messages with the peer node via network communication link and / or serial communication link; The reachability of a node is determined based on the connectivity of the network communication link and the serial communication link.

2. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 1, characterized in that, Obtain and parse the cluster heartbeat configuration file to get the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes, including: Traverse the address configuration items in the general configuration file of the cluster heartbeat. If the address configuration item includes a preset prefix identifier, it is determined to be the serial port heartbeat link configuration information; if it does not include the preset prefix identifier, it is determined to be the network heartbeat link configuration information of the cluster node.

3. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 2, characterized in that, Based on the serial heartbeat link configuration information, the two serial ports of the node are separately initialized and configured, including: Remove the preset prefix identifier from the address configuration item to obtain the remaining configuration information of the address configuration item; The first delimiter is determined from the remaining configuration information. The information before the first delimiter is parsed as the first serial port configuration information, and the information after the first delimiter is parsed as the second serial port configuration information. Obtain the parameters required for serial communication from the cluster heartbeat configuration file; The first serial port is configured based on the first serial port configuration information and the parameters required for serial communication, and the second serial port is configured based on the second serial port configuration information and the parameters required for serial communication.

4. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 1, characterized in that, Also includes: When a first error identifier is captured during data transmission or reception via the serial heartbeat link, the current serial port is automatically closed and initialization is re-executed to rebuild the serial heartbeat link. If the rebuild fails, periodic retries are performed. The first error identifier indicates a serious error caused by the failure of underlying communication resources.

5. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 4, characterized in that, Also includes: When a second error identifier is captured during data transmission or reception, the current frame data is discarded and the next frame is received. The second error identifier indicates a data error.

6. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 1, characterized in that, Sending and receiving heartbeat messages with the peer node via a serial communication link, including: Create independent serial port transceiver threads. The serial port sending thread sends heartbeat messages to the first serial port of the peer node through the second serial port, and the serial port receiving thread receives heartbeat messages sent by the peer node through the second serial port through the first serial port.

7. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 1, characterized in that, The frame structure used in the heartbeat message includes: a preamble field, a checksum field, a payload length field, and a payload field; wherein the preamble field uses 16 consecutive bytes of 0xAA mode.

8. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 1, characterized in that, Determining node reachability based on the connectivity of the network communication link and the serial communication link includes: The connectivity status of the network communication link and the serial communication link is statistically analyzed. When both the network communication link and the serial communication link are disconnected, the node is determined to be faulty; when at least one of the network communication link and the serial communication link is connected, the node is determined to be reachable.

9. The method for implementing a high-availability cluster heartbeat link based on serial communication according to claim 1, characterized in that, Also includes: Once the network communication link and serial heartbeat link are configured and running, a link status structure is created, which includes a link type identifier, a configuration completion identifier, link status information, and link parameters. Perform any of the following operations based on the link state structure: Choose to send and receive heartbeat messages with the peer node via network communication link and / or serial communication link; Node reachability determination; Determine the link where the fault is located.

10. A high-availability cluster heartbeat link implementation system based on serial communication, characterized in that, include: The configuration parsing module is used to obtain and parse the general configuration file for cluster heartbeat, and to obtain the network heartbeat link configuration information and serial port heartbeat link configuration information of the cluster nodes; The multi-link scheduling module is used to configure the network communication links of the node based on the network heartbeat link configuration information, and to perform separate initialization configuration of the two serial ports of the node based on the serial port heartbeat link configuration information, and to establish two full-duplex independent serial communication links, wherein the two serial ports include a first serial port for receiving external data and a second serial port for sending data to the outside. The serial communication module is used to select between sending and receiving heartbeat messages with the peer node via a network communication link and / or a serial communication link. The heartbeat timeout detection module is used to determine the reachability of a node based on the connectivity of the network communication link and the serial communication link.

Images