Cross-layer collaborative dual-machine hot standby control method and system for industrial control systems

By using a controller to monitor TCP heartbeats and process status in a dual-machine hot standby system, establishing persistent transaction records, and performing embedded database recovery and business process verification, the problem of inaccurate node status judgment in existing technologies is solved. This achieves more reliable business recovery and network address configuration, reduces risks, and ensures business continuity.

CN123123356APending Publication Date: 2026-11-17SIYUE INTELLIGENCE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611614603.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing dual-machine hot standby systems lack unified execution constraints during node failure detection, database recovery, and network address switching, making it difficult to accurately determine node status. This can lead to inconsistent database, business process, and network resource statuses, and may result in data version rollback or simultaneous service provision by both nodes after the primary node recovers.

Method used

By acquiring TCP application heartbeat messages and process status between the primary and standby nodes through the controller of the standby node, persistent takeover transaction records are established, embedded database recovery and business process verification are performed, the sequential execution of virtual Internet Protocol address configuration and external announcements is ensured, recovery anomalies within the primary node are handled first, and data readiness records are generated to confirm the readiness status of business processes and network devices.

Benefits of technology

It improves the reliability and recoverability of dual-machine hot standby control, reduces the risk of business recovery due to database loading errors or inconsistent network conditions, reduces unnecessary cross-node switching and data version rollback, and ensures the continuity of business services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123356A_ABST
    Figure CN123123356A_ABST
Patent Text Reader

Abstract

This application discloses a cross-layer collaborative dual-machine hot standby control method and system for industrial control systems, relating to the field of industrial control. The method includes: when a TCP application heartbeat anomaly is detected on the master node, performing intra-node recovery on the master node and executing a takeover action through the standby node; based on the database copy corresponding to the master node, performing embedded database recovery on the standby node and generating a data ready record associated with the takeover transaction record; configuring the virtual Internet Protocol address and sending protocol announcements externally according to the dependency relationship between the business process and the corresponding virtual Internet Protocol address; and starting the business client in the standby node. This can prevent databases that do not meet preset verification conditions from entering the subsequent business startup stage; reduce the risk of business recovery being affected by loading an incorrect database. Recovery execution or continuation of recovery processing is based on persistent records, improving the recoverability of the takeover process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of industrial control, specifically to a cross-layer collaborative dual-machine hot standby control method and system for industrial control systems. Background Technology

[0002] Dual-machine hot standby technology in industrial control systems involves dual-node fault detection, database copy recovery, business process collaborative control, and operating system network address switching. It can be applied to manufacturing execution systems (MES), equipment monitoring systems, edge computing nodes, and other computer systems that need to collaboratively transfer data services, network access addresses, and business processes between dual nodes.

[0003] A dual-machine hot standby system typically includes a primary node (also known as the first node, denoted as HP) running the operating system and a standby node (also known as the second node, denoted as OP). Upper-level systems such as the Manufacturing Execution System (MES) access business service programs through a fixed Virtual Internet Protocol Address (VIP). Existing dual-machine hot standby solutions usually determine the operating status based on node heartbeats, transfer the VIP upon detecting a failure, and start the business program via scripts or process management tools.

[0004] The aforementioned master / standby status control and network address switching can be implemented through high availability software and operating system network interfaces. For example, high availability software (such as Keepalived) supports master / standby status transitions, VIP configuration and deletion, gratuitous Address Resolution Protocol (ARP) advertisements, network interface card (NIC) or process status tracking, consecutive failure thresholds, and status callback mechanisms based on the Virtual Router Redundancy Protocol (VRRP). The operating system's Internet Protocol Helper (IP Helper) interface can be used to create unicast IP addresses and query duplicate address detection status.

[0005] To ensure the availability of business data after the standby node takes over, the dual-machine hot standby system also protects data through database replication and replica recovery mechanisms. For example, continuous database replication tools (such as Litestream) can perform continuous replication and replica recovery of the SQLite database. The database can be continuously replicated to an object storage service compatible with the Simple Storage Service (S3) protocol via Litestream, which can be deployed on the standby node or other designated nodes. Embedded database (such as SQLite database) file replication can also be used as an alternative implementation. The above data protection mechanisms provide a database replica for node takeover and recovery of the original primary node.

[0006] However, when the aforementioned mechanisms handle node election, database recovery, network address configuration, or process startup separately, and no unified execution constraints are established between the database recovery result, the actual database loading status and VIP availability of the business process (e.g., the ThinkingTop software server program, referred to as TKT Server), and the startup of the business client (e.g., the ThinkingTop software client, referred to as TKT Client), a successful return from a single interface or process makes it difficult to determine whether the entire business chain is ready to provide external services. Specifically, the following problems exist: 1. If the OP does not receive the HP's Transmission Control Protocol (TCP) application heartbeat within the specified time, it can only be determined that the application heartbeat messages are continuously missing, which is not enough to prove that the HP system has stopped running. Abnormal business processes, network interruptions between nodes, abnormal message forwarding, or abnormal receiving threads may all produce the same observation result.

[0007] 2. In a dual-node architecture without an independent arbitration component, when a network partition occurs between the HP and the OP, the OP cannot strictly distinguish between a complete HP shutdown and an HP still running but with a communication link interrupted, solely based on heartbeat timeouts. If automatic takeover is blindly executed without differentiation, it is possible for both nodes to simultaneously run business processes, hold the same VIP, or write to their local SQLite instances.

[0008] 3. Based solely on the exit status of the database recovery process, it is difficult to determine whether the target database file has been generated and meets the requirements for openability, structural integrity, schema version, and recovery point; based solely on the existence of the business process or the service port being in a listening state, it is difficult to determine whether the business process has completed initialization and loaded the database corresponding to this takeover; based solely on the VIP creation interface returning a success, it is difficult to determine whether the address has passed the duplicate address detection, and it is also difficult to determine whether the network device has updated the mapping between the VIP and the node link address.

[0009] 4. When database recovery, business process control, and network address configuration are executed by different components, the lack of clear stage dependencies, process status records, and failure recovery mechanisms may lead to inconsistent states of the database, business processes, and network resources.

[0010] 5. If the active role is automatically reacquired after the original master node is restored but before data synchronization is complete, it may cause the data version accessed by the business to be rolled back, or there may be a situation where two nodes attempt to provide services at the same time. Summary of the Invention

[0011] To address the aforementioned issues, this application proposes a cross-layer collaborative dual-machine hot standby control method for industrial control systems, the method comprising: The controller of the standby node obtains TCP application heartbeat messages, inter-node heartbeats and related process status between the master node and the standby node, and performs intra-node recovery for the master node when it is determined that the master node has a TCP application heartbeat abnormality, and performs takeover action through the standby node when the preset takeover trigger conditions are met. For the aforementioned takeover action, a persistent takeover transaction record with a unique identifier is established; Based on the database replica corresponding to the master node, the embedded database is restored on the standby node, and the corresponding first restoration result is jointly verified. After the verification is passed, a data ready record associated with the takeover transaction record is generated. Based on the data ready record, the actual database loaded by the business process on the standby node is verified, and the virtual Internet Protocol address is configured and a protocol announcement is sent to the outside according to the dependency relationship between the business process and the corresponding virtual Internet Protocol address. Start the service client in the backup node, and once the service client reaches the preset running state, confirm that the takeover action is complete.

[0012] In one example, the controller of the standby node obtains TCP application heartbeat packets between the primary and standby nodes, inter-node heartbeats, and related process statuses, specifically including: Based on the controller set in the backup node, the system receives TCP application heartbeat messages sent by the business processes in the master node through a TCP connection at a preset period. The TCP application heartbeat message includes at least one of the following: sending node identifier, business process instance identifier, monotonically increasing sequence number, and sending time. Record the receiving time and monotonically increasing sequence number of the most recent valid TCP application heartbeat message, and collect the heartbeat and related process status between the master node.

[0013] In one example, upon determining that the master node has a TCP application heartbeat anomaly, intra-node recovery is performed on the master node, and upon meeting preset takeover trigger conditions, a takeover action is performed through the backup node, specifically including: If the controller of the backup node fails to receive a valid TCP application heartbeat message within several consecutive preset periods, the master node is marked as a suspected fault state. In a preset fault confirmation window, the controller continues to detect the TCP application heartbeat message, the heartbeat between nodes, and the status of related processes. When the controller resumes receiving valid TCP application heartbeat messages and the number of consecutive confirmations reaches a preset number, the suspected fault state is revoked. If the master node is in the suspected fault state and the heartbeat between the nodes is normal, it is determined to be a candidate event for business process abnormality. The controller of the master node performs intra-node recovery for the master node and verifies the corresponding second recovery result. If the verification fails, the takeover determination process is initiated. If the master node is in the suspected fault state, and the heartbeat between the nodes is abnormal, and the preset abnormal duration and number of consecutive failures are reached, then the takeover determination process is initiated. Regarding the takeover determination process, in availability-first mode, the takeover action is automatically executed through the backup node, performing a virtual Internet Protocol address conflict check and attempting to confirm the activity status of the master node through storage probes or network probes. If no address conflict is detected or it is determined that the master node is not providing business services to the outside, the takeover action continues. In data security-first mode, the backup node enters a takeover confirmation state and continues to execute the takeover action after receiving a takeover confirmation instruction sent by the administrator.

[0014] In one example, for the takeover action, a persistent takeover transaction record with a unique identifier is established, specifically including: For the takeover action, a takeover transaction record is established and persisted locally. The takeover transaction record includes: a unique transaction identifier, fault type, initiation time, current stage, execution result status of each stage, completed action stack, and key metadata required for recovery. Idempotency of operations is achieved based on the unique identifier of the transaction. If the controller of the backup node exits abnormally during the execution of the takeover action, the controller of the backup node is restarted, and subsequent processing is performed based on the persistent takeover transaction records.

[0015] In one example, based on the database replica corresponding to the primary node, an embedded database is restored on the standby node, and a joint verification is performed on the corresponding first restoration result. Upon successful verification, a data-ready record associated with the takeover transaction record is generated, specifically including: Stop the business processes, business clients, and replication processes that occupy the embedded database on the backup node, and move the corresponding original database files and associated files to the isolation directory; The new database file is restored to a temporary path based on the database copy corresponding to the master node stored in the object storage; For the new database file, a joint verification is performed using the recovery process exit status, recovery summary, existence and non-emptiness of temporary database files, database read-only open result, integrity check result, key tables, schema version, and recovery point. If the first recovery result is successful, the database access path is made to point to the new database file by updating the configuration file or changing the symbolic link; Generate a data ready record associated with the takeover transaction record; the data ready record includes: database recovery point, database file fingerprint, schema version, integrity check result, and database access path.

[0016] In one example, based on the data-ready record, the actual database loaded by the business process on the standby node is verified, specifically including: Start the business process on the standby node and perform data consistency verification on the continuous running status of the process, the listening status of the service port, the initialization status of the main business loop, the database connection status, and the application layer health response. The business process outputs at least one of the following: the database access path of the actual database loaded, the database recovery point, the database file fingerprint, and the schema version. It then compares this information with the data ready record to verify the actual database loaded.

[0017] In one example, based on the dependency relationship between the business process and the corresponding virtual Internet Protocol (VPN) address, the virtual Internet Protocol (IP) address is configured and a protocol announcement is sent externally. Specifically, this includes: Determine the dependencies between the business process and the corresponding virtual Internet Protocol address; If the dependency is that the business process can bind to a physical address or all local addresses, then after the business process is ready, a virtual Internet Protocol address is created on the specified network card through the operating system network configuration interface, and after the virtual Internet Protocol address enters the available state, a free address resolution protocol announcement is sent to the outside. If the dependency relationship is as follows: the business process needs to bind a virtual Internet Protocol address, then the local configuration of the virtual Internet Protocol address is executed, and a virtual Internet Protocol address is created on the specified network card through the operating system network configuration interface, and external requests are blocked through firewall rules or business port access control; the business process is started, and the data consistency verification is performed, and after the verification is passed, a free address resolution protocol announcement is sent to the outside.

[0018] In one example, the method further includes: During the execution of the takeover action, if it is determined that the current stage does not meet the corresponding access conditions, the subsequent stages will not be executed. Instead, the completed actions recorded in the takeover transaction record will be read through the controller of the standby node, and the corresponding recovery actions will be executed in reverse order of the actual completion order of the completed actions.

[0019] In one example, the method further includes: If it is determined that the master node has recovered, then start the controller of the master node; The controller of the master node obtains the latest database copy recovery data generated by the standby node, recovers the data based on the database copy, verifies the recovery result, enters standby state after verification, and performs reverse switching action after receiving the planned switchback instruction sent by the administrator, based on the dependency relationship between the business process and the corresponding virtual Internet Protocol address.

[0020] On the other hand, this application also proposes a cross-layer collaborative dual-machine hot standby control system for industrial control systems. The system includes: a master node and a standby node. The master node and the standby node are respectively deployed with a controller, a business process, a business client, an embedded database, and a database replication component. An object storage service is deployed on a designated node, and the currently active node holds a fixed virtual Internet Protocol address.

[0021] The controller is used to execute the cross-layer collaborative dual-machine hot standby control method for industrial control systems as described in any of the above examples.

[0022] The cross-layer collaborative dual-machine hot standby control method for industrial control systems proposed in this application can bring the following beneficial effects: 1. By jointly verifying the recovery results of the embedded database and generating a data ready record associated with the takeover transaction record, databases that do not meet the preset verification conditions can be prevented from entering the subsequent business startup stage. By comparing the database information actually loaded by the business process with the data ready record, it is possible to identify situations where the business process has been running but the actual loaded database does not meet the requirements of this takeover, thereby reducing the risk of business recovery being affected by loading an incorrect database.

[0023] 2. By controlling the execution order of business process startup, address configuration, and external service access based on the dependency relationship between business processes and virtual Internet Protocol (VIP) addresses, external requests can be restricted from entering the takeover node before the business process is ready. Verifying address availability and sending gratuitous address resolution protocol (GRP) announcements can encourage network devices to update corresponding address mappings. When business link verification is enabled, network paths, business processing, and database access status can be further checked, reducing the risk of assuming business availability solely based on process execution or successful address configuration.

[0024] 3. By establishing persistent takeover transaction records with unique identifiers, recording the takeover phase, execution results, and completed actions, and stopping subsequent phases and performing corresponding reverse recovery when the current phase fails to meet the admission conditions, the risk of repeatedly handling the same fault event and inconsistencies in the database, business processes, and network resource states caused by incomplete takeover can be reduced. After the controller exits abnormally and restarts, execution can be resumed or recovery processing can continue based on the persistent records, improving the recoverability of the takeover process.

[0025] 4. By combining TCP application heartbeat messages, inter-node heartbeats, and related process status for fault determination, and prioritizing the handling of business anomalies that can be recovered within the primary node, unnecessary cross-node switching can be reduced. After the original primary node recovers, data recovery and verification are completed first, and it enters standby status. After the administrator initiates a planned back-switch and the final data synchronization is completed, a reverse switch is performed, which can reduce the risk of data version rollback and two nodes providing services simultaneously due to the original primary node prematurely resuming its active role. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 This is a flowchart illustrating the cross-layer collaborative dual-machine hot standby control method for industrial control systems in this application embodiment; Figure 2 This is a schematic diagram of the framework of a cross-layer collaborative dual-machine hot standby control system for industrial control systems, as described in one embodiment of this application. Figure 3 This is a functional module architecture diagram of the controller in one scenario of this application embodiment; Figure 4 This is a flowchart illustrating a cross-layer collaborative takeover process under one scenario in this application embodiment; Figure 5 This is a state transition diagram of a takeover transaction under one scenario in an embodiment of this application; Figure 6 This is a flowchart illustrating database recovery, joint verification, and data readiness record generation in one scenario of this application embodiment. Figure 7 This is a flowchart illustrating the consistency verification process of a business process loading the database under one scenario in this application embodiment. Figure 8 This is a flowchart illustrating VIP configuration, external access control, network publishing, and business link verification in one scenario of this application embodiment. Figure 9 This is a flowchart illustrating the reverse recovery process after a takeover failure, as described in one scenario of this application embodiment. Figure 10 This is a flowchart illustrating the original master node recovery and planned switchback process in one scenario of this application embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, this application provides a cross-layer collaborative dual-machine hot standby control method for industrial control systems, including: S101: Obtain TCP application heartbeat messages, inter-node heartbeats, and related process status between the primary node and the backup node through the controller of the backup node, and when it is determined that the primary node has a TCP application heartbeat abnormality, perform intra-node recovery for the primary node, and when the preset takeover trigger condition is met, perform takeover action through the backup node.

[0031] like Figure 2As shown, the system includes a first node (which acts as the primary node) HP, a second node (which acts as the backup node) OP, a High Availability Controller (HAController) deployed on both nodes, business processes, business clients, an embedded database (such as an SQLite database), a database replication component (such as Litestream), an object storage service, and a fixed Virtual Internet Protocol Address (VIP). Among these, as... Figure 3 As shown, the high availability controller (hereinafter referred to as the controller) includes a health status acquisition unit, a fault type determination unit, a takeover transaction control unit, a database recovery and verification unit, a business process control unit, a VIP configuration and release unit, a business link verification unit, and a failure recovery unit.

[0032] Specifically, such as Figure 4 As shown, based on the controller set in the standby node, it receives TCP application heartbeat messages sent by the service processes in the primary node through a TCP connection at a preset period. The TCP application heartbeat message includes at least one of the following: sending node identifier, service process instance identifier, monotonically increasing sequence number, and sending time. The service processes on the HP send application heartbeat messages (referred to as TCP application heartbeat messages) to the HA Controller on the OP through the existing TCP connection at a set period. The TCP application heartbeat message contains at least one or more of the following information: sending node identifier, service process instance identifier, monotonically increasing sequence number, and sending time.

[0033] Record the receiving time and monotonically increasing sequence number of the most recent valid TCP application heartbeat message, and collect the heartbeat data between master nodes and related process status. The OP's HA Controller records the receiving time and sequence number of the most recent valid message, and simultaneously collects the heartbeat data between HP nodes and related process status.

[0034] like Figure 5As shown, if the standby node's controller fails to receive a valid TCP application heartbeat message within several consecutive preset periods, the primary node is marked as a suspected fault. Within a preset fault confirmation window, it continues to monitor TCP application heartbeat messages, inter-node heartbeats, and related process statuses. The suspected fault status is revoked when valid TCP application heartbeat messages are received again and the number of consecutive confirmations reaches a preset number. When the OP's HA Controller fails to receive a valid TCP application heartbeat message within several consecutive monitoring periods, the HP is marked as a suspected transaction fault (APP_SUSPECT) instead of being directly considered down. The OP's HA Controller continues to monitor application heartbeats, inter-node heartbeats, and process statuses within the fault confirmation window; the suspected fault status is revoked when application heartbeats recover and the set number of consecutive confirmations is reached.

[0035] like Figure 4 As shown, if the master node is in a suspected fault state and the heartbeat between nodes is normal, it is determined to be a candidate event for business process anomaly. The master node's controller performs intra-node recovery on the master node and verifies the corresponding second recovery result. If the verification fails, the takeover decision process begins. When the TCP application heartbeat is abnormal (still in a suspected fault state) while the HP inter-node heartbeat is still normal, it is determined to be a candidate event for business process anomaly. The HP HA Controller prioritizes performing intra-node recovery on the business process and verifies the recovery result. If the intra-node recovery fails verification, the takeover decision process begins. When a business client exits independently, business client recovery is performed first. If business client recovery fails, it is escalated to application layer fault handling. If consecutive failures reach a preset threshold, the node takeover process may still be triggered.

[0036] If the master node is in a suspected fault state, and the heartbeats between nodes are abnormal, and the preset abnormal duration and consecutive failure count are reached, the takeover determination process will be initiated. When both the TCP application heartbeat and the heartbeats between HP nodes are continuously abnormal, and the preset abnormal duration and consecutive failure count are reached, the OP's HA Controller determines that the preset takeover trigger conditions are met and initiates the takeover determination process.

[0037] like Figure 5As shown, different branches are executed according to the system's configured operating strategy: Regarding the takeover decision process, in availability-first mode, the takeover action is automatically performed via a standby node. This includes performing VIP address conflict checks and attempting to confirm the primary node's activity status via storage probes or network probes. If no address conflict is detected or it is determined that the primary node is not providing services to external systems, the takeover action continues. Availability-first mode allows the operator to automatically execute the takeover process after meeting preset takeover trigger conditions, which is suitable for shortening service recovery time when the entire HP system is down. In availability-first mode, the operator automatically establishes a takeover transaction and executes the subsequent takeover process; before takeover, a VIP address conflict check is performed, and the HP's activity status is attempted to be confirmed via storage probes or network probes; if a conflict is detected or other conditions indicating that HP may still provide services are detected, VIP issuance is suspended and an alarm is triggered.

[0038] In Data Security Priority Mode, the standby node enters a takeover confirmation state and continues the takeover process after receiving a takeover confirmation command from the administrator. In Data Security Priority Mode, the OP enters a takeover confirmation state after meeting suspected fault conditions. Only after receiving a takeover confirmation command or confirmation that HP has ceased service does the subsequent takeover process proceed, thus reducing the risk of dual active nodes. In Data Security Priority Mode, the OP enters the takeover confirmation state, sends an alarm to the administrator, and waits for a clear takeover command. Only after receiving administrator confirmation or confirming out-of-band that HP has ceased service does the takeover transaction begin and the takeover is executed.

[0039] S102: For the takeover action, establish a persistent takeover transaction record with a unique identifier.

[0040] like Figure 4 As shown, for a takeover action, a takeover transaction record is established and persisted locally. The takeover transaction record includes: a universally unique identifier (UUID), the fault type, the initiation time, the current stage, the execution status of each stage (not started / in progress / successful / failed), the completed action stack, and key metadata required for recovery (such as the expected database recovery time). The HA Controller on the standby node (OP) establishes a takeover transaction record after determining that the preset takeover trigger conditions are met (or after receiving a confirmed takeover instruction).

[0041] Operation idempotency is achieved based on unique transaction identifiers. The OP enters the takeover preparation state (TAKEOVER_PREPARING) and ensures operation idempotency based on transaction identifiers to prevent the same failure event from being processed repeatedly.

[0042] If the standby node's controller exits abnormally during the takeover process, the standby node's controller will be restarted, and subsequent processing will be performed based on the persisted takeover transaction records. If the HA Controller process exits abnormally during the takeover process, after restarting, it will determine the subsequent actions based on the persisted transaction records: for incomplete transactions, it will automatically resume execution or perform reverse recovery according to the configured policy; for transactions marked as failed, it will enter an alarm state, waiting for administrator intervention.

[0043] S103: Based on the database replica corresponding to the master node, perform embedded database recovery on the standby node, perform joint verification on the corresponding first recovery result, and generate a data ready record associated with the takeover transaction record after the verification is passed.

[0044] like Figure 6 As shown, stop the business processes, business clients, and replication processes that are using the embedded database on the standby node, and move the corresponding original database files and related files to the isolation directory. Stop the business processes, business clients, and replication processes that may be using SQLite files on the standby node, and move the original database files and their related files (such as -wal, -shm, -journal) to the isolation directory.

[0045] The new database file is restored to a temporary path based on the database copy corresponding to the master node stored in object storage. The new database file (the new SQLite database) is restored to a temporary path from the Litestream copy stored in object storage.

[0046] For new database files, a joint verification is performed using the recovery process exit status, recovery summary, existence and non-emptiness of temporary database files, database read-only open results, integrity check results, key tables, schema version, and recovery point.

[0047] If the first recovery passes, the database access path is changed to point to the new database file via configuration file update or symbolic link modification. Once all verification items meet the preset conditions, the temporary database is replaced with the official database in a controlled manner. For example, a "dual-file switching strategy" can be used: after the joint verification passes, an atomic configuration file update or symbolic link change is performed to point the database access path of the business process to the new file, thereby reducing the impact on the service; or a "write-rename-delete" method can be used to complete the switch.

[0048] Generate a data-ready record associated with the takeover transaction record. This record includes: database recovery point, database file fingerprint, schema version, integrity check results, and database access path. Once database verification is successful, a data-ready record associated with the takeover transaction is generated. A valid data-ready record is a prerequisite for business process startup and VIP external deployment.

[0049] S104: Based on the data ready record, verify the actual database loaded by the business process on the standby node, and configure the virtual Internet Protocol address and send a protocol announcement to the outside according to the dependency relationship between the business process and the corresponding virtual Internet Protocol address.

[0050] like Figure 7 As shown, the business process on the standby node is started, and data consistency verification is performed on the continuous running status of the process, the service port listening status, the business main loop initialization status, the database connection status, and the application layer health response.

[0051] The business process outputs at least one of the following: the actual database access path to the loaded database, the database recovery point, the database file fingerprint, and the schema version. This is compared with the data readiness record to verify the actual database being loaded. Once the comparison matches and the application-layer health check reaches the set number of consecutive successful checks, the server is confirmed to be ready (SERVER_READY).

[0052] like Figure 8 As shown, this solution supports two dependency modes between business processes and VIPs. Choose the appropriate mode based on your actual deployment conditions: Determine the dependencies between business processes and their corresponding virtual Internet Protocol (IP) addresses.

[0053] Mode 1: The business process can bind to a physical address or all local addresses. If the dependency is that the business process can bind to a physical address or all local addresses, then after the business process is ready, a virtual Internet Protocol address is created on the specified network card through the operating system's network configuration interface. After the virtual Internet Protocol address enters an available state (e.g., the IP address duplicate address detection state is Preferred), a gratuitous Address Resolution Protocol (ARP) announcement is sent to the outside.

[0054] Mode 2: The business process must be bound to a VIP to start. If the dependency is: the business process needs to bind a virtual Internet Protocol (VPN) address, then the local configuration of the VPN address is executed, and a VPN address is created on the specified network card through the operating system's network configuration interface. External requests are blocked through firewall rules or business port access control. The business process is started, and data consistency verification is performed (consistent with the data consistency verification in S104). After the verification is passed, the external access restrictions previously set through firewall rules or business port access control are lifted, and a free address resolution protocol announcement is sent to the outside world, completing the opening of the VIP service to the outside world.

[0055] Therefore, the VIP's local configuration and external service access are set up as two phases subject to different constraints.

[0056] S105: Start the service client in the backup node, and when the service client reaches the preset running state, determine that the takeover action has been completed.

[0057] Optionally, a TKT application protocol request can be sent via the VIP through a non-loopback network path or a dedicated probe node to verify the VIP network path, Server service processing status, and database access status. If the verification passes, the VIP service path is confirmed to be fully ready (VIP_READY). If the verification cannot be completed due to network policy restrictions or other reasons, an alarm should be generated based on the local listening status and ARP announcement results, instructing the administrator to perform manual verification. The system should not block subsequent client startups to avoid service unavailability in environments with strict network policies.

[0058] The business client starts when the database is ready (DB_READY), the server is ready (SERVER_READY), and the VIP business path is ready (VIP_READY, if this verification is enabled). After the client reaches the preset running state, it submits the takeover transaction, sets the takeover node to the active state, and publishes the service ready state.

[0059] In one embodiment, such as Figure 9As shown, during the takeover process, if the current stage does not meet the corresponding admission conditions, subsequent stages are not executed. Instead, the controller on the standby node reads the completed actions recorded in the takeover transaction record and executes the corresponding recovery actions in reverse order of their actual completion. If any stage fails to meet the admission conditions, subsequent stages are not executed. The HA Controller reads the completed actions recorded in the takeover transaction and selectively executes operations such as stopping the Client, stopping the Server, restricting external access via the VIP, removing the VIP, stopping the new replication source role, and isolating and restoring the database, in the reverse order of the takeover sequence. The recovery results are written to the takeover transaction; after completion, the transaction is marked as a takeover failure (TAKEOVER_FAILED) and an alarm is output. This process allows the HA Controller to continue resource reclamation based on the transaction record even after an abnormal exit and restart.

[0060] In one embodiment, such as Figure 10 As shown, if it is determined that the master node has recovered, the master node controller is started; after HP recovery, only the HA Controller is started, and the VIP is not automatically configured, nor are the business processes or business clients started automatically.

[0061] The primary node's controller retrieves the latest database replica recovery data from the standby node, restores the data based on this replica, and verifies the recovery results. Upon successful verification, the system enters standby mode. Upon receiving a planned switchback command from the administrator, the system executes a reverse switchover based on the dependencies between business processes and their corresponding virtual Internet Protocol (VLAN) addresses. HP restores data from the latest database replica generated by the currently active node and performs the same database verification as during the takeover phase; upon successful verification, the system enters standby mode. After the administrator initiates a planned switchback and final data synchronization is completed, the system executes a reverse switchover according to the aforementioned stage dependencies.

[0062] 1. By jointly verifying the recovery results of the embedded database and generating a data ready record associated with the takeover transaction record, databases that do not meet the preset verification conditions can be prevented from entering the subsequent business startup stage. By comparing the database information actually loaded by the business process with the data ready record, it is possible to identify situations where the business process has been running but the actual loaded database does not meet the requirements of this takeover, thereby reducing the risk of business recovery being affected by loading an incorrect database.

[0063] 2. By controlling the execution order of business process startup, address configuration, and external service access based on the dependency relationship between business processes and virtual Internet Protocol (VIP) addresses, external requests can be restricted from entering the takeover node before the business process is ready. Verifying address availability and sending gratuitous address resolution protocol (GRP) announcements can encourage network devices to update corresponding address mappings. When business link verification is enabled, network paths, business processing, and database access status can be further checked, reducing the risk of assuming business availability solely based on process execution or successful address configuration.

[0064] 3. By establishing persistent takeover transaction records with unique identifiers, recording the takeover phase, execution results, and completed actions, and stopping subsequent phases and performing corresponding reverse recovery when the current phase fails to meet the admission conditions, the risk of repeatedly handling the same fault event and inconsistencies in the database, business processes, and network resource states caused by incomplete takeover can be reduced. After the controller exits abnormally and restarts, execution can be resumed or recovery processing can continue based on the persistent records, improving the recoverability of the takeover process.

[0065] 4. By combining TCP application heartbeat messages, inter-node heartbeats, and related process status for fault determination, and prioritizing the handling of business anomalies that can be recovered within the primary node, unnecessary cross-node switching can be reduced. After the original primary node recovers, data recovery and verification are completed first, and it enters standby status. After the administrator initiates a planned back-switch and the final data synchronization is completed, a reverse switch is performed, which can reduce the risk of data version rollback and two nodes providing services simultaneously due to the original primary node prematurely resuming its active role.

[0066] like Figure 2 and Figure 3 As shown in the embodiment of this application, a cross-layer collaborative dual-machine hot standby control system for industrial control systems is also proposed. The system includes: a master node and a standby node. The master node and the standby node are respectively deployed with a controller, a business process, a business client, an embedded database, and a database replica replication component. An object storage service is deployed on a designated node, and the currently active node holds a fixed virtual Internet Protocol address.

[0067] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A cross-layer collaborative dual-machine hot standby control method for industrial control systems, characterized in that, The method includes: The controller of the standby node obtains TCP application heartbeat messages, inter-node heartbeats and related process status between the master node and the standby node, and performs intra-node recovery for the master node when it is determined that the master node has a TCP application heartbeat abnormality, and performs takeover action through the standby node when the preset takeover trigger conditions are met. For the aforementioned takeover action, a persistent takeover transaction record with a unique identifier is established; Based on the database replica corresponding to the master node, the embedded database is restored on the standby node, and the corresponding first restoration result is jointly verified. After the verification is passed, a data ready record associated with the takeover transaction record is generated. Based on the data ready record, the actual database loaded by the business process on the standby node is verified, and the virtual Internet Protocol address is configured and a protocol announcement is sent to the outside according to the dependency relationship between the business process and the corresponding virtual Internet Protocol address. Start the service client in the backup node, and once the service client reaches the preset running state, confirm that the takeover action is complete.

2. The method according to claim 1, characterized in that, The standby node's controller obtains TCP application heartbeat messages between the primary and standby nodes, inter-node heartbeats, and related process statuses, specifically including: Based on the controller set in the backup node, the system receives TCP application heartbeat messages sent by the business processes in the master node through a TCP connection at a preset period. The TCP application heartbeat message includes at least one of the following: sending node identifier, business process instance identifier, monotonically increasing sequence number, and sending time. Record the receiving time and monotonically increasing sequence number of the most recent valid TCP application heartbeat message, and collect the heartbeat and related process status between the master node.

3. The method according to claim 2, characterized in that, When a TCP application heartbeat anomaly is detected on the primary node, intra-node recovery is performed on the primary node, and when a preset takeover trigger condition is met, a takeover action is performed through the backup node, specifically including: If the controller of the backup node fails to receive a valid TCP application heartbeat message within several consecutive preset periods, the master node is marked as a suspected fault state. In a preset fault confirmation window, the controller continues to detect the TCP application heartbeat message, the heartbeat between nodes, and the status of related processes. When the controller resumes receiving valid TCP application heartbeat messages and the number of consecutive confirmations reaches a preset number, the suspected fault state is revoked. If the master node is in the suspected fault state and the heartbeat between the nodes is normal, it is determined to be a candidate event for business process abnormality. The controller of the master node performs intra-node recovery for the master node and verifies the corresponding second recovery result. If the verification fails, the takeover determination process is initiated. If the master node is in the suspected fault state, and the heartbeat between the nodes is abnormal, and the preset abnormal duration and number of consecutive failures are reached, then the takeover determination process is initiated. Regarding the takeover determination process, in availability-first mode, the takeover action is automatically executed through the backup node, performing a virtual Internet Protocol address conflict check and attempting to confirm the activity status of the master node through storage probes or network probes. If no address conflict is detected or it is determined that the master node is not providing business services to the outside, the takeover action continues. In data security-first mode, the backup node enters a takeover confirmation state and continues to execute the takeover action after receiving a takeover confirmation instruction sent by the administrator.

4. The method according to claim 1, characterized in that, For the aforementioned takeover action, a persistent takeover transaction record with a unique identifier is established, specifically including: For the takeover action, a takeover transaction record is established and persisted locally. The takeover transaction record includes: a unique transaction identifier, fault type, initiation time, current stage, execution result status of each stage, completed action stack, and key metadata required for recovery. Idempotency of operations is achieved based on the unique identifier of the transaction. If the controller of the backup node exits abnormally during the execution of the takeover action, the controller of the backup node is restarted, and subsequent processing is performed based on the persistent takeover transaction records.

5. The method according to claim 1, characterized in that, Based on the database replica corresponding to the primary node, the embedded database is restored on the standby node, and the corresponding first restoration result is jointly verified. After the verification passes, a data ready record associated with the takeover transaction record is generated, specifically including: Stop the business processes, business clients, and replication processes that occupy the embedded database on the backup node, and move the corresponding original database files and associated files to the isolation directory; The new database file is restored to a temporary path based on the database copy corresponding to the master node stored in the object storage; For the new database file, a joint verification is performed using the recovery process exit status, recovery summary, existence and non-emptiness of temporary database files, database read-only open result, integrity check result, key tables, schema version, and recovery point. If the first recovery result is successful, the database access path is made to point to the new database file by updating the configuration file or changing the symbolic link; Generate a data ready record associated with the takeover transaction record; the data ready record includes: database recovery point, database file fingerprint, schema version, integrity check result, and database access path.

6. The method according to claim 1, characterized in that, Based on the data readiness record, the actual database loaded by the business processes on the standby node is verified, specifically including: Start the business process on the standby node and perform data consistency verification on the continuous running status of the process, the listening status of the service port, the initialization status of the main business loop, the database connection status, and the application layer health response. The business process outputs at least one of the following: the database access path of the actual database loaded, the database recovery point, the database file fingerprint, and the schema version. It then compares this information with the data ready record to verify the actual database loaded.

7. The method according to claim 6, characterized in that, Based on the dependency relationship between the business process and the corresponding virtual Internet Protocol (VPN) address, the VPN address is configured and protocol announcements are sent externally, specifically including: Determine the dependencies between the business process and the corresponding virtual Internet Protocol address; If the dependency is that the business process can bind to a physical address or all local addresses, then after the business process is ready, a virtual Internet Protocol address is created on the specified network card through the operating system network configuration interface, and after the virtual Internet Protocol address enters the available state, a free address resolution protocol announcement is sent to the outside. If the dependency relationship is as follows: the business process needs to bind a virtual Internet Protocol address, then the local configuration of the virtual Internet Protocol address is executed, and a virtual Internet Protocol address is created on the specified network card through the operating system network configuration interface, and external requests are blocked through firewall rules or business port access control; the business process is started, and the data consistency verification is performed, and after the verification is passed, a free address resolution protocol announcement is sent to the outside.

8. The method according to claim 1, characterized in that, The method further includes: During the execution of the takeover action, if it is determined that the current stage does not meet the corresponding access conditions, the subsequent stages will not be executed. Instead, the completed actions recorded in the takeover transaction record will be read through the controller of the standby node, and the corresponding recovery actions will be executed in reverse order of the actual completion order of the completed actions.

9. The method according to claim 1, characterized in that, The method further includes: If it is determined that the master node has recovered, then start the controller of the master node; The controller of the master node obtains the latest database copy recovery data generated by the standby node, recovers the data based on the database copy, verifies the recovery result, enters standby state after verification, and performs reverse switching action after receiving the planned switchback instruction sent by the administrator, based on the dependency relationship between the business process and the corresponding virtual Internet Protocol address.

10. A cross-layer collaborative dual-machine hot standby control system for industrial control systems, characterized in that, The system includes: a master node and a backup node, on which a controller, a business process, a business client, an embedded database, and a database replication component are respectively deployed; an object storage service is deployed on a designated node, and a fixed virtual Internet Protocol address is held on the currently active node; The controller is used to execute the cross-layer collaborative dual-machine hot standby control method for industrial control systems as described in any one of claims 1 to 9.