Guard system and method of multi-process control system

Abstract

The invention discloses a guarding system and method for a multi-process control system, and relates to the technical field of guarding of the multi-process control system.The system comprises a shared memory and a process management module, and one service process in the multi-process control system corresponds to one message area in the shared memory; the business process stores the operation state of the business process in the message area, and for each business process, the process management module obtains the operation state of the business process from the message area, performs health diagnosis on the business process based on the operation state of the business process to obtain a health score, and sends the health score to the business process; whether the service process needs to be restarted or not is decided on the basis of the health score, the process state, the resource load, the network state and the forced restarting score of the service process, a decision result is obtained, and when the decision result is restarting, the service process is restarted. According to the method, the fault of the service process can be found in time, the service process can be restarted in time, and stable, safe and reliable operation of the multi-process control system is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of multi-process control system guarding, and in particular to a guarding system and method for a multi-process control system. Background Art

[0002] At present, multi-process control systems have been widely used. Multi-process control systems are a software architecture model that implements specific functions or services by creating and managing multiple business processes. They are widely used in operating systems, server software, distributed systems and other fields. The typical feature of multi-process control systems is that multiple business processes run simultaneously, and multiple business processes need to run stably for a long time. Once a business process becomes abnormal, it will affect the security of the entire multi-process control system.

[0003] Traditional process daemonization methods, such as watchdog, supervisord, or systemd, provide a certain degree of process daemonization functionality, but they often fail to detect and restart business process failures in a timely manner, thus failing to meet the stringent requirements for high availability, security, and self-recovery. Therefore, an efficient, real-time process daemonization method is needed that can quickly detect and recover from abnormal situations, ensuring the stable, secure, and reliable operation of multi-process control systems. Summary of the Invention

[0004] The purpose of this application is to provide a guard system and method for a multi-process control system, which can promptly detect business process failures and restart business processes in a timely manner, ensuring the stable, safe and reliable operation of the multi-process control system.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In the first aspect, the present application provides a guardian system for a multi-process control system, wherein the guardian system of the multi-process control system includes: a shared memory and a process management module, wherein a business process in the multi-process control system corresponds to a message area in the shared memory, and the business process is used to store the running status of the business process in the message area, and the process management module includes a process diagnosis unit, a process decision unit and a process restart unit; the running status includes a heartbeat timestamp, process status, resource load, network status and error alarm status.

[0007] The process diagnosis unit is used to obtain the running status of each business process from the message area, perform health diagnosis on the business process based on the running status of the business process, obtain a health score, and store the health score in the message area.

[0008] The process decision unit is used to obtain the health score, process status, resource load and network status of each business process from the message area, decide whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, obtain a decision result, and store the decision result in the message area; the forced restart score is determined based on whether a forced restart command for the business process is input externally.

[0009] The process restarting unit is configured to obtain a decision result of each of the service processes from the message area, and restart the service process when the decision result is restart.

[0010] Optionally, the service process is further configured to send a heartbeat signal to the message area at intervals of a first preset time, and the message area records a timestamp of the heartbeat signal to obtain a heartbeat timestamp.

[0011] When performing a health diagnosis on the business process based on the running status of the business process to obtain a health score, the process diagnosis unit is configured to, when the heartbeat timestamp has not been updated for a second preset time, reduce the health score obtained from the previous health diagnosis to obtain a first score; when the process status is zombie, reduce the health score obtained from the previous health diagnosis to obtain a second score; when the process status is running, increase the health score obtained from the previous health diagnosis to obtain a second score; when the resource load exceeds a first preset threshold, reduce the health score obtained from the previous health diagnosis to obtain a third score; when the network status is network interruption, transmission failure, or round-trip delay exceeds a second preset threshold, reduce the health score obtained from the previous health diagnosis to obtain a fourth score; when the error alarm status is multiple consecutive errors or alarms, reduce the health score obtained from the previous health diagnosis to obtain a fifth score; when the error alarm status is no errors or alarms, increase the health score obtained from the previous health diagnosis to obtain the fifth score; and perform a weighted average of the first score, the second score, the third score, the fourth score, and the fifth score to perform a health diagnosis on the business process based on the running status of the business process to obtain a health score.

[0012] Optionally, the resource load includes CPU occupancy and memory occupancy. In this case, the resource load exceeding the first preset threshold specifically includes: the CPU occupancy exceeds the CPU occupancy threshold or the memory occupancy exceeds the memory occupancy threshold.

[0013] The upper limit of the first score, the second score, the third score, the fourth score, and the fifth score is 100, and the lower limit is 0.

[0014] Optionally, when a decision is made on whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, and a decision result is obtained, the process decision unit is used to determine the resource load score based on the resource load, and determine the network status score based on the network status; when the process status is running, the health score, the resource load score, the network status score and the forced restart score are weightedly summed to obtain a comprehensive score; when the process status is dead, the comprehensive score is set equal to 0 to obtain a comprehensive score; when the comprehensive score is less than a third preset threshold, it is determined that the business process needs to be restarted, and the decision result is restart; when the comprehensive score is greater than or equal to the third preset threshold, it is determined that the business process does not need to be restarted, and the decision result is not restart.

[0015] Optionally, the daemon system of the multi-process control system also includes: a daemon process module, the daemon process module includes a daemon process, the process diagnosis unit includes a diagnosis process, the process decision unit includes a decision process, the process restart unit includes a restart process, the daemon process, the diagnosis process, the decision process and the restart process respectively correspond to a message area in the shared memory, the daemon process is used to store the running status of the daemon process in the message area, the diagnosis process is used to store the running status of the diagnosis process in the message area, the decision process is used to store the running status of the decision process in the message area, and the restart process is used to store the running status of the restart process in the message area.

[0016] The daemon process module is used to perform health diagnosis, decision-making and restart on the diagnosis process, the decision-making process, the restart process and each of the business processes.

[0017] The process diagnosis unit is further used to perform health diagnosis on the daemon process, the process decision unit is further used to make a decision on the daemon process, and the process restart unit is further used to restart the daemon process.

[0018] Optionally, the process decision unit is also used to store the comprehensive score in the message area, and the process management module also includes a process warning unit, which is used to obtain the comprehensive score of the business process from the message area for each business process. If the comprehensive score is greater than or equal to a fourth preset threshold, the business process is determined to be normal; if the comprehensive score is less than the fourth preset threshold and greater than or equal to the third preset threshold, the business process is determined to be abnormal; when it is determined that the business process is abnormal for multiple consecutive times, an alarm message is generated and the alarm message is stored in the message area.

[0019] Optionally, the process management module also includes a synchronization unit, which is used to write the running status of all business processes in the shared memory into a local log file every third preset time, and upload the running status and health score of all business processes in the shared memory to the central server every fourth preset time.

[0020] Optionally, the process management module also includes a memory allocation unit. When the business process is started, the business process is used to send registration information to the memory allocation unit. When the memory allocation unit receives the registration information, it allocates an idle message area to the business process; the registration information includes the PID of the business process.

[0021] The memory allocation unit stores a memory mapping table, which is used to mark the occupancy status of each message area. The occupancy status includes being used, idle, and to be cleared.

[0022] Optionally, when restarting the business process, the process restart unit is used to terminate the business process and restart the business process; the startup mode used during the restart includes a hot start mode and a cold start mode.

[0023] In a second aspect, the present application provides a guarding method for a multi-process control system, which works based on the guarding system of the multi-process control system described above. The guarding method for the multi-process control system includes the following steps.

[0024] For each business process, the process diagnosis unit obtains the running status of the business process from the message area, performs health diagnosis on the business process based on the running status of the business process, obtains a health score, and stores the health score in the message area; the running status includes heartbeat timestamp, process status, resource load, network status and error alarm status.

[0025] For each business process, the process decision unit obtains the health score, process status, resource load and network status of the business process from the message area, makes a decision on whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, obtains a decision result, and stores the decision result in the message area; the forced restart score is determined based on whether a forced restart command for the business process is input externally.

[0026] The process restarting unit obtains a decision result of each service process from the message area, and restarts the service process when the decision result is restart.

[0027] According to the specific embodiments provided in this application, this application has the following technical effects.

[0028] The present application provides a guard system and method for a multi-process control system, which is provided with a shared memory. A business process in the multi-process control system corresponds to a message area in the shared memory. The business process stores the running status of the business process in the message area. For each business process, a process diagnosis unit obtains the running status of the business process from the message area, performs health diagnosis on the business process based on the running status of the business process, obtains a health score, and stores the health score in the message area. A process decision unit obtains the health score, process status, resource load and network status of the business process from the message area, makes a decision on whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, obtains a decision result, and stores the decision result in the message area. A process restart unit obtains the decision result of the business process from the message area, and restarts the business process when the decision result is to restart. This application introduces shared memory, and each business process can store its own running status in the shared memory. At this time, the process diagnosis unit, process decision unit and process restart unit can directly access the shared memory to complete the health diagnosis, decision and restart of each business process. There is no need to traverse the running status of the business process. Business process failures can be discovered in time and the business process can be restarted in time, ensuring the stable, safe and reliable operation of the multi-process control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1This is a system architecture diagram of a guard system for a multi-process control system provided in Example 1 of the present application.

[0031] Figure 2 This is a business flow chart of a guard system for a multi-process control system provided in Example 1 of the present application.

[0032] Figure 3 This is a flowchart of the process daemon operation provided in Example 1 of the present application.

[0033] Figure 4 Detailed operation flow chart of the process daemon provided in Example 1 of this application.

[0034] Figure 5 This is the operation flow chart of the process restart provided in Example 1 of the present application.

[0035] Figure 6 A flowchart of the guarding method for a multi-process control system provided in Example 2 of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] Example 1.

[0038] This embodiment provides a guardian system for a multi-process control system, wherein the guardian system of the multi-process control system includes: a shared memory and a process management module. A business process in the multi-process control system corresponds to a message area in the shared memory. The business process is used to store the running status of the business process in the message area. The running status includes a heartbeat timestamp, process status, resource load, network status, and error alarm status. The process management module includes a process diagnosis unit, a process decision unit, and a process restart unit.

[0039] The process diagnosis unit is used to obtain the running status of each business process from the message area, perform health diagnosis on the business process based on the running status of the business process, obtain a health score, and store the health score in the message area.

[0040] The process decision unit is used to obtain the health score, process status, resource load, and network status of each business process from the message area. Based on the business process's health score, process status, resource load, network status, and forced restart score, it determines whether the business process needs to be restarted, obtains a decision result, and stores the decision result in the message area. The forced restart score is determined based on whether an external forced restart command for the business process has been input. In a score range of 0-100, if an external forced restart command for the business process has been input, the forced restart score is 100; otherwise, the forced restart score is 0.

[0041] The process restart unit is used to obtain the decision result of each business process from the message area, and restart the business process when the decision result is restart.

[0042] This embodiment can be applied to a multi-process control system using a domestic Linux operating system, and a guard system of a multi-process control system based on a domestic Linux operating system is provided. Of course, the multi-process control system can also use other operating systems.

[0043] A Distributed Control System (DCS) is used for automated control in nuclear power plants, undertaking critical tasks such as reactor monitoring, alarms, and interlocking. It requires 24 / 7 continuous operation and fault tolerance. Since DCS typically employs a multi-process architecture, where multiple business processes must operate stably and long-term, an abnormal exit of a business process will impact the security of the entire system. Traditional process daemonization methods cannot meet the stringent requirements of nuclear industry DCS for high availability, security, and self-recovery. The multi-process control system of this embodiment can be a distributed control system in the nuclear industry. The daemonization system of this multi-process control system is suitable for use in the nuclear industry DCS field. This daemonization system can ensure the high availability of the nuclear industry DCS by ensuring the timely and automatic recovery of the business processes in the nuclear industry DCS under abnormal circumstances, thereby improving the reliability and security of the nuclear industry DCS.

[0044] The multi-process control system provided in this embodiment is a guard system, such as Figures 1-4 As shown, Figure 1 In the process, process daemon includes process decision-making and process monitoring, and process monitoring includes process diagnosis and process restart. It adopts master-child process collaboration, shared memory-based process diagnosis, process decision-making, process restart and other technologies to ensure that the nuclear industry DCS can automatically recover and operate stably for a long time under abnormal circumstances.

[0045] (1) Process diagnosis based on shared memory.

[0046] This embodiment maintains the running status of all business processes through shared memory (SHM), and then uses a health scoring algorithm to evaluate the running status of the business processes to obtain the health scores of the business processes and store them in the shared memory.

[0047] The management process (i.e., the process management module) communicates with the managed business processes through shared memory. The shared memory is divided into multiple message areas and maintained by the management process. When business processes 1 to N (N is a positive integer) start, they send registration information to the management process. The registration information includes the business process's PID (Process Identifier) ​​and running status, which is equivalent to registering with the management process. The management process allocates a message area for it. The diagnostic process (i.e., the process diagnostic unit) included in the management process periodically queries the running status of the business process in the shared memory based on the business process's PID, determines the health score of the business process based on the running status of the business process, and writes the health score to the message area corresponding to the business process in the shared memory. The decision process (i.e., the process decision unit) included in the management process periodically queries the running status and health score of the business process in the shared memory to further decide whether the business process needs to be restarted.

[0048] The health diagnosis process specifically includes the following steps.

[0049] (1) Initialize shared memory.

[0050] Create shared memory and initialize its data structure. Specifically, divide the shared memory into multiple message areas, each storing the running status of a corresponding business process. The diagnostic process begins a loop monitoring the running status changes within the shared memory. Specifically, a while loop is used to read the contents of the shared memory at regular intervals to diagnose the running status of the business process. Furthermore, the management process maintains a memory mapping table to mark which message areas are in use, free, or awaiting cleanup.

[0051] (2) Process registration.

[0052] After the business process is started, it sends registration information to the management process. The management process allocates a message area of ​​shared memory for the business process and sets the initial health score of the business process. When the score range is 0-100, the initial health score is generally 100.

[0053] (3) Operation status collection and health score setting.

[0054] The diagnostic process periodically queries the running status of all business processes, including heartbeat timestamps, process status, resource load, network status, and error alarm status, and adjusts the health score of the business process as follows:

[0055] 1) Heartbeat Mechanism: Each service process periodically sends a heartbeat signal to the message area in shared memory. The message area records the timestamp of the heartbeat signal and updates the heartbeat timestamp in the message area. The diagnosis process checks the heartbeat timestamp every cycle to see if it has been updated. If the heartbeat timestamp has not been updated for X consecutive seconds, the heartbeat is considered disconnected and the health score from the previous health diagnosis is lowered. A health diagnosis is then performed based on the heartbeat timestamp to obtain the first score. X is a positive integer, the size of which depends on user requirements.

[0056] 2) Process status: Use / proc / pid / status to determine whether the process exists. If the process status is zombie, that is, the process is blocked for a long time or crashed, the health score obtained from the previous health diagnosis is reduced. If the process status is running, that is, the process is alive and running normally, the health score obtained from the previous health diagnosis is increased. A health diagnosis based on the process status is performed to obtain a second score.

[0057] 3) System resource usage: Use / proc / pid / stat to obtain CPU usage and memory usage. If the CPU usage exceeds the CPU usage threshold or the memory usage exceeds the memory usage threshold, it means that the resource load exceeds the first preset threshold. In this case, the health score obtained from the previous health diagnosis is lowered to perform a health diagnosis based on the resource load to obtain a third score.

[0058] 4) Communication status: For business processes with communication interfaces, connectivity is confirmed through socket interaction to determine the network status. If the network status is network interruption, transmission failure (if the network is not interrupted, the transmission failure may be caused by reasons such as the interface being occupied), or the RTT (Round-Trip Time) exceeds the second preset threshold, the health score obtained from the previous health diagnosis is reduced, and a health diagnosis based on the network status is performed to obtain a fourth score.

[0059] 5) Error / alarm event statistics: The output logs, error codes, and number of exceptions of the business process are collected. Receiving an error code represents an error, and confirming an exception represents an alarm. If there are multiple consecutive errors or alarms, the health score obtained from the last health diagnosis will be reduced. If it is operating normally, that is, there are no errors or alarms, the health score can be gradually restored, that is, the health score obtained from the last health diagnosis will be increased. The health diagnosis is performed based on the error and alarm status to obtain the fifth score.

[0060] It should be noted that when performing a health diagnosis for the first time, the health score obtained from the previous health diagnosis is the initial health score. When lowering or increasing the health score obtained from the previous health diagnosis, the extent of the reduction and increase can be set by the user depending on the situation.

[0061] Based on the collected operational status of the business process, the first, second, third, fourth, and fifth scores of the business process are determined as described above. Weights are then assigned to the first, second, third, fourth, and fifth scores, and a weighted average is used to calculate the health score of the business process. The sum of the weights of the first, second, third, fourth, and fifth scores is 1.

[0062] This embodiment may adopt a dynamic adjustment strategy as shown below.

[0063] 1) Dynamic recovery: When the abnormality is resolved, the health score does not recover immediately, but gradually recovers. This is achieved by increasing the health score obtained from the previous health diagnosis.

[0064] 2) Damping mechanism: When multiple anomalies occur consecutively, the health score decreases faster to prevent fluctuations. This is achieved by reducing the health score obtained from the previous health diagnosis by a larger number of consecutive errors or alarms when performing health diagnosis based on the error alarm state, and the resulting fifth score is smaller.

[0065] 3) Upper limit: The highest health score shall not exceed 100 and the lowest shall not be lower than 0.

[0066] 4) Delayed Update: To prevent frequent restarts caused by momentary misjudgments, the health score is lowered only after Y consecutive abnormality detections. Specifically, when performing health diagnosis based on heartbeat timestamps, process status, resource load, network status, and error alarm status, the health score obtained from the previous health diagnosis must be lowered Y times in a row. Y is a positive integer, and its size depends on user needs.

[0067] (4) Shared memory data synchronization.

[0068] To ensure the traceability of key operating data and the ability to reconstruct problems, this embodiment designs a multi-level data synchronization mechanism, as shown below.

[0069] 1) Local persistence mechanism: Every 5 seconds (which can be replaced by other times based on user needs), a snapshot of the running status of all business processes in shared memory is written to a local log file. The log file format is JSON / CSV, which is easy for analysis tools to read. Each log file can be rolled over on an hourly / daily basis.

[0070] 2) Remote synchronization mechanism: The management process regularly uploads the operating status and health scores obtained from health diagnosis to the central server through the intranet interface. It uses encrypted transmission and supports packet loss retransmission mechanism. A synchronization failure retry policy can be set, that is, re-upload after an upload failure.

[0071] 3) Abnormal event push: If a business process is diagnosed as abnormal three or more times in a row (this number can be replaced by other times, depending on user needs), the management process will generate an alarm message, write the alarm information to the shared memory, and then send it to the HMI (Human-Machine Interface) or maintenance terminal through the protocol.

[0072] 4) Cache full warning mechanism: The management process issues a warning when the remaining available capacity of the shared memory falls below the lower limit.

[0073] In this embodiment, the process management module further includes a memory allocation unit. When a service process starts, the service process sends registration information to the memory allocation unit. Upon receiving the registration information, the memory allocation unit allocates an idle message area to the service process. The registration information includes the service process's PID. The memory allocation unit stores a memory mapping table that marks the occupancy status of each message area, including used, idle, and pending cleanup.

[0074] The business process sends a heartbeat signal to the message area at intervals of a first preset time, and the message area records the timestamp of the heartbeat signal to obtain a heartbeat timestamp. When a health diagnosis is performed on the business process based on the running status of the business process to obtain a health score, the process diagnosis unit is used to reduce the health score obtained from the last health diagnosis when the heartbeat timestamp has not been updated for a second preset time to obtain a first score; when the process state is dead, reduce the health score obtained from the last health diagnosis to obtain a second score; when the process state is running, increase the health score obtained from the last health diagnosis to obtain a second score; when the resource load exceeds the first preset threshold, reduce the health score obtained from the last health diagnosis to obtain a second score; The health score obtained by the health diagnosis is used to obtain a third score; when the network status is network interruption, transmission failure or round-trip delay exceeds the second preset threshold, the health score obtained by the previous health diagnosis is reduced to obtain a fourth score; when the error alarm status is multiple consecutive errors or alarms, the health score obtained by the previous health diagnosis is reduced to obtain a fifth score, and when the error alarm status is no error or alarm, the health score obtained by the previous health diagnosis is increased to obtain a fifth score; the first score, the second score, the third score, the fourth score and the fifth score are weighted averaged to perform a health diagnosis on the business process based on the running status of the business process to obtain a health score.

[0075] The resource load includes CPU usage and memory usage. In this case, the resource load exceeding the first preset threshold specifically includes: the CPU usage exceeding the CPU usage threshold or the memory usage exceeding the memory usage threshold. The upper limit of the first score, the second score, the third score, the fourth score, and the fifth score is 100, and the lower limit is 0.

[0076] The process management module also includes a synchronization unit, which is used to write the running status of all business processes in the shared memory into a local log file every third preset time, and upload the running status and health score of all business processes in the shared memory to the central server every fourth preset time.

[0077] (2) Process decision-making based on intelligent decision-making mechanism.

[0078] The diagnosis process periodically evaluates the running status of the business process every 50ms (other times can be used, depending on user needs) to obtain a health score. The decision-making process periodically calculates a comprehensive score every 50ms (other times can be used, depending on user needs) based on the health score, process status, resource load score determined by resource load, network status score determined by network status, and forced restart score determined by external forced restart command to decide whether to restart the business process. Intelligent decision-making is made by combining the health score, process status, resource load, network status and external forced restart command.

[0079] The process decision unit is the core intelligent judgment unit and serves as the decision maker. It has the following advantages: high frequency and low latency: it can quickly identify business process anomalies and respond promptly; strong configurability: it supports setting weights and thresholds for different control systems, as well as different polling times (this embodiment uses a polling time of 50ms); flexible logic: it can expand more scoring dimensions to obtain a more accurate comprehensive score.

[0080] The decision-making process calls the shared memory interface to access the output parameters of the diagnosis process, that is, the operating status and health score. After making a decision based on the operating status and health score, the decision result is stored in the shared memory for access by the restart process (that is, the process restart unit) included in the management process. The input parameters of the decision-making process are the output parameters of the diagnosis process. Logical calculations are performed based on the values ​​of these output parameters to output the host action (business process restart or no action).

[0081] The decision-making process is divided into 5 key stages, as shown below.

[0082] (1) Input collection phase: Read the running status and health score of each business process in the shared memory, as well as the external input forced restart command, a higher-level instruction, etc.

[0083] (2) State fusion stage: Comprehensively evaluate multi-dimensional indicators such as process status, resource load and network status, health score and forced restart command in the running state to obtain a comprehensive score.

[0084] (3) Logical judgment stage: Perform logical judgment of actions based on the strategy table to obtain decision results.

[0085] (4) Instruction generation stage: write the decision result into shared memory.

[0086] (5) Output synchronization phase: for the restart process to read and execute the decision results in real time.

[0087] The input parameters of the decision-making process mainly come from two parts, as shown below.

[0088] (1) Output parameters of the diagnostic process: Health Score (H), IsAlive (A), Resource Load (R), and Network Score (N).

[0089] The resource load score is determined based on CPU usage and memory usage. The specific calculation process is as follows.

[0090] 1) CPU load score .

[0091] Read the / proc / PID / stat file to get the CPU idle time and total CPU time, = (1-CPU idle time / total CPU time) 100.

[0092] 2) Memory load score .

[0093] Read the VmRSS field in the / proc / PID / status file to get the memory usage, and read the MemTotal field in the / proc / meinfo file to get the total system memory. = (memory usage / total system memory) 100.

[0094] 3) Resource load score R= , and They are the weights of CPU and memory respectively, which can be adjusted according to specific circumstances.

[0095] The network status score is determined as follows: if the network is unobstructed, the network status score is 100 points; if the network is interrupted / abnormal, the network status score is 0 points.

[0096] (2) Externally input forced restart command: The forced restart score (ForceCommand, F) determined by the forced restart command. When the external input is a forced restart command for the business process, the forced restart score is 100; otherwise, the forced restart score is 0.

[0097] The decision logic adopts the state scoring and priority scheduling mode: it supports multi-indicator weighted decision-making in complex operating environments, sets weights and priorities for each variable, calculates the comprehensive score, formulates actions, and obtains decision results.

[0098] The formula for calculating the comprehensive score is shown below.

[0099] .

[0100] in, For comprehensive rating; 、 、 、 Health score , Network Status Score , Resource Load Score and forced restart rating The weight of Indicates that the process status is running. Indicates that the process status is zombie.

[0101] After calculating the comprehensive score, the decision result is obtained according to the strategy table, which is: ≥80, no action, normal recording; At 60-79, the abnormality is slight, and we will continue to observe; If the value is between 40 and 59, it is abnormal and a restart is recommended. <40, severe abnormality, immediate restart. It should be noted that the above values ​​(i.e. 80, 60, 40) can be adjusted according to user needs.

[0102] At this time, in this embodiment, when a decision is made on whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, and a decision result is obtained, the process decision unit is used to determine the resource load score based on the resource load, and determine the network status score based on the network status. When the process status is running, the health score, resource load score, network status score and forced restart score are weighted and summed to obtain a comprehensive score. When the process status is dead, the comprehensive score is set to 0 to obtain a comprehensive score; when the comprehensive score is less than a third preset threshold (i.e., 40), it is determined that the business process needs to be restarted, and the decision result is restart; when the comprehensive score is greater than or equal to the third preset threshold, it is determined that the business process does not need to be restarted, and the decision result is not to restart.

[0103] The process decision unit is also used to store the comprehensive score in the message area. The process management module also includes a process warning unit. The process warning unit is used to obtain the comprehensive score of the business process from the message area for each business process. If the comprehensive score is greater than or equal to the fourth preset threshold (i.e., 80), the business process is determined to be normal. If the comprehensive score is less than the fourth preset threshold and greater than or equal to the third preset threshold, the business process is determined to be abnormal. When it is determined that the business process is abnormal for multiple consecutive times, an alarm message is generated and the alarm message is stored in the message area.

[0104] (3) Process restart and exception recovery.

[0105] When a business process is detected to have exited abnormally or deadlocked, that is, when the decision is to restart, the restart process terminates the faulty process and restarts it, ensuring uninterrupted business. During the restart, cold start and hot start modes are provided to prevent frequent process restarts due to misjudgments. This function cyclically scans the decision results issued by the decision process. If the decision result is to restart the process, it obtains information about all faulty processes, records the faulty process crash log, terminates the faulty process, restarts a new process, and updates shared memory.

[0106] The process restart function is implemented by a restart process, and calls the shared memory access interface and the logging interface. The process restart operation flow is as follows Figure 5 As shown, Figure 5 In the example, the Usleep function represents the suspension of execution, 25ms is a polling cycle time, and after the process is restarted, it waits for 25ms. After the process is restarted, the decision result is re-queried.

[0107] This embodiment provides multiple exception handling methods, as shown below.

[0108] (1) The restart process callback timed out.

[0109] Before restarting the failed process, all other non-faulty processes will transition to the APPRestarting state, and their process restart callbacks will be called to notify them of the impending restart. If the non-faulty processes do not transition to the same state within the specified time, the non-faulty processes that have not completed the transition will be terminated.

[0110] (2) The restart process command failed to execute successfully for 5 consecutive times.

[0111] When executing the restart command, if the return value is less than 0, it means that the command execution failed. You can repeat the command five times. If the command fails five times in a row, the log information will be printed.

[0112] At this time, in this embodiment, when the service process is restarted, the process restart unit is used to terminate the service process and restart the service process. The startup mode used during the restart includes a hot start mode and a cold start mode.

[0113] (4) Master-subprocess collaboration mechanism.

[0114] This embodiment employs a process daemon architecture, specifically a master-subprocess collaboration mechanism. A daemon process (NicRdDaemon) monitors the operational status of multiple subprocesses (including management processes and service processes) to ensure that critical DCS processes do not exit abnormally. The daemon monitors management processes (diagnosis processes, decision-making processes, restart processes, warning processes within the process warning unit, synchronization processes within the synchronization unit, and allocation processes within the memory allocation unit). The daemon is also included in the process scope managed by the managed processes. If a subprocess fails, the daemon restarts the failed process. If the daemon fails, the management process also restarts the daemon. Both the daemon and management processes register with the memory allocation unit to obtain the corresponding message area.

[0115] The daemon is responsible for managing all child processes, including startup, health diagnosis, decision-making, and restart for exception recovery. It uses inter-process communication to ensure efficient interaction between the daemon and child processes. The daemon calls a shared memory interface to access the registered PIDs of child processes, polling the health of the child processes using those PIDs. If a child process crashes, it restarts it.

[0116] At this time, in this embodiment, the daemon system of the multi-process control system also includes: a daemon process module, the daemon process module includes a daemon process, the process diagnosis unit includes a diagnosis process, the process decision unit includes a decision process, the process restart unit includes a restart process, the daemon process, the diagnosis process, the decision process and the restart process respectively correspond to a message area in the shared memory, the daemon process is used to store the running status of the daemon process in the message area, the diagnosis process is used to store the running status of the diagnosis process in the message area, the decision process is used to store the running status of the decision process in the message area, and the restart process is used to store the running status of the restart process in the message area.

[0117] The daemon module is used to perform health diagnosis, decision-making and restart of the diagnosis process, decision-making process, restart process and each business process.

[0118] The process diagnosis unit is also used to perform health diagnosis on the daemon process, the process decision unit is also used to make decisions on the daemon process, and the process restart unit is also used to restart the daemon process.

[0119] The warning process, synchronization process, and allocation process each correspond to a message area in shared memory. The warning process stores its running status in the message area, the synchronization process stores its running status in the message area, and the allocation process stores its running status in the message area. The daemon module is responsible for health diagnosis, decision-making, and restarting of the warning, synchronization, and allocation processes.

[0120] Compared with the traditional process daemon solution, this embodiment has the following advantages.

[0121] (1) Higher system stability: Using a master-subprocess architecture and shared memory monitoring to improve fault response speed and ensure that DCS key processes are always running.

[0122] (2) Accurate health assessment: Through multi-dimensional monitoring (CPU, memory, heartbeat, etc.), the true health status of the process is judged, the health score is accurately determined, and the false restart is reduced.

[0123] (3) Intelligent decision-making: Based on health scores and network status, adaptively adjust process management strategies to improve system robustness.

[0124] (4) Efficient exception recovery: When a process exits abnormally or deadlocks, the system can automatically detect and recover without manual intervention, thereby improving operation and maintenance efficiency.

[0125] Example 2.

[0126] This embodiment provides a multi-process control system guard method, based on the multi-process control system guard system described in Example 1 to work, such as Figure 6 As shown, the guard method of the multi-process control system includes the following steps.

[0127] S1: For each business process, the process diagnosis unit obtains the running status of the business process from the message area, performs health diagnosis on the business process based on the running status of the business process, obtains a health score, and stores the health score in the message area; the running status includes heartbeat timestamp, process status, resource load, network status and error alarm status.

[0128] S2: For each business process, the process decision unit obtains the health score, process status, resource load and network status of the business process from the message area, makes a decision on whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, obtains a decision result, and stores the decision result in the message area; the forced restart score is determined based on whether a forced restart command for the business process is input externally.

[0129] S3: The process restarting unit obtains the decision result of each business process from the message area, and restarts the business process when the decision result is restart.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A guard system for a multi-process control system, characterized in that: The daemon system of the multi-process control system includes: a shared memory and a process management module, wherein a business process in the multi-process control system corresponds to a message area in the shared memory, the business process is used to store the running status of the business process in the message area, the process management module includes a process diagnosis unit, a process decision unit and a process restart unit; the running status includes a heartbeat timestamp, process status, resource load, network status and error alarm status; The process diagnosis unit is configured to obtain, for each of the business processes, an operating status of the business process from the message area, perform health diagnosis on the business process based on the operating status of the business process to obtain a health score, and store the health score in the message area; The process decision unit is configured to obtain, for each of the business processes, a health score, a process status, a resource load, and a network status of the business process from the message area, make a decision on whether the business process needs to be restarted based on the health score, process status, resource load, network status, and forced restart score of the business process, obtain a decision result, and store the decision result in the message area; the forced restart score is determined based on whether a forced restart command for the business process is input externally; The process restarting unit is configured to obtain a decision result of each of the service processes from the message area, and restart the service process when the decision result is restart.

2. The multi-process control system guard system according to claim 1, characterized in that: The business process is further configured to send a heartbeat signal to the message area at intervals of a first preset time, and the message area records a timestamp of the heartbeat signal to obtain a heartbeat timestamp; When performing a health diagnosis on the business process based on the running status of the business process to obtain a health score, the process diagnosis unit is configured to, when the heartbeat timestamp has not been updated for a second preset time, reduce the health score obtained from the previous health diagnosis to obtain a first score; when the process status is zombie, reduce the health score obtained from the previous health diagnosis to obtain a second score; when the process status is running, increase the health score obtained from the previous health diagnosis to obtain a second score; when the resource load exceeds a first preset threshold, reduce the health score obtained from the previous health diagnosis to obtain a third score; when the network status is network interruption, transmission failure, or round-trip delay exceeds a second preset threshold, reduce the health score obtained from the previous health diagnosis to obtain a fourth score; when the error alarm status is multiple consecutive errors or alarms, reduce the health score obtained from the previous health diagnosis to obtain a fifth score; when the error alarm status is no errors or alarms, increase the health score obtained from the previous health diagnosis to obtain the fifth score; and perform a weighted average of the first score, the second score, the third score, the fourth score, and the fifth score to perform a health diagnosis on the business process based on the running status of the business process to obtain a health score.

3. The multi-process control system guard system according to claim 2, characterized in that: The resource load includes a CPU occupancy rate and a memory occupancy rate. At this time, the resource load exceeding the first preset threshold specifically includes: the CPU occupancy rate exceeds the CPU occupancy rate threshold or the memory occupancy rate exceeds the memory occupancy rate threshold; The upper limit of the first score, the second score, the third score, the fourth score, and the fifth score is 100, and the lower limit is 0.

4. The multi-process control system guard system according to claim 1, characterized in that: When a decision is made on whether the business process needs to be restarted based on the health score, process status, resource load, network status and forced restart score of the business process, and a decision result is obtained, the process decision unit is used to determine the resource load score based on the resource load, and determine the network status score based on the network status; when the process status is running, the health score, the resource load score, the network status score and the forced restart score are weightedly summed to obtain a comprehensive score; when the process status is dead, the comprehensive score is set to 0 to obtain a comprehensive score; when the comprehensive score is less than a third preset threshold, it is determined that the business process needs to be restarted, and the decision result is restart; when the comprehensive score is greater than or equal to the third preset threshold, it is determined that the business process does not need to be restarted, and the decision result is not to restart.

5. The multi-process control system guard system according to claim 1, characterized in that: The daemon system of the multi-process control system further includes: a daemon process module, the daemon process module includes a daemon process, the process diagnosis unit includes a diagnosis process, the process decision unit includes a decision process, the process restart unit includes a restart process, the daemon process, the diagnosis process, the decision process and the restart process respectively correspond to a message area in the shared memory, the daemon process is used to store the running status of the daemon process in the message area, the diagnosis process is used to store the running status of the diagnosis process in the message area, the decision process is used to store the running status of the decision process in the message area, and the restart process is used to store the running status of the restart process in the message area; The daemon process module is used to perform health diagnosis, decision-making and restart on the diagnosis process, the decision-making process, the restart process and each of the business processes; The process diagnosis unit is further used to perform health diagnosis on the daemon process, the process decision unit is further used to make a decision on the daemon process, and the process restart unit is further used to restart the daemon process.

6. The multi-process control system guard system according to claim 4, characterized in that: The process decision unit is also used to store the comprehensive score in the message area. The process management module also includes a process warning unit. The process warning unit is used to obtain the comprehensive score of the business process from the message area for each business process. If the comprehensive score is greater than or equal to a fourth preset threshold, the business process is determined to be normal. If the comprehensive score is less than the fourth preset threshold and greater than or equal to the third preset threshold, the business process is determined to be abnormal. When it is determined that the business process is abnormal for multiple consecutive times, an alarm message is generated and the alarm message is stored in the message area.

7. The multi-process control system guard system according to claim 1, characterized in that: The process management module also includes a synchronization unit, which is used to write the running status of all business processes in the shared memory into a local log file every third preset time, and upload the running status and health score of all business processes in the shared memory to the central server every fourth preset time.

8. The multi-process control system guard system according to claim 1, characterized in that: The process management module further includes a memory allocation unit. When the business process is started, the business process is configured to send registration information to the memory allocation unit. The memory allocation unit is configured to allocate an idle message area to the business process upon receiving the registration information. The registration information includes the PID of the business process. The memory allocation unit stores a memory mapping table, which is used to mark the occupancy status of each message area. The occupancy status includes being used, idle, and to be cleared.

9. The multi-process control system guard system according to claim 1, characterized in that: When restarting the service process, the process restart unit is used to terminate the service process and restart the service process; the startup mode used during restart includes a hot start mode and a cold start mode.

10. A guarding method for a multi-process control system, operating based on the guarding system of the multi-process control system according to any one of claims 1 to 9, characterized in that: The guard method of the multi-process control system includes: The process diagnosis unit obtains the running status of each business process from the message area, performs health diagnosis on the business process based on the running status of the business process, obtains a health score, and stores the health score in the message area; the running status includes heartbeat timestamp, process status, resource load, network status and error alarm status; The process decision unit obtains the health score, process status, resource load, and network status of each business process from the message area, and makes a decision on whether the business process needs to be restarted based on the health score, process status, resource load, network status, and forced restart score of the business process, obtains a decision result, and stores the decision result in the message area; the forced restart score is determined based on whether a forced restart command for the business process is input externally; The process restarting unit obtains a decision result of each service process from the message area, and restarts the service process when the decision result is restart.

Images