Storage system process management method, electronic device, storage medium, and program product

Abstract

The application discloses a storage system process management method, an electronic device, a storage medium and a program product, relates to the technical field of storage systems, and comprises the following steps: setting a first watchdog module in a kernel state and setting a second watchdog module in a daemon process, the first watchdog module is used for monitoring the daemon process, and the second watchdog module is used for monitoring an input / output process; based on the first watchdog module and the second watchdog module, state information of the daemon process and state information of the input / output process are determined, and then a current fault scenario is determined; based on a corresponding relationship table of fault scenarios and recovery strategies, a target recovery strategy corresponding to the current fault scenario is determined, and process recovery is performed based on the target recovery strategy. The application avoids the situation that the system is unresponsive for a long time and key fault log information is lost during abnormal restart, solves the technical problem that a process exception handling mechanism in the prior art reduces the reliability of a storage system, and achieves the technical effect of improving the reliability of the storage system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage systems, and particularly relates to a storage system process management method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] In a storage system, a daemon process and an input output (Input Output, referred to as: IO) process are key components for guaranteeing continuous and stable operation of services, wherein the IO process is responsible for processing IO business, and the daemon process is a parent process of the IO process and undertakes an important responsibility of monitoring a life cycle of the IO process. In related technologies, a storage system adopts a dual-process daemon mechanism to implement exception handling of the IO process and the daemon process. Specifically, when the IO process abnormally exits, the daemon process can capture an abnormal exit state of the IO process, and complete data collection of a fixed memory region and an IO process exception log dumping operation. When the daemon process abnormally exits, a memory snapshot file of the daemon process is generated by relying on a Linux system, so as to facilitate a developer to locate an abnormal reason of the daemon process.

[0003] However, this mechanism may cause a system to be unresponsive for a long time, or lose key fault log information at an abnormal restart, in an extreme scenario, such as a central processing unit (Central Processing Unit, referred to as: CPU) resource being in a starvation state, the daemon process being deadlocked, a debugging tool being interfered, and the like, thereby reducing reliability of the storage system. SUMMARY

[0004] The present application provides a storage system process management method, an electronic device, a storage medium and a program product, to at least solve the problem that a process exception handling mechanism in related technologies reduces reliability of a storage system.

[0005] The present application provides a storage system process management method, comprising:

[0006] setting a first watchdog module in a kernel state, and setting a second watchdog module in a daemon process, wherein the first watchdog module is used for monitoring the daemon process, and the second watchdog module is used for monitoring an input output process;

[0007] determining state information of the daemon process and state information of the input output process based on the first watchdog module and the second watchdog module;

[0008] determining a current fault scenario based on the state information of the daemon process and the state information of the input output process;

[0009] determining a target recovery strategy corresponding to the current fault scenario based on a correspondence relationship table of the fault scenario and the recovery strategy;

[0010] recovering the daemon process and / or the input output process based on the target recovery strategy.

[0011] The application further provides an electronic device, comprising a memory for storing a computer program, and a processor for executing the computer program to implement the steps of any of the storage system process management methods.

[0012] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the storage system process management methods.

[0013] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of any of the storage system process management methods.

[0014] According to the application, the first watchdog module is set in the kernel state, and the second watchdog module is set in the daemon process, wherein the first watchdog module is used for monitoring the daemon process, and the second watchdog module is used for monitoring the input / output process; based on the first watchdog module and the second watchdog module, the state information of the daemon process and the state information of the input / output process are determined; based on the state information of the daemon process and the state information of the input / output process, the current fault scenario is determined; based on the corresponding relationship table of the fault scenario and the recovery strategy, the target recovery strategy corresponding to the current fault scenario is determined; and based on the target recovery strategy, the daemon process and / or the input / output process is recovered. By determining the current fault scenario when the daemon process or the input / output process is abnormal, and recovering the daemon process or the input / output process based on the target recovery strategy corresponding to the current fault scenario, the situation that the system is unresponsive for a long time and the critical fault log is lost during abnormal restart caused by the process abnormality processing mechanism in the related art is avoided, thereby the technical problem that the process abnormality processing mechanism in the related art reduces the reliability of the storage system is solved, and the technical effect of improving the reliability of the storage system is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A structural schematic diagram of a storage system process management system provided by an embodiment of the application;

[0017] Figure 2 A flowchart of a storage system process management method provided by an embodiment of the application;

[0018] Figure 3A schematic diagram of setting a watchdog module provided by an embodiment of the present application is provided.

[0019] Figure 4 A flowchart of another storage system process management method provided by an embodiment of the present application is provided.

[0020] Figure 5 A schematic diagram of a finite state machine model provided by an embodiment of the present application is provided.

[0021] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0023] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0024] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0025] In a storage system, a daemon process and an IO process are key components to ensure continuous and stable operation of storage services, wherein the IO process is a process for processing IO business in the storage system, and the daemon process is a parent process of the IO process, responsible for monitoring the life cycle of the child process, i.e., the IO process, automatically restarting when detecting abnormal exit of the IO process, and ensuring continuous operation of key storage services. In related technologies, a dual-process daemon mechanism is used to implement abnormal processing of the IO process and the daemon process. Specifically, when the IO process abnormally exits, the daemon process captures the abnormal exit state of the IO process through a wait mechanism, completes data collection in a Memory Pining area and dumping of a dump log of the IO process. After the dump log is generated, the memory is safely unpinned to avoid resource leakage.

[0026] Memory Pinning is a mechanism in Linux systems that forces critical data to reside in physical memory. By preventing memory from being swapped out or reclaimed, it ensures that the data segments of IO processes can still be accessed after the process exits.

[0027] When the daemon itself exits abnormally, the Linux system's core dump mechanism is relied on to generate a core dump file for the daemon, helping developers locate the cause of the crash and then restart the storage system. Core dump is a memory snapshot file generated by Linux when a process terminates abnormally (such as crash or signal kill), recording the memory state (heap, stack, global variables), register values (CPU context), and thread information (call stack, lock status) at the time of the crash. It is used for post-mortem debugging to help developers locate the cause of the process crash.

[0028] However, this mechanism has failure scenarios and systemic risks in actual running environments. Table 1 shows the correspondence between failure scenarios and specific manifestations, impact ranges.

[0029] Table 1

[0030]

[0031] As shown in Table 1, the above dual-process daemon mechanism may cause the system to be unresponsive for a long time or lose critical fault log information during abnormal restart in the above failure scenarios. In the case of losing critical fault log information, it is necessary to rely on black box inference to find the abnormal reason, which prolongs the mean time to repair (MTTR) and reduces the reliability of the storage system.

[0032] To solve the above problems, the embodiment of the present application provides a storage system process management method, an electronic device, a storage medium and a program product. The method comprises the following steps: setting a first watchdog module in a kernel state, and setting a second watchdog module in a daemon process, wherein the first watchdog module is used for monitoring the daemon process, and the second watchdog module is used for monitoring an input / output process; determining state information of the daemon process and state information of the input / output process based on the first watchdog module and the second watchdog module; determining a current fault scenario based on the state information of the daemon process and the state information of the input / output process; determining a target recovery strategy corresponding to the current fault scenario based on a corresponding relationship table of fault scenarios and recovery strategies; and recovering the daemon process and / or the input / output process based on the target recovery strategy. The method provided by the above scheme can monitor the daemon process and the input / output process through the first watchdog module and the second watchdog module, determine the current fault scenario when the daemon process or the input / output process is abnormal, recover the daemon process or the input / output process based on the target recovery strategy corresponding to the current fault scenario, and thus the technical problem that the process abnormality processing mechanism in the related art reduces the reliability of the storage system can be solved, and the technical effect of improving the reliability of the storage system is achieved.

[0033] In combination with a specific application environment architecture or a specific hardware architecture on which the storage system process management method is dependent, the specific application environment architecture or the specific hardware architecture is described herein.

[0034] The storage system process management method, the electronic device, the storage medium and the program product provided by the embodiment of the present application are suitable for managing the processes of the storage system. Figure 1 As shown in FIG. 1, a structure schematic diagram of a storage system process management system provided by the embodiment of the present application is shown, and the storage system process management system comprises a first watchdog module, a second watchdog module, a state collector and a recovery strategy module. The first watchdog module is used for monitoring a daemon process, and the second watchdog module is used for monitoring an input / output process. The state collector is used for acquiring state information of the daemon process and state information of the input / output process, and determining a current fault scenario based on the state information of the daemon process and the state information of the input / output process. The recovery strategy module is used for determining a target recovery strategy corresponding to the current fault scenario based on the current fault scenario, and recovering the daemon process and / or the input / output process based on the target recovery strategy. The target recovery strategy can be process restart, system restart, key log recording, etc.

[0035] The embodiment of the present application provides a storage system process management method, which is applied to a central processing unit in a storage system, Figure 2A flowchart of a storage system process management method provided by an embodiment of the present application is shown in FIG. 8. Figure 2 The storage system process management method includes the following steps.

[0036] In step S201, a first watchdog module is set in a kernel state, and a second watchdog module is set in a daemon process. The first watchdog module is used to monitor the daemon process, and the second watchdog module is used to monitor an input / output process.

[0037] Figure 3 A schematic diagram of setting the watchdog modules provided by an embodiment of the present application is shown in FIG. 9. Figure 3 The first watchdog module is set in the kernel state, and the first watchdog module is simulated based on a timer in the Linux software kernel state, so as to monitor the daemon process. The second watchdog module is set in the daemon process in the user state, and the second watchdog module is used to monitor the IO process in the user state.

[0038] The first watchdog module and the second watchdog module are externally provided with a watchdog timeout function, a dog feeding function, and a thread wake-up function after the dog barking. The watchdog timeout function is a basic judgment logic of the watchdog module, and the core is to preset a time threshold and start the built-in timer to start timing. If the monitored process does not feed the dog within the time threshold, it is determined that the monitored process is abnormal. The dog feeding function is a feedback mechanism of the monitored process to the watchdog module that “I am running normally”, which is used to reset the timer of the watchdog module. When the monitored process is running normally, it will actively send a dog feeding signal to the corresponding watchdog module at a fixed period, and the watchdog module will reset the timer immediately after receiving the dog feeding signal, so as to avoid triggering the timeout judgment. The “dog barking” is an image description of the timeout without feeding the dog, that is, the timeout judgment is triggered, and the “thread wake-up function” is an execution mechanism of the watchdog module to start the abnormal processing flow after the dog barking.

[0039] It should be noted that when the IO process is not started, the dog feeding mode of the daemon process is to send a dog feeding signal to the first watchdog module every third preset time interval, so as to reset the first preset time threshold in the first watchdog module, so that the timer is in a state of not exceeding the first preset time threshold.

[0040] After the IO process is started, the dog feeding mode of the daemon process is to send a dog feeding signal to the first watchdog module every first preset time interval, so as to make the timer in the first watchdog module start timing again. It can be understood that after the IO is started, the first preset time threshold is updated to the initial state.

[0041] After the IO process is started, the dog feeding mode of the IO process is to send a dog feeding signal to the second watchdog module every second preset time interval, so as to make the timer in the second watchdog module start timing again.

[0042] The kernel state provides components of the watchdog module, such as a timer, etc. The IO process and the daemon process in the user state feed the watchdog in the manner as described above, and the watchdog module is used to monitor whether the watchdog is barking, and then a corresponding processing strategy is taken.

[0043] In step S202, state information of the daemon process and state information of the input / output process are determined based on the first watchdog module and the second watchdog module.

[0044] In step S203, a current fault scenario is determined based on the state information of the daemon process and the state information of the input / output process.

[0045] In step S204, a target recovery strategy corresponding to the current fault scenario is determined based on a correspondence table of fault scenarios and recovery strategies.

[0046] In step S205, the daemon process and / or the input / output process are recovered based on the target recovery strategy.

[0047] It can be understood that if the current fault scenario is that the daemon process is normal and the input / output process is normal, it is determined that the target recovery strategy is: no operation, and the daemon process continues to be monitored based on the first watchdog module and the input / output process continues to be monitored based on the second watchdog module.

[0048] The storage system process management method provided by the embodiment of the present application can monitor the daemon process and the input / output process through the first watchdog module and the second watchdog module, determine a current fault scenario when the daemon process or the input / output process is abnormal, recover the daemon process or the input / output process based on a target recovery strategy corresponding to the current fault scenario, and thus avoid the situation that the process abnormality processing mechanism in the related art causes a long-time non-response of the system and a loss of critical fault logs in abnormal restart in some scenarios. Therefore, the technical problem that the process abnormality processing mechanism in the related art reduces the reliability of the storage system can be solved, and the technical effect of improving the reliability of the storage system is achieved.

[0049] By establishing a hybrid monitoring strategy in cooperation of the user state and the kernel state, the abnormal states of the IO process and the daemon process can be monitored at the same time, all abnormal states (including but not limited to: deadlocks, resource leaks, etc.) of the IO process and the daemon process are tracked in real time, a recovery strategy is established, an optimal recovery scheme is matched for different abnormal types, the log integrity in an extreme scenario is ensured, and the whole recording of the fault scene (including: register state, memory snapshot, call stack, etc.) is realized.

[0050] The embodiment of the present application provides a storage system process management method, which is applied to a central processing unit in a storage system, Figure 4 A flowchart of the storage system process management method provided by the embodiment of the present application is shown in FIG.Figure 4 As shown, the storage system process management method comprises:

[0051] At step S401, a first watchdog module is set in a kernel state, and a second watchdog module is set in a daemon process, wherein the first watchdog module is used to monitor the daemon process, and the second watchdog module is used to monitor an input / output process. For details, please refer to Figure 2 At step S201 of the embodiment shown, no further elaboration is given here.

[0052] At step S402, based on the first watchdog module and the second watchdog module, state information of the daemon process and state information of the input / output process are determined.

[0053] Specifically, the above step S402 comprises:

[0054] At step S4021, in a case where a timing time of a timer in the first watchdog module exceeds a first preset time threshold, the state information of the daemon process is determined as daemon process abnormality, wherein after the input / output process is started and begins to run, the daemon process sends a watchdog feeding signal to the first watchdog module every first preset time length, so as to make the timer in the first watchdog module restart timing.

[0055] Wherein, the first preset time length is less than the first preset time threshold. The first preset time threshold and the first preset time length are set by technicians, and no specific limitation is given here.

[0056] At step S4022, in a case where a timing time of a timer in the second watchdog module exceeds a second preset time threshold, the state information of the input / output process is determined as input / output process abnormality, wherein after the input / output process is started and begins to run, the input / output process sends a watchdog feeding signal to the second watchdog module every second preset time length, so as to make the timer in the second watchdog module restart timing.

[0057] Wherein, the second preset time length is less than the second preset time threshold. The second preset time threshold and the second preset time length are set by technicians, and no specific limitation is given here.

[0058] At step S403, based on the state information of the daemon process and the state information of the input / output process, a current fault scenario is determined.

[0059] Specifically, the above step S403 comprises:

[0060] At step S4031, in a case where the state information of the daemon process is daemon process abnormality, and the state information of the input / output process is input / output process abnormality, an abnormal type of the daemon process and an abnormal type of the input / output process are acquired, and based on the abnormal type of the daemon process and the abnormal type of the input / output process, the current fault scenario is determined.

[0061] In step S4032, when the state information of the daemon process is daemon process abnormal and the state information of the input / output process is input / output process normal, the abnormal type of the daemon process is acquired, and the current fault scenario is determined based on the abnormal type of the daemon process.

[0062] In step S4033, when the state information of the daemon process is daemon process normal and the state information of the input / output process is input / output process abnormal, the abnormal type of the input / output process is acquired, and the current fault scenario is determined based on the abnormal type of the input / output process.

[0063] In step S404, the target recovery strategy corresponding to the current fault scenario is determined based on the correspondence table of fault scenarios and recovery strategies. For details, refer to step S204 of the embodiment shown in Figure 2 The step S204 of the embodiment shown in FIG. 2 will not be repeated here.

[0064] In step S405, the daemon process and / or the input / output process are recovered based on the target recovery strategy. For details, refer to step S205 of the embodiment shown in Figure 2 The step S205 of the embodiment shown in FIG. 2 will not be repeated here.

[0065] The storage system process management method provided by the embodiment can more accurately locate the fault scenario by judging the states of the daemon process and the input / output process respectively and combining the abnormal types of the daemon process and the input / output process, and then take corresponding recovery measures according to the fault scenario, thereby improving the system reliability.

[0066] In some optional embodiments, the step S404 includes:

[0067] In step a1, when the current fault scenario is daemon process abnormal, input / output process normal, and the abnormal type of the daemon process is that the daemon process exists but the process state is a stop state, it is determined that the target recovery strategy corresponding to the current fault scenario is that a continue running signal is sent to the daemon process based on the first watchdog module to make the daemon process resume normal operation, and logs of the daemon process abnormality are recorded.

[0068] The daemon process exists but the process state is a stop state (STOP), which means that the daemon process is suspended by the operating system, and at this time, a continue running signal is sent to the daemon process based on the first watchdog module to make the daemon process resume normal operation.

[0069] Step a2, in the case that the current fault scenario is daemon abnormality, the input and output process is normal, and the abnormality type of the daemon is that the daemon exists but the process state is not the stop state, it is determined that the target recovery strategy corresponding to the current fault scenario is: based on the first watchdog module, the input and output process is actively terminated, the log information and the memory data of the input and output process are collected and saved, and the daemon is terminated, the memory snapshot file of the daemon is generated, and the daemon is restarted.

[0070] In the case that the current fault scenario is daemon abnormality, the input and output process is normal, and the abnormality type of the daemon is that the daemon exists but the process state is not the stop state, based on the first watchdog module, the IO process is actively terminated (kill), the log information (dump log) and the memory data (harddata data) of the IO process are collected and saved. At the same time, the daemon is killed, the coredump log (memory snapshot file) of the daemon is generated, and the daemon is restarted.

[0071] The storage system process management method provided by the embodiment of the present application improves the accuracy of system recovery by taking different target recovery strategies according to the abnormality type of the daemon in the case that the daemon is abnormal and the input and output process is normal.

[0072] In some optional embodiments, the storage system process management method further includes:

[0073] Step b1, if the number of times of executing the step of restarting the daemon exceeds a preset number threshold within a preset time period, the storage system is restarted, and the abnormality log of the daemon is recorded.

[0074] The preset time period can be any 24 hours, which is not limited specifically herein. The preset number threshold can be 5, which is only exemplarily described herein and is not limited specifically.

[0075] The storage system process management method provided by the embodiment of the present application automatically upgrades to the system-level recovery strategy when the recovery strategy of a single process fails, i.e., restarts the entire storage system, enhances the self-healing capability of the system under complex abnormal conditions, and reduces manual intervention.

[0076] In some optional embodiments, the step S404 includes:

[0077] Step c1, in the case that the current fault scenario is daemon abnormality, the input and output process is normal, and the abnormality type of the daemon is that the daemon does not exist, it is determined that the target recovery strategy corresponding to the current fault scenario is: based on the first watchdog module, the input and output process is actively terminated, the log information and the memory data of the input and output process are collected and saved, and the storage system is restarted.

[0078] The management strategy includes: automatically generating a memory snapshot file of the daemon when the daemon abnormally exits; starting an operating system level audit log, and setting a filter for the audit log to retain abnormal exit information of the daemon.

[0079] Specifically, when the daemon starts, the management strategy of the daemon is pre-configured based on a Linux core dump mechanism. The management strategy includes: enabling the Core Dump mechanism by ulimit -c unlimited to generate a core when the daemon abnormally exits. <pid>The file is a memory snapshot file, recording the daemon process state (memory state, thread information) at the time of crash.

[0080] The management policy further comprises: starting an operating system level audit log, focusing on monitoring the daemon process, recording external signals and operations that cause the daemon process to exit according to the monitoring level, that is, recording abnormal exit information of the daemon process.

[0081] That is, when the daemon process does not exist, the internal or external cause of the daemon process exit can be obtained due to the preconfigured management policy, the internal cause can be determined by analyzing the memory snapshot file of the daemon process, and the external cause can be determined by analyzing the audit log.

[0082] The storage system process management method provided by the embodiment of the application improves the accuracy of system recovery by taking a corresponding target recovery strategy according to the abnormal type of the daemon process when the daemon process is abnormal and the input / output process is normal.

[0083] In some optional embodiments, the step S404 comprises:

[0084] In step d1, when the current fault scenario is that the daemon process is normal, the input / output process is abnormal, and the abnormal type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scenario is determined as: based on the second watchdog module, the daemon process is actively awakened to make the daemon process actively terminate the input / output process, collect and save the abnormal log and memory data of the input / output process, and start a new input / output process.

[0085] It should be noted that if the number of times of the fault scenario that the daemon process is normal, the input / output process is abnormal, and the abnormal type of the input / output process is that the input / output process exists reaches a preset number threshold within a preset time period, the storage system is restarted.

[0086] The storage system process management method provided by the embodiment of the application improves the accuracy of system recovery by taking a corresponding target recovery strategy according to the abnormal type of the input / output process when the daemon process is normal and the input / output process is abnormal.

[0087] In some optional embodiments, the step S404 comprises:

[0088] In step e1, in the case that the current fault scenario is daemon abnormality and input / output process abnormality, the abnormality type of the daemon is that the daemon exists but the process state is a stop state, and the abnormality type of the input / output process is that the input / output process does not exist, it is determined that the target recovery strategy corresponding to the current fault scenario is: based on the first watchdog module, a continue running signal is sent to the daemon to make the daemon resume normal operation, and a log of the abnormality of the daemon is recorded; based on the second watchdog module, the abnormality log and the memory data of the input / output process are collected and saved.

[0089] It can be understood that after the abnormality log and the memory data of the input / output process are collected and saved, a new input / output process is started.

[0090] The storage system process management method provided by the embodiment of the present application improves the accuracy of system recovery by taking the corresponding target recovery strategy according to the abnormality type of the input / output process and the abnormality type of the daemon in the case that the daemon is abnormal and the input / output process is abnormal.

[0091] In some optional embodiments, the above step S404 includes:

[0092] In step f1, in the case that the current fault scenario is daemon abnormality and input / output process abnormality, the abnormality type of the daemon is that the daemon exists but the process state is not a stop state, and the abnormality type of the input / output process is that the input / output process does not exist, it is determined that the target recovery strategy corresponding to the current fault scenario is: based on the first watchdog module, the daemon is terminated, a memory snapshot file of the daemon is generated, and the daemon is restarted; based on the second watchdog module, the abnormality log and the memory data of the input / output process are collected and saved.

[0093] It can be understood that after the abnormality log and the memory data of the input / output process are collected and saved, a new input / output process is started.

[0094] The storage system process management method provided by the embodiment of the present application improves the accuracy of system recovery by taking the corresponding target recovery strategy according to the abnormality type of the input / output process and the abnormality type of the daemon in the case that the daemon is abnormal and the input / output process is abnormal.

[0095] In some optional embodiments, the above step S404 includes:

[0096] In step g1, in the case that the current fault scenario is daemon abnormality and input / output process abnormality, the abnormality type of the daemon is that the daemon does not exist, and the abnormality type of the input / output process is that the input / output process does not exist, it is determined that the target recovery strategy corresponding to the current fault scenario is: the storage system is restarted.

[0097] The storage system process management method provided in the embodiments of the present application improves the accuracy of system recovery by taking corresponding target recovery strategies according to the abnormal type of the input / output process and the abnormal type of the daemon process in the case of daemon process abnormality and input / output process abnormality.

[0098] In some optional embodiments, the step S404 comprises:

[0099] In the step h1, in the case of daemon process abnormality and input / output process abnormality, and the abnormal type of the daemon process is that the daemon process exists but the process state is a stop state and the abnormal type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scenario is determined as: sending a continue running signal to the daemon process based on the first watchdog module to make the daemon process resume normal running, and recording the log of the abnormality of the daemon process; actively terminating the input / output process based on the second watchdog module, and collecting and saving the abnormal log and memory data of the input / output process.

[0100] It can be understood that after the abnormal log and memory data of the input / output process are collected and saved, a new input / output process is started.

[0101] The storage system process management method provided in the embodiments of the present application improves the accuracy of system recovery by taking corresponding target recovery strategies according to the abnormal type of the input / output process and the abnormal type of the daemon process in the case of daemon process abnormality and input / output process abnormality.

[0102] In some optional embodiments, the step S404 comprises:

[0103] In the step i1, in the case of daemon process abnormality and input / output process abnormality, and the abnormal type of the daemon process is that the daemon process exists but the process state is not a stop state and the abnormal type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scenario is determined as: terminating the daemon process based on the first watchdog module, generating a memory snapshot file of the daemon process, and restarting the daemon process; actively terminating the input / output process based on the second watchdog module, and collecting and saving the abnormal log and memory data of the input / output process.

[0104] The storage system process management method provided in the embodiments of the present application improves the accuracy of system recovery by taking corresponding target recovery strategies according to the abnormal type of the input / output process and the abnormal type of the daemon process in the case of daemon process abnormality and input / output process abnormality.

[0105] In some optional embodiments, the step S404 comprises:

[0106] Step j1, in the case that the current fault scenario is daemon abnormality, input / output process abnormality, the abnormality type of the daemon is daemon nonexistence, and the abnormality type of the input / output process is input / output process existence, it is determined that the target recovery strategy corresponding to the current fault scenario is: restarting the storage system.

[0107] The storage system process management method provided by the embodiments of the present application improves the accuracy of system recovery by taking corresponding target recovery strategies according to the abnormality type of the input / output process and the abnormality type of the daemon in the case of daemon abnormality and input / output process abnormality.

[0108] In some optional embodiments, the above-mentioned storage system process management method further comprises:

[0109] Step k1, for any target fault scenario, if the number of occurrences of the target fault scenario reaches a preset number threshold within a preset time period, the storage system is restarted. The target fault scenario is any fault scenario except the fault scenario that the daemon is normal and the input / output process is normal.

[0110] It can be understood that after restarting the storage system, it is necessary to verify whether the daemon, the IO process, the first watchdog module and the second watchdog module are successfully initialized, if at least one of them is not successfully initialized, the step of restarting the storage system is returned. If the number of times of executing the step of restarting the storage system reaches a preset number threshold, and at least one of the daemon, the IO process, the first watchdog module and the second watchdog module is still not successfully initialized, an alarm is performed.

[0111] The storage system process management method provided by the embodiments of the present application triggers system restart when a certain fault scenario frequently occurs within a preset time period for all other fault scenarios except "daemon and IO process are normal", and clears persistent faults through system-level restart, thereby improving the self-healing capability of the system.

[0112] After the system is restarted, it is required to perform initialization state checking on the daemon, the IO process, the first watchdog module and the second watchdog module, if any module is not successfully initialized, the system is continued to be restarted, forming a closed-loop mechanism of verification-restart-reverification, ensuring that the system is in a truly available state after being restarted, rather than being formally restarted but the key modules are still unavailable. An alarm mechanism is set for the case that the number of times of restarting reaches a preset number threshold but the key modules are still not successfully initialized, avoiding the system from falling into an infinite restart dead loop, and timely reminding manual intervention to prevent the service from being unavailable for a long time.

[0113] Based on the storage system process management method provided in the embodiments of the present application, a finite state machine model can be constructed to cover recovery strategies for all fault scenarios. Figure 5 As shown in the schematic diagram of the finite state machine model provided in the embodiments of the present application, Figure 5 when the fault scenario is scenario 1, i.e., the daemon process is normal and the IO process is normal, it is determined that the recovery strategy is normal monitoring by the watchdog module; when the fault scenario is scenario 2, i.e., the daemon process is abnormal and the IO process is normal, it is determined that the recovery strategy is to recover the daemon process, for details, refer to the foregoing corresponding description, and here is not described again. After the daemon process is recovered, the fault scenario is updated to scenario 1. When the fault scenario is scenario 3, i.e., the daemon process is normal and the IO process is abnormal, it is determined that the recovery strategy is to recover the IO process, for details, refer to the foregoing corresponding description, and here is not described again. After the IO process is recovered, the fault scenario is updated to scenario 1. When the fault scenario is scenario 4, i.e., the daemon process is abnormal and the IO process is abnormal, it is determined that the recovery strategy is to recover the daemon process and the IO process, for details, refer to the foregoing corresponding description, and here is not described again.

[0114] It can be understood that the daemon process and the IO process can be handled according to the finite state machine model.

[0115] The storage system process management method provided in the embodiments of the present application adopts a high-reliability process monitoring mechanism based on a Linux software kernel timer to realize full-life cycle management of the daemon process and the IO process in the storage system, effectively guaranteeing the reliability, availability and serviceability (Reliability, Availability, Serviceability, referred to as: RAS) of the entire storage system, improving product quality and competitiveness. By adopting a hybrid detection mechanism: a timer is designed in the kernel mode, a monitoring mechanism is designed in the user mode, and a finite state machine model is combined to ensure monitoring of all abnormal states of the daemon process and the IO process of the storage device. All kinds of abnormal states are supported, such as process deadlock, debugging tool interference, and central processing unit resource in a starvation state. Corresponding hierarchical recovery strategies are designed for all kinds of fault scenarios, including process restart, system restart, data protection, and key log recording. Optimal recovery strategies are matched for different scenarios to realize rapid recovery after an abnormality. Log recording, fault positioning and other functions in all kinds of abnormal states are supported to ensure log integrity in all kinds of fault scenarios, to realize full-process recording of the fault scene, including: dump log of the IO process, core dump file of the daemon process, and audit log.

[0116] The storage system process management method provided by the embodiments of the present application can enable the storage system IO process to safely and reliably process IO services, match optimal recovery strategies for different abnormal types, ensure log integrity in different fault scenarios, realize full-process recording of fault scenes, reduce fault processing costs, reduce manual intervention, shorten fault positioning time, and have strong compatibility and can be deployed without hardware modification.

[0117] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware, but in many cases the former is a better embodiment.

[0118] The embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, comprising a processor 601 and a memory 602, the memory 602 storing a computer program, and the processor 601 is configured to run the computer program to perform the steps in any of the above storage system process management method embodiments. Figure 6

[0119] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above storage system process management method embodiments when running.

[0120] In an exemplary embodiment, the above computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0121] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps in any of the above storage system process management method embodiments.

[0122] The embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in any of the above storage system process management method embodiments.

[0123] ​Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the claimed application. Accordingly, modifications and / or additions, other than those explicitly described herein, can be obvious to those skilled in the art in the light of this disclosure. The claimed application is intended to embrace all such modifications and / or additions.

[0124] The above provides a kind of storage system process management method, electronic equipment, storage medium and program product provided by the present application in detail.The principle and implementation of the present application are described in the specific examples in this paper, the above description of examples is only applicable to help understanding the method and its core idea of the present application.It should be pointed out that, for those skilled in the art, without departing from the principle of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.< / pid>

Claims

1. A storage system process management method characterized by comprising: The method comprises the following steps: setting a first watchdog module in a kernel state and a second watchdog module in a daemon process, wherein the first watchdog module is used to monitor the daemon process, and the second watchdog module is used to monitor an input / output process; determining state information of the daemon process and state information of the input / output process based on the first watchdog module and the second watchdog module; determining a current fault scenario based on the state information of the daemon process and the state information of the input / output process; determining a target recovery strategy corresponding to the current fault scenario based on a corresponding relationship table of fault scenarios and recovery strategies; recovering the daemon process and / or the input / output process based on the target recovery strategy; The method further comprises the following steps: in a case where the state information of the daemon process is daemon process abnormality and the state information of the input / output process is input / output process abnormality, obtaining an abnormality type of the daemon process and an abnormality type of the input / output process, and determining the current fault scenario based on the abnormality type of the daemon process and the abnormality type of the input / output process; in a case where the state information of the daemon process is daemon process abnormality and the state information of the input / output process is input / output process normality, obtaining an abnormality type of the daemon process, and determining the current fault scenario based on the abnormality type of the daemon process; in a case where the state information of the daemon process is daemon process normality and the state information of the input / output process is input / output process abnormality, obtaining an abnormality type of the input / output process, and determining the current fault scenario based on the abnormality type of the input / output process; The method further comprises the following steps: in a case where the current fault scenario is daemon process abnormality, the input / output process is normality, and the abnormality type of the daemon process is that the daemon process exists but the process state is a stop state, determining that the target recovery strategy corresponding to the current fault scenario is that a continue running signal is sent to the daemon process based on the first watchdog module, so that the daemon process resumes normal operation, and a log of the abnormality of the daemon process is recorded; in a case where the current fault scenario is daemon process abnormality, the input / output process is normality, and the abnormality type of the daemon process is that the daemon process exists but the process state is not a stop state, determining that the target recovery strategy corresponding to the current fault scenario is that the input / output process is actively terminated based on the first watchdog module, log information and memory data of the input / output process are collected and saved, the daemon process is terminated, a memory snapshot file of the daemon process is generated, and the daemon process is restarted. In a case where the current fault scenario is daemon abnormality, input / output process abnormality, the abnormality type of the daemon is daemon nonexistence, and the abnormality type of the input / output process is input / output process nonexistence, it is determined that the target recovery strategy corresponding to the current fault scenario is: restarting the storage system.

2. The method of claim 1, wherein, The state information of the daemon and the state information of the input / output process are determined based on the first watchdog module and the second watchdog module, and the determination includes: In a case where the timing time of the timer in the first watchdog module exceeds a first preset time threshold, it is determined that the state information of the daemon is daemon abnormality, wherein after the input / output process is started and begins to run, the daemon sends a watchdog feeding signal to the first watchdog module every interval of a first preset time length, so that the timer in the first watchdog module restarts timing. In a case where the timing time of the timer in the second watchdog module exceeds a second preset time threshold, it is determined that the state information of the input / output process is input / output process abnormality, wherein after the input / output process is started and begins to run, the input / output process sends a watchdog feeding signal to the second watchdog module every interval of a second preset time length, so that the timer in the second watchdog module restarts timing.

3. The method of claim 1, wherein, The method further includes: If the number of times of executing the step of restarting the daemon exceeds a preset number threshold within a preset time period, restarting the storage system, and recording the abnormality log of the daemon.

4. The method of claim 1, wherein, The target recovery strategy corresponding to the current fault scenario is determined based on the correspondence relationship table of fault scenarios and recovery strategies, and the determination includes: In a case where the current fault scenario is daemon abnormality, the input / output process is normal, and the abnormality type of the daemon is daemon nonexistence, it is determined that the target recovery strategy corresponding to the current fault scenario is: based on the first watchdog module, actively terminating the input / output process, collecting and saving the log information and memory data of the input / output process, and restarting the storage system. Wherein, when the daemon is started, a management strategy is pre-configured, and the management strategy includes: when the daemon abnormally exits, automatically generating a memory snapshot file of the daemon; starting an operating system level audit log, and setting a filter for the audit log, so that the audit log retains the abnormal exit information of the daemon.

5. The method of claim 1, wherein, The target recovery strategy corresponding to the current fault scenario is determined based on the correspondence relationship table of fault scenarios and recovery strategies, and the determination includes: In a case where the current fault scenario is daemon normality, the input / output process is abnormality, and the abnormality type of the input / output process is input / output process nonexistence, it is determined that the target recovery strategy corresponding to the current fault scenario is: based on the second watchdog module, actively waking up the daemon, so that the daemon actively terminates the input / output process, collects and saves the abnormality log and memory data of the input / output process, and starts a new input / output process.

6. The method of claim 1, wherein, The correspondence table based on the fault scene and the recovery strategy determines a target recovery strategy corresponding to the current fault scene, and includes: In a case where the current fault scene is the daemon process exception, the input / output process exception, the exception type of the daemon process is that the daemon process exists but the process state is not the stop state, and the exception type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scene is determined as follows: based on the first watchdog module, the daemon process is terminated, a memory snapshot file of the daemon process is generated, and the daemon process is restarted; based on the second watchdog module, the exception log and the memory data of the input / output process are collected and saved.

7. The method of claim 1, wherein, The correspondence table based on the fault scene and the recovery strategy determines a target recovery strategy corresponding to the current fault scene, and includes: In a case where the current fault scene is the daemon process exception, the input / output process exception, the exception type of the daemon process is that the daemon process exists but the process state is not the stop state, and the exception type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scene is determined as follows: based on the first watchdog module, the daemon process is terminated, a memory snapshot file of the daemon process is generated, and the daemon process is restarted; based on the second watchdog module, the exception log and the memory data of the input / output process are collected and saved.

8. The method of claim 1, wherein, The correspondence table based on the fault scene and the recovery strategy determines a target recovery strategy corresponding to the current fault scene, and includes: In a case where the current fault scene is the daemon process exception, the input / output process exception, the exception type of the daemon process is that the daemon process exists but the process state is not the stop state, and the exception type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scene is determined as follows: based on the first watchdog module, the daemon process is terminated, a memory snapshot file of the daemon process is generated, and the daemon process is restarted; based on the second watchdog module, the exception log and the memory data of the input / output process are collected and saved.

9. The method of claim 1, wherein, In a case where the current fault scene is the daemon process exception, the input / output process exception, the exception type of the daemon process is that the daemon process exists but the process state is not the stop state, and the exception type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scene is determined as follows: based on the first watchdog module, the daemon process is terminated, a memory snapshot file of the daemon process is generated, and the daemon process is restarted; based on the second watchdog module, the exception log and the memory data of the input / output process are collected and saved. The correspondence table based on the fault scene and the recovery strategy determines a target recovery strategy corresponding to the current fault scene, and includes:

10. The method of claim 1, wherein, In a case where the current fault scene is the daemon process exception, the input / output process exception, the exception type of the daemon process is that the daemon process exists but the process state is not the stop state, and the exception type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scene is determined as follows: based on the first watchdog module, the daemon process is terminated, a memory snapshot file of the daemon process is generated, and the daemon process is restarted; based on the second watchdog module, the exception log and the memory data of the input / output process are collected and saved. In a case where the current fault scenario is that the daemon process is abnormal and the input / output process is abnormal, and the abnormal type of the daemon process is that the daemon process exists but the process state is not the stop state and the abnormal type of the input / output process is that the input / output process exists, the target recovery strategy corresponding to the current fault scenario is determined as follows: based on the first watchdog module, the daemon process is terminated, the memory snapshot file of the daemon process is generated, and the daemon process is restarted; based on the second watchdog module, the input / output process is actively terminated, and the abnormal log and the memory data of the input / output process are collected and saved.

11. An electronic device, comprising: Comprise: a memory for storing a computer program; a processor for implementing the steps of the storage system process management method according to any one of claims 1 to 10 when executing the computer program.

12. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the storage system process management method according to any one of claims 1 to 10.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the storage system process management method according to any one of claims 1 to 10.

Images