Operating system state root cause inference method, apparatus, device, medium and product

Abstract

An operating system state root cause inference method, device, equipment, medium and product are provided. The operating system state root cause inference method comprises: obtaining a structured event with a predefined format in an operating system kernel; determining whether the operating system is abnormally stopped; in response to determining that the operating system is abnormally stopped, obtaining, by a user state service, state information of the operating system based on the structured event obtained in an operating system running cycle before abnormal stop through a predetermined state machine; and obtaining, by the user state service, an abnormal stop root cause based on the state information of the operating system through the predetermined state machine. Through the present disclosure, the structured degree and machine analyzability of operating system abnormal stop analysis can be significantly improved, different fault types can be more accurately distinguished, kernel invasiveness and maintenance costs can be reduced, the implementability of rule iteration and product deployment can be improved, and automatic inference and large-scale review can be realized.

Description

Technical Field

[0001] This disclosure relates to the field of operating systems, and more specifically, to a method, apparatus, device, medium, and product for inferring the root cause of operating system states. Background Technology

[0002] In operating systems (such as Linux), abnormal shutdowns (such as system crashes, unresponsive reboots, watchdog timer failures, and system freezes due to resource exhaustion) are quite common. These problems are usually not caused by a single error message, but rather by a series of deteriorating processes such as increased memory pressure, abnormal input / output (I / O) operations, lock contention, task blocking, soft interrupt storms, and kernel deadlocks.

[0003] However, after an operating system crashes abnormally, a large amount of transient state is lost, and engineers often have to rely on scattered logs or incomplete console output for analysis. This information is usually inconsistent in format and semantics, making it difficult to reconstruct the complete chain of how the operating system gradually evolved to the point of crashing, and also making it difficult to perform automated inference and large-scale review. Summary of the Invention

[0004] According to one aspect of an exemplary embodiment of this disclosure, an operating system state root cause inference method is provided, comprising: obtaining structured events with a predefined format in the operating system kernel; determining whether the operating system has crashed abnormally; in response to determining that the operating system has crashed abnormally, obtaining operating system state information by a user-mode service based on structured events obtained during the operating system's runtime cycle before the abnormal crash through a predetermined state machine; and obtaining the root cause of the abnormal crash by the user-mode service based on the operating system state information through the predetermined state machine, wherein the structured events are predefined in the operating system kernel as having the predefined format including at least one of event type, event level, event source, and event attribute.

[0005] The steps for obtaining structured events with a predefined format in the operating system kernel may include at least one of the following: obtaining structured events for state changes of kernel paths; obtaining structured events for state changes of different subsystems of the operating system kernel; obtaining structured events for state changes of user-space services.

[0006] The operating system state root cause inference method may further include: after obtaining a structured event, storing the obtained structured event in a temporary storage medium and / or a persistent storage medium according to a predetermined storage strategy, wherein the step of storing the obtained structured event in a temporary storage medium and / or a persistent storage medium according to the predetermined storage strategy may include: directly storing the pre-specified structured event in a persistent storage medium; and performing at least one of suppression, rate limiting, or aggregation on the same type of structured events that occur more than a predetermined number of times within a predetermined time period, or on structured events other than the pre-specified structured events, and storing them in a temporary storage medium and / or a persistent storage medium.

[0007] The operating system state root cause inference method may further include: in response to normal operation of the operating system, the user-space service obtains the operating system state information sequence based on structured events obtained through real-time subscription by the predetermined state machine.

[0008] The operating system state root cause inference method may further include: determining whether the operating system is in a risky state based on the state information sequence of the operating system through the predetermined state machine; in response to the operating system being in a risky state, performing at least one of the following: notifying at least one of the user, administrator, and server of the risky state; performing predetermined additional diagnostic actions for services related to the risky state; and restoring services related to the risky state.

[0009] The steps for a user-space service to obtain operating system state information based on structured events during the operating system's runtime cycle prior to an abnormal shutdown via a predetermined state machine may include: performing integrity verification on the obtained structured events; sorting the verified structured events in chronological order; and obtaining a sequence of operating system state information based on the sorted structured events via the predetermined state machine.

[0010] The operating system state root cause inference method may further include: notifying at least one of the user, administrator, and server of the abnormal shutdown root cause.

[0011] The root cause of an abnormal outage may include at least one of the following: the most likely root cause, candidate secondary root causes, path of state deterioration, supporting evidence, confidence level of the most likely root cause, and recommended remedial measures. In response to the confidence level of the most likely root cause being not less than a predetermined value, the root cause of the abnormal outage is notified to at least one of the user, the administrator, and the server.

[0012] The operating system can be Linux.

[0013] According to another aspect of an exemplary embodiment of this disclosure, an operating system state root cause inference apparatus is provided, comprising: an event acquisition module configured to obtain structured events with a predefined format in the operating system kernel; and a state information acquisition module configured to determine whether the operating system has crashed abnormally, wherein, in response to determining that the operating system has crashed abnormally, a user-mode service obtains the state information of the operating system based on the structured events obtained during the operating system's runtime cycle before the abnormal crash through a predetermined state machine, and obtains the root cause of the abnormal crash based on the state information of the operating system, wherein the structured events are predefined in the operating system kernel to have the predefined format having at least one of event type, event level, event source, and event attribute.

[0014] The event acquisition module can also be configured to obtain structured events in the operating system kernel through at least one of the following: obtaining structured events for state changes of kernel paths; obtaining structured events for state changes of different subsystems of the operating system kernel; obtaining structured events for state changes of user-space services.

[0015] The operating system state root cause inference device may further include: an event storage module, configured to store the obtained structured event in a temporary storage medium and / or a persistent storage medium according to a predetermined storage strategy after obtaining the structured event, wherein the event storage module may also be configured to store the obtained structured event according to the following predetermined storage strategy: for a pre-specified predetermined structured event, directly store it in a persistent storage medium; for structured events of the same type that occur more than a predetermined number of times within a predetermined time period, or for structured events other than the pre-specified structured events, perform at least one of suppression, rate limiting, or aggregation and store them in a temporary storage medium and / or a persistent storage medium.

[0016] The state information acquisition module can also be configured to perform the following operations: in response to normal operation of the operating system, the user-space service obtains the state information sequence of the operating system through the predetermined state machine based on structured events obtained through real-time subscription.

[0017] The status information acquisition module can also be configured to perform the following operations: determine whether the operating system is in a risky state based on the operating system's status information sequence through the predetermined state machine; in response to the operating system being in a risky state, perform at least one of the following: notify at least one of the users, administrators, and servers of the risky state; perform predetermined additional diagnostic actions for services related to the risky state; and restore services related to the risky state.

[0018] The state information acquisition module can also be configured to perform the following operations: perform integrity verification on the acquired structured events; sort the verified structured events in chronological order; and acquire the operating system's state information sequence based on the sorted structured events using the predetermined state machine.

[0019] The status information acquisition module can also be configured to notify at least one of the users, administrators, and servers of the root cause of the abnormal shutdown.

[0020] The root cause of an abnormal shutdown may include at least one of the following: most likely root cause, candidate secondary root cause, state deterioration path, supporting evidence, confidence level of the most likely root cause, and recommended remedial measures. In response to the confidence level of the most likely root cause being not less than a predetermined value, the state information acquisition module may also be configured to notify at least one of the users, administrators, and servers of the root cause of the abnormal shutdown.

[0021] The operating system can be Linux.

[0022] According to another aspect of an exemplary embodiment of the present disclosure, an electronic device is provided, including: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform an operating system state root cause inference method according to the present disclosure.

[0023] According to another aspect of an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by at least one processor, causes the at least one processor to perform an operating system state root cause inference method according to the present disclosure.

[0024] According to an example embodiment of the present disclosure, a computer program product is provided, including computer instructions, characterized in that, when the computer instructions are executed by at least one processor, they implement an operating system state root cause inference method according to the present disclosure.

[0025] According to the example embodiments of this disclosure, by uniformly defining structured events in the kernel, different types of state changes (such as state changes across subsystems) can be incorporated into the same event stream expression, significantly improving the structured nature and machine resolvability of operating system abnormal shutdown analysis; by using an inference method based on non-single events, different fault types such as resource exhaustion, deadlock, driver anomalies, device failures, file system failures, and critical service failures can be distinguished more accurately; by defining structured events in the kernel and obtaining root causes through user-space services in a layered design, both kernel invasiveness and maintenance costs can be reduced, while the feasibility of rule iteration and product deployment can be improved; by using a recovery mechanism based on structured events within the operating system's runtime cycle before the abnormal shutdown, the sequence of key state changes before the shutdown can be obtained even when the operating system has lost its complete context, enabling automated inference and large-scale review. Attached Figure Description

[0026] The above and other objects and features of the present invention will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart illustrating an operating system state root cause inference method according to an example embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating an operating system state root cause inference method according to an example embodiment of the present disclosure; Figure 3 This is a timing diagram illustrating an operating system state root cause inference method according to an example embodiment of the present disclosure; Figure 4 This is a block diagram illustrating an operating system state root cause inference apparatus according to an example embodiment of the present disclosure; Figure 5 This is a schematic diagram illustrating an operating system state root cause inference apparatus according to an example embodiment of the present disclosure. Detailed Implementation

[0027] In the following description, various exemplary embodiments of the present disclosure are illustrated with reference to the accompanying drawings, wherein the same reference numerals are used to denote the same or similar elements, features, and structures. However, it is not intended to limit the present disclosure to the specific embodiments described herein, and it is intended that the present disclosure cover all modifications, equivalents, and / or substitutions of the present disclosure, provided they fall within the scope of the appended claims and their equivalents. The terms and words used in the following description and claims are not limited to their dictionary meanings, but are used only to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various exemplary embodiments of the present disclosure is for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the appended claims and their equivalents.

[0028] It should be understood that, unless the context clearly indicates otherwise, the singular form includes the plural form. The terms “comprising,” “including,” and “having” as used herein indicate the presence of a function, operation, or element of this disclosure, but do not exclude other functions, operations, or elements.

[0029] For example, the expression “A or B”, “A and / or B”, “at least one of A and B” or “at least one of A or B” may include A, B, and both A and B. For example, the expression “A or B” or “at least one of A and B” may indicate (1) A alone, (2) B alone, or (3) both A and B.

[0030] In the various exemplary embodiments of this disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to, or being "coupled" or "connected" to, another component (e.g., a second component), the component may be directly connected to, or may be connected via, another component (e.g., a third component). In contrast, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to, or being directly coupled to or directly connected to, another component (e.g., a second component), there is no other component (e.g., a third component) between the component and the other component.

[0031] The expression “configured as” used in describing the various exemplary embodiments of this disclosure may be used interchangeably, for example, with expressions such as “suitable for,” “capable of,” “designed to,” “suitable for,” “manufactured as,” and “capable.” The term “configured as” may not necessarily indicate that the hardware is “specifically designed for.” Rather, in some cases, the expression “a device configured as…” may indicate that the device and another device or part are “capable of….” For example, the expression “a processor configured to perform A, B, and C” may indicate a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit CPU or application processor (AP)) for performing the respective operations by executing at least one software program stored in a memory device.

[0032] The terminology used herein is intended to describe certain exemplary embodiments of this disclosure, but is not intended to limit the scope of other exemplary embodiments. Unless otherwise stated herein, all terms used herein (including technical or scientific terms) may have the same meaning as commonly understood by one of ordinary skill in the art. Generally, terms defined in dictionaries should be considered to have the same meaning as in the context of the relevant field, and should not be interpreted differently or as having an overly formal meaning unless expressly defined herein. In no event should the terminology defined in this disclosure be construed as excluding the exemplary embodiments of this disclosure.

[0033] Figure 1 This is a flowchart illustrating an operating system state root cause inference method according to an example embodiment of the present disclosure. According to the example embodiment of the present disclosure, the operating system may include, but is not limited to, Linux.

[0034] like Figure 1 As shown, in operation S110, structured events with a predefined format can be obtained in the operating system kernel. According to an example embodiment of this disclosure, the step of obtaining structured events with a predefined format in the operating system kernel may include at least one of the following: obtaining structured events for state changes of kernel paths; obtaining structured events for state changes of different subsystems of the operating system kernel (such as scheduling subsystem, memory subsystem, resource management subsystem, lock detection subsystem, file system subsystem, block layer subsystem, driver framework subsystem, power management subsystem, security subsystem, device subsystem, etc.), that is, a unified event encapsulation interface can be added to different subsystems; obtaining structured events for state changes of user-space services. In addition, structured events can also be obtained by translating events already collected by the operating system (such as events collected through tracepoints, notifiers, Linux Security Modules (LSM) hooks, exception callbacks, or work queue state changes); furthermore, for critical user-space services, the service lifecycle and crash state can be mapped to unified structured events through the kernel-user-space coordination interface.

[0035] Furthermore, according to example embodiments of this disclosure, structured events can be predefined in the operating system kernel as having a predefined format that includes at least one of event type, event level, event source, and event attributes. For illustrative purposes only and not as a limitation, event types may include, but are not limited to: 1. Kernel sleep / wake events (such as suspend, resume, etc.); 2. Kernel exception events (such as kernel panic, kernel diagnostic information (oops), kernel bug, kernel error, warning events, etc.); 3. Resource stress events, such as memory stress events, stress level transition events, processor stress events, I / O stress events, kernel idle state events; 4. Application crash events, service startup events, service exit events, service blocking events, service restart failure events; 5. Device error events, file system error events, I / O error events; 6. Kernel deadlock events (such as softlockup, hardlockup, long-running blocking detection (hungtask), read-copy update (RCU) blocking, interrupt storm, etc.); 7. Kernel invasive pollution events (such as security software intrusion, hooks from third-party kernel modules, etc.). Event levels may include, but are not limited to, critical errors, serious errors, errors, warnings, and general. Event sources may include, but are not limited to, event source subsystems (such as scheduling subsystems, memory subsystems, resource management subsystems, lock detection subsystems, file system subsystems, block layer subsystems, driver framework subsystems, power management subsystems, security subsystems, and device subsystems). Event attributes may include, but are not limited to, target object identifiers (such as device ID, process ID, service name, or file system instance identifier), critical status values ​​(such as stress level, error code, and suspension duration), and associated event identifiers or associated session identifiers. Furthermore, the above format may also include, but is not limited to, a unique event identifier, a timestamp (when the event occurred), a processor identifier, a process identifier, a thread identifier, and an interrupt context identifier.

[0036] Furthermore, according to the example embodiments of this disclosure, after operation S110, the obtained structured events can be stored in temporary storage media and / or persistent storage media according to a predetermined storage strategy (such as, but not limited to, write strategy, overwrite strategy, flush strategy, and retention strategy under abnormal conditions). More specifically, the predetermined storage strategy may include: directly storing pre-specified pre-specified structured events in persistent storage media (flushing strategy); and suppressing, rate limiting, or aggregating at least one of the following for structured events of the same type that occur more than a predetermined number of times within a predetermined time period, or for structured events other than the pre-specified structured events, and storing them in temporary storage media and / or persistent storage media (write strategy). Furthermore, structured events can be stored in temporary storage media and / or persistent storage media in chronological order, and structured events recorded in temporary storage media can be written to persistent storage media according to their priority and / or periodically written to persistent storage media. According to example embodiments of this disclosure, persistent storage media can be circular buffers, flash reserved pages, persistent storage (pstore, which can be used for emergency saving in case of severe anomalies (such as kernel crashes) via the pstore mechanism), ramoops reserved partitions, non-volatile random access memory (NVRAM), etc., but the present invention is not limited to these. Moreover, those skilled in the art can select write strategies, overwrite strategies, flush strategies, and retention strategies under abnormal conditions according to actual needs, which will not be elaborated here.

[0037] By operating S120, it can be determined whether the operating system has crashed abnormally.

[0038] In operation S130, in response to determining that the operating system has abnormally stopped, the user-space service can obtain the operating system's state information through a predetermined state machine based on structured events obtained during the operating system's runtime cycle before the abnormal stop. According to example embodiments of this disclosure, the state information may include, but is not limited to, normal operation state, mild abnormality symptom state, resource pressure accumulation state, subsystem abnormality propagation state, fault approaching state, final state before abnormal stop, and root cause confirmation state.

[0039] According to an example embodiment of this disclosure, the operating system state root cause inference method according to an example embodiment of this disclosure may further include: in response to normal operation of the operating system, obtaining a sequence of operating system state information based on structured events obtained through real-time subscription via a predetermined state machine. Furthermore, it may also be possible to determine whether the operating system is in a risky state based on the operating system state information sequence via a predetermined state machine; in response to the operating system being in a risky state, performing at least one of the following: notifying at least one of the user, administrator, and server of the risky state; performing predetermined additional diagnostic actions for services related to the risky state; and restoring services related to the risky state. According to an example embodiment of this disclosure, for example, when the operating system has not yet stopped but is already in a risky state, such as when there are consecutive memory pressure escalation events, repeated crashes of critical services, long-term blocking detection events, or device anomaly accumulation events, it can be determined that the operating system is in a risky state, and the above-mentioned subsequent processing can be performed. In addition to real-time subscription, subscription can also be implemented through an event bus, message channel, or a kernel-user space shared buffer mechanism.

[0040] Furthermore, according to the example embodiments of this disclosure, the step of obtaining the operating system's state information by a user-space service or diagnostic service through a predetermined state machine based on structured events obtained during the operating system's runtime cycle before the abnormal shutdown may include: performing integrity verification on the obtained structured events (in addition, operations such as sequence repair and session boundary identification, power-down interrupt identification, partial writing and overwrite boundary can be performed on the structured events, which will not be elaborated here); sorting the verified structured events in chronological order; and obtaining the operating system's state information sequence based on the sorted structured events through a predetermined state machine. More specifically, user-mode services or diagnostic services can locate the event storage area in the temporary and / or persistent storage media corresponding to structured events during the operating system's runtime cycle before the abnormal shutdown and read the structured events. Furthermore, user-mode services or diagnostic services can identify integrity and / or clean structured events by verifying at least one of the event headers, check fields, timestamp continuity, and session boundary identifiers recorded in the structured events. Additionally, user-mode services or diagnostic services can sort the verified structured events by merging data from different processors, different event channels, and different temporary and / or persistent storage media according to a global timeline to construct a sequence of structured events before the abnormal shutdown. Furthermore, user-mode services or diagnostic services can input the constructed sequence of structured events into a predetermined state machine or inference rule engine to perform state replay and transition judgments to obtain a sequence of operating system state information.

[0041] In operation S140, the user-space service can obtain the root cause of the abnormal shutdown based on the state information sequence of the operating system through a predetermined state machine; and notify at least one of the user, administrator, and server of the root cause of the abnormal shutdown. Furthermore, the root cause of the abnormal shutdown obtained by the user-space service or diagnostic service may include at least one of the following: most likely root cause, candidate secondary root cause, state deterioration path, supporting evidence, confidence level of the most likely root cause, and suggested remedial measures. In this case, in response to the confidence level of the most likely root cause being not less than a predetermined value, the root cause of the abnormal shutdown can be notified to at least one of the user, administrator, and server; in response to the confidence level of the most likely root cause being less than the predetermined value, candidate secondary root causes and / or supplementary evidence can be notified to at least one of the user, administrator, and server. Moreover, the state machine is not limited to a single implementation form and can be a finite state machine, hierarchical state machine, rule tree, causal graph, Bayesian network constraint graph, or a hybrid inference engine based on a combination of rules and scoring; those skilled in the art can choose according to actual needs.

[0042] By using the same event model (such as a predefined state machine) to both retrospectively analyze abnormal system shutdowns and conduct online diagnostics during normal system operation, continuous diagnostic capabilities before, during, and after a fault can be formed, enabling early detection of system anomalies.

[0043] Based on the exemplary embodiments of this disclosure, which are merely examples and not limitations, for example, when a structured event sequence sequentially includes events such as memory pressure increase, frequent direct reclamation, critical service blocking, and user forced shutdown, the state machine can determine that the operating system has experienced a service failure and scheduling deterioration process caused by resource pressure, and prioritize the most likely root cause as abnormal shutdown caused by resource exhaustion; for example, when a structured event sequence sequentially includes events such as kernel memory access to kernel diagnostic information, kernel deadlock, kernel crash, or user forced shutdown, the state machine can determine that kernel memory access to kernel diagnostic information caused the kernel state to lose consistency, thereby triggering kernel deadlock and ultimately leading to abnormal shutdown; for example, when a structured event sequence sequentially includes system sleep events and system wake-up events, and an application crash, kernel crash, or device malfunction occurs immediately after system wake-up, the state machine can determine that the operating system's sleep-wake process caused memory malfunction or device malfunction, ultimately leading to abnormal shutdown; for example, if the final event is a system sleep event, but the subsequent system wake-up event is missing, the state machine can determine that the operating system failed to wake up successfully after sleep, resulting in abnormal shutdown.

[0044] According to the example embodiments of this disclosure, the state machine obtains the root cause of abnormal shutdown not only based on the last error, but also focuses on the temporal combination relationship and state transition relationship between structured events. Preferably, the state machine can obtain the root cause of abnormal shutdown through at least the following dimensions of structured events: the sequence of events; the time interval between events; the severity of events; the subsystem from which the events originate; the strength of the correlation between events; whether the expected subsequent events occur after a high-risk event is triggered; and whether a certain type of abnormal event persists, spreads, or cascades. Furthermore, the preferred inference rules for the state machine to obtain the root cause of abnormal shutdowns include, but are not limited to: if a device malfunction occurs before suspension, and a driver error event, bus timeout event, or service disconnection event occurs immediately after recovery, the most likely root cause can be identified as a suspension / recovery related fault; if soft deadlock events, read / copy / update blocking events, or long-term blocking detection events occur consecutively, accompanied by a watchdog reset event, the most likely root cause can be identified as a deadlock or scheduling stall fault; if a memory pressure event, pressure level escalation event, out-of-memory (OOM) event, critical process core dump, or kill event occurs first, the most likely root cause can be identified as a memory resource link malfunction; if an interrupt request storm occurs... For events such as Request (IRQ) Storm, abnormal processor usage, hard deadlock, or prolonged interrupt shutdown, the most likely root cause can be identified as an interrupt storm or a low-level driver anomaly. If a prolonged blocking event, security interception event, or anomaly hook registration event occurs, followed by a prolonged blocking of critical services or system instability, security intervention or abnormal module injection can be identified as a candidate secondary root cause. Furthermore, to improve interpretability, it is preferable to output the abnormal shutdown root cause, including the root cause conclusion, the state deterioration path, and the confidence level of the most likely root cause. For example, the most likely root cause could be a suspected storage device anomaly leading to a file system error and triggering a system restart, with the state deterioration path being a device error event -> input / output error event -> file system error event -> kernel warning event -> watchdog restart event, and the confidence level of the most likely root cause being, for example, 0.86. However, this invention is not limited to this.

[0045] According to the example embodiments of this disclosure, the risk status or abnormal downtime root cause obtained above can be output as a local report, or as a structured audit log, a remote operation and maintenance platform event, an alarm notification, or a visual analysis result.

[0046] According to the example embodiments of this disclosure, preferably, the event types may include core events such as suspend, kernel diagnostic information (oops), pressure, and coredump of critical processes, and may also be expanded to include more subsystem events; the temporary storage medium may be approximately 4MB, or may be adjusted according to platform resource conditions; the predetermined storage strategy of the persistent storage medium may preferably enable priority storage for critical events (such as kernel panic, kernel diagnostic information (oops), hard lockup, etc.); the merging window of the predetermined storage strategy for structured events of the same type that occur more than a predetermined number of times within a predetermined time period may preferably be approximately 1000ms, in order to suppress short-term recurring events; the online diagnostic cycle may preferably be approximately 1s; the number of event records retained may preferably be the most recent 8192 critical events; the risk alarm threshold under normal operating system conditions may preferably be approximately 70 points; the number of root cause outputs may preferably be one most likely root cause and several candidate secondary root causes. It should be understood that all the above parameters are only examples, and those skilled in the art can adjust them according to actual conditions through kernel configuration, startup parameters, daemon configuration files, or system management interfaces. Figure 2 This is a schematic diagram illustrating an operating system state root cause inference method according to an example embodiment of the present disclosure. Figure 2 As shown above, refer to the above. Figure 1 The system defines structured events, records and stores them, and determines whether the operating system has experienced an abnormal shutdown. When the operating system has not experienced an abnormal shutdown, it can subscribe to structured events in real time. Based on the structured events obtained from the real-time subscription, it can determine whether the operating system is in a risky state. If it is in a risky state, it can notify the system of the risk status, perform predetermined additional diagnostic actions, and / or perform recovery. The operation then returns to the stored structured events. Furthermore, when the operating system experiences an abnormal shutdown, it can obtain structured events from the operating system's runtime cycle before the abnormal shutdown. These structured events are then verified, sorted, and / or cleaned. A state machine is used to obtain the root cause of the abnormal shutdown, and the confidence level of the most probable root cause is determined. When the confidence level of the most probable root cause is low, candidate secondary root causes and supplementary evidence can be output, and the results can be recorded and notified. Conversely, when the confidence level of the most probable root cause is high, the most probable root cause, the state deterioration path, supporting evidence, and suggested remedial measures can be output, and the results can be recorded and notified.

[0047] Figure 3 This is a timing diagram illustrating an operating system state root cause inference method according to an example embodiment of the present disclosure.

[0048] like Figure 3As shown, structured events (defined in the operating kernel) are generated based on structured event sources (such as kernel path state changes, different subsystems of the operating system kernel, etc.). These structured events are written to temporary storage media in chronological order and to persistent storage media according to a predetermined storage strategy. If the operating system experiences an abnormal shutdown and / or reset and restarts, the user-space service can read the structured events obtained during the operating system's runtime cycle before the abnormal shutdown and / or reset from the persistent module after the restart. It then performs integrity verification, time sorting, and cleaning operations to obtain a state information sequence. Based on the operating system's state information sequence, a predetermined state machine or rule graph is used to obtain the root cause of the abnormal shutdown (such as performing state backtracking and / or root cause analysis on the state information sequence). The user-space service or a separate output module outputs the root cause of the abnormal shutdown and / or records it to the audit report.

[0049] Figure 4 This is a block diagram illustrating an operating system state root cause inference apparatus according to an example embodiment of the present disclosure. Figure 4 As shown, the operating system state root cause inference apparatus 400 according to an example embodiment of the present disclosure may include an event acquisition module 410 and a state information acquisition module 420.

[0050] The event acquisition module 410 can be configured to obtain structured events with a predefined format in the operating system kernel. Furthermore, the event acquisition module 410 can also be configured to obtain structured events in the operating system kernel through at least one of the following: obtaining structured events based on state changes of kernel paths; obtaining structured events based on state changes of different subsystems of the operating system kernel; obtaining structured events based on state changes of user-space services. According to an example embodiment of this disclosure, structured events can be predefined in the operating system kernel with the predefined format having at least one of event type, event level, event source, and event attributes. Here, a separate module (such as an event definition module) can be set up to predefine structured events with a predefined format in the operating system kernel, or existing modules can be used to perform the above operations, which will not be elaborated further here.

[0051] The status information acquisition module 420 can be configured to determine whether the operating system has crashed abnormally. In response to determining that the operating system has crashed abnormally, the user-mode service obtains the status information of the operating system through a predetermined state machine based on the structured events obtained during the operating system's runtime cycle before the abnormal crash, and obtains the root cause of the abnormal crash based on the status information of the operating system.

[0052] Furthermore, the state information acquisition module 420 can also be configured to perform the following operations: in response to normal operation of the operating system, the user-space service obtains a sequence of operating system state information based on structured events obtained through real-time subscription via a predetermined state machine. Additionally, the state information acquisition module is also configured to perform the following operations: determine whether the operating system is in a risky state based on the operating system state information sequence via the predetermined state machine; in response to the operating system being in a risky state, perform at least one of the following: notify at least one of the users, administrators, and servers of the risky state; perform predetermined additional diagnostic actions for services related to the risky state; and restore services related to the risky state.

[0053] Furthermore, the state information acquisition module 420 can also be configured to perform the following operations: perform integrity verification on the acquired structured events; sort the verified structured events in chronological order; and acquire the operating system's state information sequence based on the sorted structured events using a predetermined state machine. Additionally, the state information acquisition module 420 can also be configured to notify at least one of the following operations: notify the user, administrator, and server of the root cause of the abnormal shutdown. Furthermore, the root cause of the abnormal shutdown may include at least one of the following: most likely root cause, candidate secondary root cause, state deterioration path, supporting evidence, confidence level of the most likely root cause, and recommended remedial measures. In response to the confidence level of the most likely root cause being not less than a predetermined value, the state information acquisition module 420 can also be configured to notify at least one of the user, administrator, and server of the root cause of the abnormal shutdown. It should be understood that the various operations performed by the state information acquisition module 420 can be completed by a single module, or the state information acquisition module 420 can be divided into multiple modules to perform different operations. Those skilled in the art can choose according to actual needs.

[0054] Furthermore, according to an example embodiment of this disclosure, the operating system state root cause inference apparatus 400 may further include: an event storage module configured to, after obtaining a structured event, store the obtained structured event in a temporary storage medium and / or a persistent storage medium according to a predetermined storage strategy. More specifically, the event storage module may also be configured to store the obtained structured events according to the following predetermined storage strategy: for pre-specified structured events, directly store them in a persistent storage medium; for structured events of the same type that occur more than a predetermined number of times within a predetermined time period, or for structured events other than the pre-specified structured events, perform at least one of suppression, rate limiting, or aggregation and store them in a temporary storage medium and / or a persistent storage medium.

[0055] Figure 5 This is a schematic diagram illustrating an operating system state root cause inference apparatus according to an example embodiment of the present disclosure.

[0056] like Figure 5 As shown, the operating system state root cause inference apparatus according to an example embodiment of this disclosure may include an event definition module 510, an event acquisition module 520, an event storage module 530, an abnormal shutdown event recovery module 540, an abnormal shutdown root cause inference module 550, a normal operation root cause inference module 560, and a result output and auditing module 570. The event definition module 510, event acquisition module 520, and event storage module 530 belong to the kernel-side event recording link, while the abnormal shutdown event recovery module 540, abnormal shutdown root cause inference module 550, normal operation root cause inference module 560, and result output and auditing module 570 belong to the user-mode service diagnostic link. It should be understood that the event definition module 510, event acquisition module 520, and event storage module 530 can execute the above-mentioned... Figure 4 The operations described above will not be repeated here; the abnormal shutdown event recovery module 540, the abnormal shutdown root cause inference module 550, the normal operation root cause inference module 560, and the result output and auditing module 570 can constitute the above-mentioned reference. Figure 4 The state information acquisition module 420 is described above, and the operations of the state information acquisition module 420 are executed respectively. For example, the event recovery module can be used to obtain the operating system running cycle before the abnormal shutdown by the user-space service after the system abnormal shutdown and restart, and perform operations such as integrity verification, time sorting, sequence repair and session boundary identification to obtain the state information sequence. The abnormal shutdown root cause inference module can be used to obtain the abnormal shutdown root cause based on the operating system state information sequence through a predetermined state machine or rule graph (such as performing state backtracking and / or root cause analysis on the state information sequence). The normal operation root cause inference module can be used to determine whether the operating system is in a risk state based on the structured events subscribed to in real time during normal operation of the operating system, such as identifying abnormal precursor states, risk accumulation states and / or fault approach states, and triggering risk state notifications and / or predetermined additional diagnostic actions. The output and audit module can be used to output the abnormal shutdown root cause and / or risk state obtained by the abnormal shutdown root cause inference module and the normal operation root cause inference module, and can be recorded to the local diagnostic report, system audit log or operation and maintenance platform.

[0057] According to an example embodiment of the present disclosure, an electronic device is provided, including a memory and at least one processor, wherein the memory stores computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform an operating system state root cause inference method according to the present disclosure.

[0058] According to an example embodiment of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform an operating system state root cause inference method according to this disclosure. Examples of computer-readable storage media may include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (e.g., multimedia card, secure digital (SD) card, or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as client, host, agent, or server. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0059] According to an example embodiment of the present disclosure, a computer program product is provided, including computer instructions that, when executed by at least one processor, implement an operating system state root cause inference method according to the present disclosure.

[0060] According to the example embodiments of this disclosure, by uniformly defining structured events in the kernel, different types of state changes (such as state changes across subsystems) can be incorporated into the same event stream expression, significantly improving the structured nature and machine resolvability of operating system abnormal shutdown analysis; by using an inference method based on non-single events, different fault types such as resource exhaustion, deadlock, driver anomalies, device failures, file system failures, and critical service failures can be distinguished more accurately; by defining structured events in the kernel and obtaining root causes through user-space services in a layered design, both kernel invasiveness and maintenance costs can be reduced, while the feasibility of rule iteration and product deployment can be improved; by using a recovery mechanism based on structured events within the operating system's runtime cycle before the abnormal shutdown, the sequence of key state changes before the shutdown can be obtained even when the operating system has lost its complete context, enabling automated inference and large-scale review.

[0061] Although this disclosure has been shown and described with reference to specific exemplary embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this disclosure as defined by the claims and their equivalents.

Claims

1. A method for inferring the root cause of operating system states, characterized in that, include: Obtain structured events with a predefined format within the operating system kernel; Determine if the operating system has crashed abnormally; In response to the determination of an abnormal operating system shutdown, the user-space service obtains the operating system's state information through a predefined state machine based on the structured events obtained during the operating system's runtime cycle before the abnormal shutdown. The user-space service obtains the root cause of the abnormal shutdown based on the operating system's state information through the predetermined state machine. Structured events are predefined in the operating system kernel as having a predefined format that includes at least one of event type, event level, event source, and event attributes.

2. The operating system state root cause inference method as described in claim 1, characterized in that, The steps for obtaining structured events with a predefined format in the operating system kernel include at least one of the following: Obtain structured events based on state changes in the kernel path; Structured events are obtained by analyzing the state changes of different subsystems of the operating system kernel. Obtain structured events based on state changes of user-mode services.

3. The operating system state root cause inference method as described in claim 1, characterized in that, Also includes: After obtaining a structured event, the obtained structured event is stored in a temporary storage medium and / or a persistent storage medium according to a predetermined storage strategy. The predetermined storage strategy includes: For pre-specified structured events, they are directly stored in persistent storage media; For structured events of the same type that occur more than a predetermined number of times within a predetermined time period, or for structured events other than the pre-specified structured events, at least one of the following is performed: suppression, rate limiting, or aggregation, and the results are stored in a temporary storage medium and / or a persistent storage medium.

4. The operating system state root cause inference method as described in claim 1, characterized in that, Also includes: In response to normal operation of the operating system, the user-space service obtains the sequence of operating system state information based on structured events obtained through real-time subscription via the predetermined state machine.

5. The operating system state root cause inference method as described in claim 4, characterized in that, Also includes: The predetermined state machine determines whether the operating system is in a risky state based on the operating system's state information sequence. In response to the operating system being in a risky state, perform at least one of the following: notify at least one of the users, administrators, and servers of the risky state; Perform scheduled additional diagnostic actions for services related to risk status; restore services related to risk status.

6. The operating system state root cause inference method as described in claim 1, characterized in that, The steps by which user-space services obtain operating system state information based on structured events during the operating system's runtime cycle prior to the abnormal shutdown via a predefined state machine include: Perform integrity verification on the acquired structured events; The structured events to be validated are sorted in chronological order. The operating system's state information sequence is obtained through the predetermined state machine based on ordered structured events.

7. The operating system state root cause inference method as described in claim 6, characterized in that, Also includes: Notify at least one of the users, administrators, and servers of the root cause of the abnormal outage.

8. The operating system state root cause inference method as described in claim 7, characterized in that, The root cause of an abnormal shutdown includes at least one of the following: the most likely root cause, candidate secondary root causes, path of deterioration, supporting evidence, confidence level of the most likely root cause, and recommended remedial measures. Specifically, in response to a confidence level of not less than a predetermined value for the most probable root cause, at least one of the user, administrator, and server is notified of the abnormal downtime root cause.

9. The operating system state root cause inference method as described in claim 1, characterized in that, The operating system is Linux.

10. An operating system state root cause inference device, characterized in that, include: The event acquisition module is configured to obtain structured events with a predefined format from the operating system kernel; The status information acquisition module is configured to determine whether the operating system has crashed abnormally. In response to the determination that the operating system has crashed abnormally, the user-space service obtains the operating system's status information through a predetermined state machine based on structured events obtained during the operating system's runtime cycle before the abnormal crash, and then obtains the root cause of the abnormal crash based on the operating system's status information. Structured events are predefined in the operating system kernel as having a predefined format that includes at least one of event type, event level, event source, and event attributes.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the operating system state root cause inference method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor performs the operating system state root cause inference method according to any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, they implement the operating system state root cause inference method as described in any one of claims 1 to 9.

Images