Fault root cause positioning and hardware resetting system for cloud edge collaborative architecture

Abstract

The invention relates to the technical field of industrial operation and maintenance, and discloses a cloud edge collaborative architecture-oriented fault root cause positioning and hardware reset system, which comprises a graph construction module for constructing a component association graph containing power supply dependence and reset wave and span; the mapping module is used for converting the record set before and after protective reset into abnormal data residuals of a component domain; the boundary extraction module is used for calculating a data retention fall value of adjacent component domains so as to extract a maximum record abrupt change boundary; the root cause positioning module is used for determining a root cause component domain by combining the distance to the mutation boundary and the fault interpretation degree; the reset selection module is used for selecting a controlled hardware reset instruction with the minimum cost for the abnormal spreading component set; the reset execution module is used for executing domain-by-domain release according to a power supply sequence and under the condition that observable data is read for the first time after extraction record hosting; and the verification write-back module generates a reset validity index. According to the invention, misjudgment of reset by-products is avoided, and accurate positioning and height retention of field clues are realized.

Description

Technical Field

[0001] This invention relates to the field of industrial operation and maintenance technology, and more specifically, to a fault root cause localization and hardware reset system for cloud-edge collaborative architecture. Background Technology

[0002] With the deepening development of cloud-edge collaborative architecture, edge nodes are no longer just simple end-side data acquisition channels, but have evolved into a composite node system with the capabilities of heterogeneous computing resource orchestration, cross-layer collaboration, and offline autonomy. In this system, a single node often integrates multiple independent component domains such as host execution environment, out-of-band management plane, various acceleration buses, and local persistent storage.

[0003] In fault management within cloud-edge collaborative scenarios, when an edge node experiences a host operating system crash, edge agent inactivation, or prolonged application layer disconnection, a protective reset action is often triggered in the out-of-band management plane (such as the baseboard management controller or watchdog). However, the reset semantics of modern hardware exhibit multi-granularity distribution. For example, the host level may be affected by a reboot command, while the out-of-band management plane, persistent storage, or local interconnect bus may remain alive. This asynchronous survival attribute at the component level results in the fault trail within the node being severely fragmented after a protective reset.

[0004] Current root cause analysis solutions mostly rely on joint modeling and analysis of global multi-source time series or log metrics. However, these solutions generally overlook the component-level trace differences caused by multi-granularity resets. After a protective reset, the out-of-band management domain may still retain runtime error traces from before the failure, while the traces in the host execution domain have been erased and regenerated with the initial startup logs after the reset. Meanwhile, due to latency and potential disconnections in cloud-edge links, the cloud side often only obtains a delayed mirror state. This means that the fault traces recollected after a reset no longer correspond to a single time coordinate, but rather evolve into a set of residual mixtures spanning time and space and different reset granularities.

[0005] Existing fault location and recovery systems have significant limitations when faced with this type of altered fault pattern. On the one hand, the fault location logic is prone to misjudging newly generated accompanying anomalies after a reset as the true root cause of the fault, leading to diagnostic errors. On the other hand, when attempting to proceed with the next step of hardware recovery, existing methods often adopt a blind, one-size-fits-all approach of powering off and restarting the entire machine. This coarse reset action not only easily destroys any remaining valid clues in different component domains, but also fails to perform precise granular control and orderly recovery for the local component domains where the fault actually occurred, thus severely restricting the high availability and automated operation and maintenance level of cloud-edge collaborative clusters. Summary of the Invention

[0006] This invention provides a fault root cause localization and hardware reset system for cloud-edge collaborative architecture, which solves the technical problems mentioned in the background art.

[0007] This invention provides a fault root cause localization and hardware reset system for cloud-edge collaborative architectures, comprising: The graph construction module is used to construct a component association graph of the fault node. The component association graph includes power supply dependencies and calculates the independent monitoring level and reset sweep span of each component domain in the component association graph. The mapping module is used to collect fault record sets before and after protective reset, and convert the fault record sets into abnormal data residue and data retention ratio of each component domain. The boundary extraction module is used to calculate the data retention difference value between adjacent component domains based on the independent monitoring level, the reset wave and span and the data retention ratio, so as to extract the maximum record mutation boundary and mutation amplitude. The root cause localization module is used to determine the set of abnormally exposed component domains based on the fault record set, calculate the fault explanatory power of each component domain to the set of abnormally exposed component domains, and determine the root cause component domain from each component domain by combining the node distance from each component domain to the boundary of the maximum record mutation. The reset selection module is used to construct the abnormal propagation component set of the root cause component domain, and select a controlled hardware reset instruction for the abnormal propagation component set based on the hardware restart cost evaluation function. The reset execution module is used to extract the fault record set for local management, determine the hardware recovery order according to the order of the power supply dependencies, and execute the domain-by-domain release of the controlled hardware reset instruction based on the condition that observable data is read for the first time. The verification write-back module is used to calculate the drop convergence ratio of the maximum record mutation boundary and the residual abnormality in the abnormal propagation component set after the release is executed, generate a reset validity index and store it.

[0008] The beneficial effects of this invention include: by mapping the component associations, power supply dependencies, and reset spillover spans of nodes in a graphical manner, and quantifying the residual abnormal data before and after protective reset, the system accurately captures the abrupt boundary of data retention gap caused by multi-granularity reset and edge autonomy; this scheme breaks through the limitation of traditional methods that easily misjudge reset-related byproducts as the real root cause, and achieves accurate positioning of the true anomaly origin; at the same time, based on the anomaly propagation component set and cost assessment, the system can select and issue a unique controlled hardware reset command, and after extracting records for local hosting, release them domain by domain according to the power supply sequence and the condition of the first read observable data, thereby avoiding blindly shutting down the entire machine and preserving on-site clues to the greatest extent, while eliminating the abnormal linkage propagation between components, and significantly improving the fault diagnosis accuracy and automated self-healing capability of the cloud-edge collaborative cluster. Attached Figure Description

[0009] Figure 1 This is a block diagram of the fault root cause localization and hardware reset system for cloud-edge collaborative architecture of the present invention. Detailed Implementation

[0010] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0011] like Figure 1 As shown, the fault root cause localization and hardware reset system for cloud-edge collaborative architecture includes: The graph construction module is used to construct a component association graph of the fault node. The component association graph includes power supply dependencies and calculates the independent monitoring level and reset sweep span of each component domain in the component association graph. The mapping module is used to collect fault record sets before and after protective reset, and convert the fault record sets into abnormal data residue and data retention ratio of each component domain. The boundary extraction module is used to calculate the data retention difference value between adjacent component domains based on the independent monitoring level, the reset wave and span and the data retention ratio, so as to extract the maximum record mutation boundary and mutation amplitude. The root cause localization module is used to determine the set of abnormally exposed component domains based on the fault record set, calculate the fault explanatory power of each component domain to the set of abnormally exposed component domains, and determine the root cause component domain from each component domain by combining the node distance from each component domain to the boundary of the maximum record mutation. The reset selection module is used to construct the abnormal propagation component set of the root cause component domain, and select a controlled hardware reset instruction for the abnormal propagation component set based on the hardware restart cost evaluation function. The reset execution module is used to extract the fault record set for local management, determine the hardware recovery order according to the order of the power supply dependencies, and execute the domain-by-domain release of the controlled hardware reset instruction based on the condition that observable data is read for the first time. The verification write-back module is used to calculate the drop convergence ratio of the maximum record mutation boundary and the residual abnormality in the abnormal propagation component set after the release is executed, generate a reset validity index and store it.

[0012] Preferably, a component association graph of the fault node is constructed, the component association graph including power supply dependencies, and the independent monitoring level and reset sweep span of each component domain within the component association graph are calculated, including: Extract the component list and connection topology of the faulty node, and generate the component association graph: ; Construct a reset reachability matrix based on the reset propagation relationship: ; Calculate the independent monitoring level for each component domain within the component association graph: ; Calculate the reset sweep span for each component domain within the component association diagram: ; in, A collection of component domains. For the first in the set Each of the component domains; For data dependencies, To control dependencies, For the reset propagation relationship, This refers to the power supply dependency; Let be the adjacency matrix of the reset propagation relationship. The reset reachability matrix represents whether any of the component domains can be reached by reset propagation. For the component domain The aforementioned independent monitoring level; For out-of-band management observation surfaces, For the edge of local autonomous observation surface, In order to monitor the observation surface, Characterizing the monitoring observation surface in the event of a failure. Is the component domain still observable? ; For the component domain The reset wave and span; To initiate a protective reset of the component domain, To initiate a component domain from the protective reset along the reset propagation relationship. To the component domain The set of reset paths, For the reset path, The length of the reset path.

[0013] A faulty node is a cloud-edge collaborative node entity that experiences an anomaly and triggers a protective reset process. It can be identified through cloud-side node orchestration information, out-of-band management controller asset information, edge agent registration information, and current alarm location results.

[0014] The component inventory is a list of all hardware units, software execution units, management units, and storage units within the faulty node that can participate in dependency analysis, reset propagation, and data retention analysis. It can be obtained through device enumeration results, firmware description information, management controller asset lists, operating system hardware discovery results, and bus scan results.

[0015] Connection topology is a description of the connection structure between components within a faulty node at the data, control, reset, and power supply levels. It can be obtained through board-level connection relationships, device trees, firmware tables, bus enumeration results, switch chip cascading relationships, and management interface mapping results.

[0016] A component dependency graph is a graph structure that unifies the component domains and their data dependencies, control dependencies, reset propagation dependencies, and power dependencies.

[0017] The component domain set is the collection of all component domains obtained after the component division is completed.

[0018] A component domain is the smallest analytical unit within a faulty node that is relatively independent in terms of power supply, reset, control, or observation.

[0019] Data dependencies are relationships formed when the output data of one component domain is received, consumed, or used as a runtime condition by another component domain. These dependencies can be extracted through business link configurations, bus data channels, driver bindings, middleware connections, and upstream / downstream call traces in logs.

[0020] Control dependencies are relationships formed when the startup, shutdown, configuration, activation, or state transition of a component domain is controlled by another component domain. They can be extracted through controller configuration, firmware processes, management command paths, service orchestration rules, and state machine definitions.

[0021] Reset propagation relationships are the propagation relationships formed when a component domain triggers or experiences a reset, and the reset effect continues to pass to other component domains. These relationships can be extracted from reset controller mappings, management interface capability descriptions, platform reset action specifications, bus hot reset coverage relationships, and board-level reset connection information.

[0022] Power supply dependencies are relationships where one component domain must first have power or be released from its hold state before it can enter an operational or observable state. These dependencies can be extracted from power tree design documentation, timing power-on tables, board-level power supply diagrams, management controller power-on control interfaces, and firmware timing configurations.

[0023] The reset propagation relation adjacency matrix is ​​a matrix representation of the reset propagation relation after transforming it according to the component domain order.

[0024] The reset reachability matrix is ​​a matrix used to describe the reachability of a protective reset from the initiating component domain to the target component domain.

[0025] Independent monitoring levels are the number of monitoring observation surfaces that can still be independently observed in the target component domain after a failure occurs.

[0026] A monitoring observation surface is an independent observation channel that can continue to provide component domain status, records, or operational traces after a failure occurs.

[0027] The out-of-band management observation surface is a monitoring channel independently provided by the out-of-band management controller, used to continue acquiring the target component domain status when the host is abnormal.

[0028] The edge local autonomous observation surface is a local autonomous observation channel that can continue to operate when the cloud link is abnormal or the host is locally abnormal.

[0029] The monitoring observable indicator value is used to indicate whether a binary result of the target component domain can still be observed when a certain monitoring observation surface fails.

[0030] Protective reset is a protective hardware reset action triggered by the management plane when a node experiences host disconnection, agent inactivation, or operational abnormalities. It can be obtained through out-of-band management controller event logs, watchdog records, reset reason registers, and cloud-edge operation and maintenance alarm logs.

[0031] The initiating component domain is the first component domain to be confirmed to have triggered a reset action or issued a reset command in this protective reset. It can be determined through the reset event timeline, watchdog timeout records, reset reason codes, and the sequence of management controller events.

[0032] The reset path set is the set of all reset propagation paths that the initiating component domain may traverse when it propagates along the reset propagation relationship to the target component domain.

[0033] A reset path is a specific propagation path that a reset effect takes from the initiating component domain to the target component domain.

[0034] Reset propagation span is an indicator of the length of the shortest reset propagation path between the initiating component domain and the target component domain.

[0035] In detail, because traditional fault location diagrams typically only describe service call relationships or device connection relationships and cannot simultaneously explain fault propagation, reset coverage, and recovery order, this application incorporates data dependencies, control dependencies, reset propagation relationships, and power supply dependencies into the component association diagram. With this structure, the same diagram can be used to locate the direction of anomaly propagation, determine which component domains are affected by a reset, and make subsequent release decisions during recovery, thus unifying diagnosis and reset control into a single structural model.

[0036] In detail, because simply knowing the reset propagation edge is not enough to directly determine whether reset propagation reachability exists between any two component domains, this application further transforms the reset propagation relationship into a reset reachability matrix. The purpose of the reset reachability matrix is ​​to expand the multi-hop reset propagation results to any pair of component domains at once, enabling the system to quickly determine whether a component domain will be affected by this protective reset.

[0037] In detail, because different component domains in cloud-edge collaborative nodes do not simultaneously lose all observation capabilities after a failure, some component domains can still be observed by out-of-band management planes, and some component domains can still be observed by edge local autonomous planes. Therefore, this application does not treat observability as a single presence or absence state, but defines the number of observation surfaces that can still independently observe the component domain as the independent monitoring level. This can distinguish between high-autonomous traceable domains and low-autonomous easily cleared domains.

[0038] In detail, because this application addresses cloud-edge collaborative nodes, the key differences after a failure lie not only in whether the host domain is still running, but also in whether the out-of-band management controller and the edge local autonomous logic can still independently generate logs and status. Therefore, this application explicitly limits the monitoring observation surface to the out-of-band management observation surface and the edge local autonomous observation surface. Thus, the source of observability directly corresponds to the actual structure in the cloud-edge architecture, rather than relying on an abstract single monitoring score.

[0039] In detail, because the ease with which a component domain is cleared by a protective reset is directly related to its proximity to the reset source in the reset propagation chain, this application defines the shortest reset path length from the initiating component domain to the target component domain as the reset sweep span. The smaller this value, the closer the target component domain is to the core of the reset action, and the more likely it is to be directly cleared or rebuilt during a protective reset. The larger this value, the more likely the component domain is to retain pre-fault clues.

[0040] In detail, the component domain partitioning method is as follows: first, the internal units of the node are partitioned at the first level according to power supply independence; then, at the second level, they are partitioned according to reset independence; and finally, at the third level, they are partitioned according to independent observability during faults. If two units are indistinguishable in all three dimensions of power supply control, reset propagation, and fault observation, they are merged into the same component domain; if any one of these dimensions is distinguishable, they are split into different component domains. During implementation, at least the out-of-band management controller domain, host execution domain, local persistent storage domain, and accelerated interconnect domain should be partitioned into independent component domains.

[0041] In detail, the four types of relationships are constructed as follows: data dependencies are established based on the input-output links between data producers and data consumers; control dependencies are established based on the start, stop, configuration distribution, and state switching control links; reset propagation dependencies are established based on reset controller mapping, reset line connection relationships, and platform reset capability specifications; and power supply dependencies are established based on the power tree, power-on timing table, and release sequence. Only when a relationship can be verified by configuration files, hardware description tables, management interface capabilities, or historical action verification results is a corresponding edge allowed to be added to the component association graph.

[0042] In detail, the method for determining observability during a fault is as follows: Within the observation window after a fault is triggered, if a monitoring observation surface can continuously read at least one type of valid state quantity, event record, or heartbeat status of the target component domain, and this result does not depend on the failed host execution link, then it is determined that the monitoring observation surface can still observe the component domain. Valid state quantities include at least one of out-of-band sensor refresh, autonomous agent local status, bus link status, or readable results of storage indexes.

[0043] In detail, the method for determining the initiating component domain is as follows: the earliest reset cause record in the protective reset event timeline is used as the primary basis; the reset target in the management controller event log is used as the secondary basis; and the reset cause register or watchdog timeout source is used as the tertiary basis. When multiple sources exist simultaneously, the unique initiating component domain is determined in the order of priority of primary basis, verification of secondary basis, and supplementation of tertiary basis.

[0044] In detail, the reset span in unreachable scenarios is determined as follows: If there is no reset propagation path from the initiating component domain to the target component domain, the component domain is marked as an unreachable domain for this reset, and the reset span is recorded as a unified unreachable identifier value. This unreachable identifier value should be considered greater than any reachable path length when sorting, to indicate that the component domain is not directly affected by this protective reset.

[0045] In detail, the specific carriers of the component list, connection topology, and management interface mapping results shall include at least the following: For platforms using a server firmware description mechanism, the firmware description information shall primarily include the system management information table and the system configuration and power description table; for edge platforms or on-chip system platforms using a device tree description mechanism, the firmware description information shall primarily include the device tree source file, the device tree binary file, and its corresponding constraint descriptions; the operating system hardware discovery results shall at least include the device enumeration results in the device directory, the device enumeration results in the bus directory, the driver binding results, and the bus scan results; the asset information, status information, and reset capability information of the out-of-band management controller shall at least include the computing system object, chassis object, manager object, log service object, and reset action capability description in the out-of-band management interface resource tree, or the asset object, status management object, and log object in the out-of-band management controller. The above multi-source information shall be uniformly merged using at least one of the following: device unique identifier, bus address, slot number, object path, and serial number. The merged object shall serve as the basis for component domain partitioning and the construction of edges for the four types of relationships.

[0046] In detail, to standardize the measurement scale, the reset sweep span is calculated by normalizing the lengths of each hop path to a decimal weight of the same magnitude as the independent monitoring level. Simultaneously, in unreachable scenarios, a single unreachable constant is uniformly adopted. This constant is preferentially set to the sum of the maximum possible value of the independent monitoring level and one to prevent dimensional imbalance. All comparisons, sorting, and threshold judgments should treat this unreachable constant as greater than any reachable path length, and this value must not change with variations in the target component domain.

[0047] Preferably, the fault record set before and after the protective reset is collected, and the fault record set is converted into the abnormal data residual degree and data retention ratio of each component domain, including: For each of the aforementioned component domains Extract the pre-reset record subsets corresponding to protective resets from the fault record set. Accessible record subset after reset ; For any fault record element in the aforementioned subset of records Calculate the probability of its normal operating condition occurring. : Calculate the residual abnormal data before reset for each of the component domains. Residual abnormal data after reset : Calculate the data retention ratio corresponding to each of the component domains. : in, For the component domain, This is the subset of records before the reset. This refers to the subset of records accessible after the reset. For the fault record element, For the fault record element Frequency of health history; For a complete health history record, For the complete set of health history records Record elements in For the record element Frequency of health history; The probability of the aforementioned normal operating condition occurring; The residual abnormal data before the reset, The residual abnormal data after the reset, For natural logarithm operations; For the component domain The data retention ratio, To prevent the value from being zero or a minimum.

[0048] A fault log set is a collection of all fault evidence records gathered around the target fault node and the protective reset process. It can be obtained through out-of-band management logs, host logs, edge agent logs, interconnect error records, storage logs, and cloud-side image status data.

[0049] The pre-reset record subset is a collection of fault records mapped to the target component domain that occur prior to the protective reset. It can be obtained by truncating the pre-reset window by timestamp and filtering fault records according to component domain affiliation rules.

[0050] The subset of records accessible after a reset is the set of fault records mapped to the target component domain that can still be read, retrieved, or queried after a protective reset. It can be obtained through out-of-band reading, scanning after local node recovery, reading from storage media, and synchronous querying from the cloud.

[0051] Fault log elements are the smallest unit of evidence of fault that can be uniquely identified, counted, and mapped to a component domain. They can be obtained by discretizing log events, error codes, state transitions, sensor mutations, and protocol anomalies.

[0052] The complete health history record is a set of recent stable operation records extracted from the target node or similar nodes under fault-free, reset-free, and alarm-free conditions, according to a preset fixed-length sliding time window, to maintain the stability of the baseline total. It can be obtained through stable operation window records and a fixed-length alarm-free sample library within the health inspection cycle.

[0053] A record element is any countable smallest unit in the complete set of health history records. It can be obtained by applying the same discretization rules to health history samples as those for fault record elements.

[0054] The health history frequency of a fault record element is the cumulative number of times the target fault record element appears in the entire health history record set. It can be obtained by performing isomorphic matching and frequency statistics on the entire health history record set.

[0055] The health history frequency of a record element is the cumulative number of times any record element appears in the entire health history record set. This can be obtained by performing a full frequency count on the entire health history record set.

[0056] The probability of a normal operating condition is the relative probability of a certain fault record element appearing in the entire set of health history records.

[0057] The residual abnormal data before reset is an indicator of the amount of abnormal record information retained by the component domain before protective reset. During extraction, a base placeholder record with basic information is injected into the pre-reset record subset of each component domain to ensure the reference plane for subsequent proportional calculations.

[0058] Post-reset abnormal data residual is an indicator of the amount of abnormal record information that can still be accessed by a component domain after a protective reset.

[0059] The data retention ratio is the ratio of the amount of abnormal information retained by a component domain after a protective reset to the amount of abnormal information before the reset.

[0060] Zero-prevention minimum values ​​are pre-defined extremely small positive numbers used to avoid zero denominators or unstable values ​​in proportional and logarithmic operations. Preferred values ​​are between 0.000000000001 and 0.000000001. The criteria for this value are that it must be significantly smaller than any actual anomaly residual or data retention ratio, and it can only serve to stabilize the value without altering the component domain ordering.

[0061] In detail, because the quantity and quality of anomalous evidence retained across different component domains vary significantly after a protective reset, simply comparing the number of log entries cannot reflect the true information retention situation. Therefore, this application further maps the fault log set to anomalous data residual degree and data retention ratio. This not only describes the amount of anomalous information in each component domain before and after the reset, but also forms a comparable data retention scale across component domains.

[0062] In detail, because not all fault record elements have the same diagnostic value, common records that frequently occur under healthy conditions should not be treated the same as rare abnormal signs. Therefore, this application uses the proportion of health history frequency to define the probability of normal operating conditions. The more common a record element is in the health history, the lower its value as evidence of abnormality; the less common a record element is, the more worthy it is to be retained and paid attention to in a fault scenario.

[0063] In detail, because rare record elements should contribute more information, and multiple record elements need to be uniformly aggregated into a single component domain-level indicator, this application uses the method of summing the sums of the negative natural logarithms of the probabilities of occurrence under normal conditions to construct the residual value of anomalous data. Therefore, rare but critical anomalous signs will be naturally amplified, while the interference of common noise records on the final result will be weakened.

[0064] In detail, because it is necessary to measure how much original fault information a component domain retains after a protective reset, this application normalizes and compares the residual fault data after reset with the residual fault data before reset to obtain the data retention ratio. This indicator can reflect the impact of the reset action on the degree of fault evidence retention.

[0065] In detail, because in real-world scenarios there may be cases where the residual abnormal data before reset is close to zero or some component domains have very few records, direct division could lead to unstable values ​​or even make calculation impossible. Therefore, this application introduces a minimum value to prevent division by zero to ensure stable ratio calculation. This minimum value only serves the function of numerical stabilization and is not used for manually adjusting classification boundaries.

[0066] In detail, the fault record metadata discretization method is as follows: First, the original records are segmented according to the timestamp, source component domain, and event type. Then, the error code, status code, alarm type, protocol anomaly summary, sensor mutation direction, and key context summary are combined to form a unified record metadata primary key. If two records are completely identical in the primary key field and are within the same sampling period, deduplication is performed once; if the time is the same but the source component domain is different, they are retained and mapped to their respective component domains.

[0067] In detail, the method for determining the recording windows before and after reset is as follows: taking the actual time of the protective reset as the central boundary, the time period covering all abnormal signs that have appeared without crossing the previous stable window is taken as the pre-reset window, and the time period from the start of the node becoming accessible again to the completion of the first secondary reset decision is taken as the post-reset accessible window. If a component domain recovers slowly, the end point of its post-reset window should be extended to after the component domain first enters an observable state.

[0068] In detail, the complete set of health history records is constructed as follows: Prioritize selecting logs from stable operating cycles of the target node with no faults, no resets, and no abnormal alarms as health samples for that node; if there are insufficient health samples for that node, supplement them with health samples from nodes of the same hardware model, firmware version, and deployment role. Before samples are added to the database, the same discretization rule and the same component domain mapping rule should be executed, and statistics should be updated at a fixed frequency.

[0069] In detail, the zero-frequency record processing method is as follows: when a certain fault record element has never appeared in the entire set of health history records, a smoothing frequency should first be assigned to this record element before participating in the calculation of the probability of occurrence under normal operating conditions. The smoothing frequency should be less than the minimum frequency of any record element that has actually appeared, so as to ensure that record elements that have never appeared in a healthy state obtain a higher amount of abnormal information.

[0070] In detail, the method for preventing the use of zero minimum values ​​is as follows: within the same diagnostic and reset closed loop, the same minimum positive number is used, and this value is guaranteed to be less than the minimum effective order of magnitude of all real abnormal data residue and data retention ratio.

[0071] In detail, the mapping method from fault records to component domains is as follows: First, determine the primary affiliation based on the record's source channel; then, determine the secondary affiliation based on the device identifier, bus address, process affiliation, container identifier, or storage path; finally, complete the final mapping according to the component domain partitioning table. If the same record is associated with multiple component domains, it is copied and mapped to all relevant component domains according to the principle of least closure, and the same source identifier is retained during statistics to avoid duplicate weighting.

[0072] In detail, the smoothed frequency of zero-frequency records is preferentially taken as half of the smallest non-zero frequency in the current complete set of health historical records under the same statistical caliber; if there are no non-zero frequencies in the current health sample, the smoothed frequency of the zero-frequency record is fixed at half a pseudo-count. When calculating the probability of normal operating conditions, this smoothed frequency should be included in both the numerator and denominator, so that the zero-frequency record obtains a higher amount of abnormal information than any real low-frequency record, while not changing the relative size pattern between the actually occurring record elements.

[0073] In detail, when the same original record is associated with multiple component domains, a unified source identifier should be generated for the original record, and the source identifier, component domain identifier, sampling period, and record element primary key should be used together as the single-domain deduplication key. When the same original record is repeatedly mapped to the same component domain through multiple association paths, it is counted only once; when mapped to different component domains, it can be included in the single-domain statistics of its respective component domain. In subsequent cross-component domain summation or set-level statistics, deduplication should be performed again using the source identifier and record element primary key to avoid the same original evidence being repeatedly amplified due to multi-path copying.

[0074] Preferably, based on the independent monitoring level, the reset wave and span, and the data retention ratio, the data retention gap between adjacent component domains is calculated to extract the maximum record mutation boundary and mutation magnitude, including: For any two adjacent component domains within the component association graph and Calculate the difference between monitoring and reset attribute values. : Calculate the data retention gap value between adjacent component domains. : Extract the maximum record mutation boundary and the magnitude of the mutation : in, and For adjacent component domains, For the component domain The aforementioned independent monitoring level, For the component domain The reset wave and span; For the component domain The aforementioned independent monitoring level, For the component domain The reset wave and span; For the component domain With the component domain The shortest distance within the component association graph, The difference between the monitoring and reset attributes; For the component domain The data retention ratio, For the component domain The data retention ratio, For the prevention of zero minimum value, For natural logarithm operations; The difference value is retained for the data; The set of connection edges in the component association graph. For the set of connecting edges The connecting edges in the middle, The operation is performed on the independent variable to obtain the maximum value. This is for calculating the maximum value. The maximum recorded mutation boundary, The mutation magnitude is mentioned.

[0075] The attribute difference is the intermediate attribute value obtained by subtracting the reset wave span from the independent monitoring level of the component domain.

[0076] The shortest distance between component domains is the shortest graph distance between two target component domains in the component association graph. In actual calculation, the weights of the connecting edges in the association graph are all set to a preset decay decimal of less than one to improve the calculation resolution of the space penalty.

[0077] The difference between monitoring and reset attributes is a normalized difference between adjacent component domains in terms of the combined attributes of independent monitoring level and reset impact span.

[0078] The data retention gap is a measure of the intensity of the difference in the retention of abnormal data between adjacent component domains.

[0079] The connection edge set is the set of all edges in the component association graph used to connect two component domains.

[0080] The maximum record mutation boundary is the adjacent component domain connection edge with the largest data retention drop value in the component association graph.

[0081] The mutation magnitude is the maximum value of the data retention gap corresponding to the maximum recorded mutation boundary.

[0082] In detail, because looking at the independent monitoring level alone only reflects the observation retention capability after a component domain failure, and looking at the reset sweep span alone only reflects the distance of the component domain from the reset source, the real determination of the evidence breakage location is the combined contrast between the two. Therefore, this application first constructs the difference between the independent monitoring level and the reset sweep span as a composite attribute quantity of the component domain. This is the only way to simultaneously incorporate autonomous observation capability and reset vulnerability into the same comparison scale.

[0083] In detail, because even if there are attribute differences between component domains, their proximity in the structure graph needs to be considered. The closer the distance but the greater the difference, the more likely it is to form a real structural discontinuity. Therefore, this application divides the absolute difference of the attribute difference by the sum of the graph distance and one to obtain the monitoring and reset attribute difference value. The larger this value is, the more the two adjacent component domains are located on either side of the boundary where the autonomous retention and reset clearing capabilities differ significantly.

[0084] In detail, because the data retention ratio is a proportional measure, direct subtraction is easily affected by scale compression and cannot fully reflect the step difference between low and high retention areas. Therefore, this application performs a logarithmic transformation on the data retention ratio before comparing the differences. As a result, the changes in the retention ratio at different orders of magnitude are expressed more clearly and are more suitable for identifying record abrupt change boundaries.

[0085] In detail, because structural differences alone are insufficient to prove that a genuine record cliff has occurred, and retention differences alone may simply be ordinary fluctuations, this application multiplies structural differences and retention differences to form a data retention gap value. This product is amplified only when both factors are significant, thereby distinguishing ordinary differences from true record cliffs.

[0086] In detail, because after a protective repositioning, the only truly diagnostically significant boundary is usually the strongest evidence-based mutation edge, this application does not perform multi-boundary fuzzy judgment. Instead, it directly selects the connection edge with the largest data retention drop value as the maximum recorded mutation boundary and uses the corresponding maximum value as the mutation amplitude. This allows subsequent root cause inference to be concentrated near the most critical structural fault.

[0087] In detail, the method for determining adjacent component domains is as follows: if there is at least one direct data dependency edge, control dependency edge, reset propagation edge, or power supply dependency edge between two component domains, they are considered adjacent in the component association graph. The set of connection edges should be formed by deduplicating and merging the above four types of direct edges, and multi-hop relationships that require passing through intermediate component domains are not allowed to be directly regarded as adjacent edges.

[0088] In detail, the shortest distance calculation method for the component domain is as follows: To identify record mutation boundaries, the shortest distance should be calculated on an undirected structural graph composed of data dependencies, control dependencies, reset propagation dependencies, and power supply dependencies. An undirected graph is used because the distance is used to describe structural proximity, rather than propagation direction.

[0089] In detail, the unified way to express the difference between monitoring and reset attributes is as follows: First, calculate the attribute difference obtained by subtracting the reset range from the independent monitoring level of each of the two component domains. Then, take the absolute value of the difference between these two attribute differences, and finally divide by the sum of the shortest distance between the component domains and 1. The unified way to express the data retention difference is as follows: Add the minimum value (excluding zero) to the data retention ratio of each of the two component domains, take the logarithm, take the absolute value of the difference between the two logarithmic results, and finally multiply it by the difference between monitoring and reset attributes.

[0090] In detail, the selection method when the maximum values ​​are tied is as follows: first select the connection edge with the same mutation magnitude but closer to the root cause candidate set; if they are still tied, select the connection edge with the lower data retention in the component fields at both ends that is closer to the protective reset initiating component field than one side; if they are still tied, select according to the component field number order.

[0091] In detail, the method for stabilizing extremely low data retention ratios is as follows: all data retention ratios involved in logarithmic operations are uniformly increased by the same minimum value to prevent division by zero before the operation. If the data retention ratio is already higher than the minimum effective order of magnitude, it should not be further amplified or scaled.

[0092] Preferably, calculating the fault explanatory power of each component domain to the abnormal symptoms, and determining the root cause component domain from each component domain by combining the node distance from the component domain to the boundary of the maximum recorded mutation, includes: The component domains that have experienced an anomaly will be grouped into an anomaly exposed component domain set. ; For the candidate component domain Calculate the fault explanatory power. : Calculate the candidate component domain Root cause candidate composite score : Determine the root cause component domain : in, For the set of abnormally exposed component domains, For the set of abnormally exposed component domains The exposure domain in; The component domains are candidates. For the candidate component domain The degree of fault explanation; For the exposed domain The residual abnormal data before the reset, For the exposed domain The residual abnormal data after the reset; To select from the candidate component domains along data dependencies, control dependencies, and power supply dependencies. To the exposed area The directed path length; For the candidate component domain The root cause candidate comprehensive score; The maximum recorded mutation boundary, For the candidate component domain up to the maximum recorded mutation boundary The node distance; For the candidate component domain The data retention ratio; The operation is performed on the independent variable to obtain the maximum value; The root cause component domain that is finally determined.

[0093] The abnormal exposed component domain set is the collection of component domains in the current fault log that have shown abnormal symptoms and can be identified.

[0094] An exposure domain is any member component domain in the set of abnormal exposure component domains, used to represent the target domain that specifically carries abnormal symptoms.

[0095] Fault interpretability is a score that measures a candidate component domain’s ability to comprehensively explain abnormal changes in all abnormal exposed component domains.

[0096] The directed path length is the shortest path length from which a candidate component domain reaches the target exposed domain along a specified directed dependency.

[0097] The node distance to the maximum record mutation boundary is the graph distance from the candidate component domain to the closer end of the component domains at both ends of the maximum record mutation boundary.

[0098] The root cause candidate composite score is a score that combines the fault explanation, the node distance to the boundary of the maximum record mutation, and the data retention ratio.

[0099] The root cause component domain is the final diagnostic result component domain with the highest comprehensive score among root cause candidates.

[0100] In detail, because the fault symptoms after protective reset are often distributed across multiple component domains, directly inferring the root cause from a single anomaly point can easily lead to misjudgment. Therefore, this application first organizes all exposed anomaly component domains into an anomaly exposed component domain set. In this way, subsequent root cause analysis no longer relies on isolated symptoms, but rather provides a unified interpretation of multiple symptoms.

[0101] In detail, because the core of this localization is to find the source of the state cliff, the true root cause should be strongly correlated with those exposed domains that have experienced severe record loss and corruption. Therefore, this application uses the difference in abnormal residual values ​​before and after reset of each exposed domain as the damage weight, which is then accumulated after decay according to the directed path length and defined as the fault explanatory power. Thus, the better a candidate domain can correlate with and explain the symptoms of severe state abrupt changes, the higher its score.

[0102] In detail, because the maximum recorded mutation boundary is precisely where the evidence retention exhibits the greatest shift before and after protective repositioning, the true root cause component domain is usually closer to the anomalous origin side of this boundary. Therefore, this application introduces the node distance from the candidate component domain to the maximum recorded mutation boundary as a distance penalty term. This can suppress component domains that, while explaining some symptoms, are too far from the critical boundary.

[0103] In detail, because root cause component domains are more likely to exhibit characteristics of erased or damaged fault evidence after undergoing protective reset, this application introduces a difference between the root cause candidate composite score and the data retention ratio. The lower the data retention ratio, the more likely the component domain is to lose original evidence after reset, and the more likely it is to be considered as a root cause source.

[0104] In detail, because the goal of this application is not merely to identify the component domain that first reports an error, but rather to identify the component domain that best explains multiple symptoms, is closest to the critical cliff, and best matches the characteristics of trace loss, the root cause component domain is ultimately determined using the principle of maximizing the comprehensive score of root cause candidates. This result is essentially the optimal solution resulting from the combined effects of structural explanatory power, boundary location, and trace loss.

[0105] In detail, the abnormal exposed component domains are included in the set as follows: if a component domain exhibits at least one type of explicit fault record element, a persistent abnormal state, or a continuous abnormal trend within the current diagnostic window, that component domain will be included in the abnormal exposed component domain set. Explicit fault record elements include at least one of the following: error code, alarm event, protocol exception summary, runtime exception summary, and unrecoverable retry event.

[0106] In detail, the method for determining the scope of candidate component domains is as follows: taking all component domains in the component association graph as the initial candidate set, first removing component domains that are completely isolated from the abnormal exposed component domain set in terms of data dependency, control dependency, and power supply dependency, and then calculating the fault explanatory power and root cause candidate comprehensive score for the remaining component domains.

[0107] In detail, the handling of unreachable directed paths is as follows: when there is no directed path from a candidate component domain to an abnormally exposed component domain, the explanation score of that exposed component domain for the candidate component domain is recorded as 0, and it is not included in the distance decay calculation. This avoids misattributing symptoms with no propagation relationship to unrelated component domains.

[0108] In detail, the method for calculating the distance from a node to the boundary is as follows: First, take the two endpoint component domains in the component domains at both ends of the maximum record mutation boundary, then calculate the shortest graph distance from the candidate component domain to the two endpoint component domains respectively, and take the smaller value as the node distance from the candidate component domain to the maximum record mutation boundary.

[0109] In detail, the unified calculation method for the root cause candidate comprehensive score is as follows: First, divide the fault explanation degree by the sum of the node distance from the candidate component domain to the boundary of the largest record mutation and 1 to obtain the explanation degree after distance penalty. Then, multiply the result by 1 minus the data retention ratio of the candidate component domain to obtain the root cause candidate comprehensive score.

[0110] In detail, the method for determining the highest score in case of a tie is as follows: first, select the component domain with the smaller distance to the boundary node of the largest record mutation; if still tied, select the component domain with the lower data retention ratio; if still tied, select the component domain with the higher fault interpretability; if still tied, determine the component domain according to the component domain number order.

[0111] Preferably, constructing an anomalous propagation component set of the root cause component domain, and selecting a controlled hardware reset instruction for the anomalous propagation component set based on a hardware restart cost evaluation function, includes: The root cause component domain is located by following the data dependencies, the control dependencies, and the power supply dependencies. The reachable domain, and its relation to the root cause component domain. Combined into the abnormal propagation component set : For candidate reset instructions Determine its scope of application and generate instruction override identifiers. : Calculate the first cost for the component domains not within the abnormal propagation component set, and the second cost for the uncovered component domains within the abnormal propagation component set, respectively. Use the sum of the first cost and the second cost as the hardware restart cost evaluation function. Result: Select the hardware restart cost evaluation function The candidate reset instruction corresponding to the minimum result As the controlled hardware reset command : in, For the root cause component domain, For the data dependency, For the control dependency, For the power supply dependency, The function for finding reachable paths. For the set of abnormal propagation components; The candidate reset instruction. For the component domain, For the candidate reset command The actual set of component domains that play a role. The instruction override identifier; For the component domain The residual abnormal data after the reset, For the component domain The residual abnormal data before the reset, This is the hardware restart cost evaluation function; The set of all candidate reset instructions supported by the system. The operation corresponding to the minimum value of the independent variable is performed. This is the final selected control hardware reset instruction.

[0112] Reachable domains are the set of component domains that a root component domain can propagate to along data dependencies, control dependencies, and power dependencies.

[0113] An anomaly propagation component set is an anomaly propagation closure composed of the root cause component domain and all component domains reachable along multiple dependency relationships.

[0114] Candidate reset instructions are single hardware reset actions that the platform currently supports, can be scheduled for execution, and have a identifiable scope. They can be obtained through out-of-band management interface capability enumeration, platform reset action list, firmware command descriptions, and management controller interface queries.

[0115] The actual set of affected component domains is the set of all component domains that will actually be affected by a reset after a candidate reset instruction is executed. It can be obtained through a reset capability mapping table, reset propagation relationships, platform management interface specifications, and historical action verification results.

[0116] The instruction overwrite flag is a binary flag used to indicate whether a candidate reset instruction overwrites the target component field.

[0117] The first cost is the collateral damage cost introduced when a candidate reset instruction covers a component domain outside the abnormal propagation component set.

[0118] The second cost is the residual fault cost introduced when the candidate reset instruction fails to cover the component domain within the abnormal propagation component set.

[0119] The hardware restart cost evaluation function is a reset instruction evaluation function that quantifies both the associated damage cost and the underwrite cost.

[0120] The candidate reset instruction set is the collection of all candidate reset instructions that the platform can invoke on the current node. It can be obtained through out-of-band management controller capability queries, reset action interface lists, firmware command enumeration, and platform driver capability discovery.

[0121] A controlled hardware reset instruction is the only target reset instruction that is ultimately selected for execution after cost evaluation.

[0122] In detail, because real-world faults often do not remain within a single root cause component domain but spread outwards along data paths, control paths, and power supply sustaining paths, this application does not only focus on a single root cause for reset decisions but also further constructs a set of fault-proliferating components. This allows the root cause domain and its potentially affected neighboring domains to be included in the subsequent reset coverage analysis.

[0123] In detail, because different hardware reset instructions have different coverage areas—some only affect the host domain, while others affect the entire system or local interconnect domains—this application establishes an instruction coverage identifier for each candidate reset instruction and each component domain. This allows the subsequent cost function to clearly calculate which component domains a reset action should cover and which should not be covered.

[0124] In detail, because resetting component domains outside the abnormal propagation component set often destroys valuable evidence still preserved in these component domains, which is a typical over-reset loss, this application defines this part of the accidental damage cost as the first cost. The more component domains are accidentally damaged, and the greater the abnormal information they retain after resetting, the higher the first cost.

[0125] In detail, because if the candidate reset command does not cover specific component domains, especially those that experienced severe state gaps and record loss during the initial protective reset, it means that these blind spots that have fallen into state disorder have not been repaired. Therefore, this application defines the potential risks caused by not covering such record gap domains as a second cost. The higher this cost, the less the reset action covers those damaged core areas that urgently need to be forcibly cleaned.

[0126] In detail, because controlled hardware reset must avoid both over-reset causing evidence damage and under-reset causing fault residue, this application unifies the costs of over-reset and under-reset into a single cost evaluation function and selects the candidate reset instruction with the smallest total result as the final execution action. This approach transforms the reset granularity selection from an empirical decision to a computable decision.

[0127] In detail, the reachability domain search method is as follows: starting from the root component domain, search outwards along the directed edges of data dependencies, control dependencies, and power dependencies to find all reachable component domains; set access flags for visited component domains, and if a visited component domain is reached again, the branch will not be expanded further; when a branch has no more unvisited successor component domains, the search for that branch will be terminated.

[0128] In detail, the candidate reset instruction set is generated as follows: All executable reset types are first extracted from the reset action interfaces exposed by the out-of-band management controller, and then supplemented with platform firmware-defined reset actions and local bus reset actions to form a unified set. All candidate reset instructions should be standardized with a unique identifier, unique scope, and unique triggering method before being included in the set.

[0129] In detail, the actual active component domain set is generated as follows: first, the nominal active object of the candidate reset instruction is read, then all component domains that will be affected by the reset action are expanded outward along the reset propagation relationship, and finally the nominal active object and the propagation coverage object are merged into the actual active component domain set of the candidate reset instruction.

[0130] In detail, the weights of the cost function are determined as follows: unless otherwise specified by the platform security constraints, the first cost and the second cost are added together with equal weights; if the platform sets a constraint that prevents accidental damage to a certain type of core component domain, the weight of the first cost is allowed to be increased without changing the monotonicity of the function, but the weights must remain fixed in the same node decision.

[0131] In detail, the selection method when the minimum cost is tied is as follows: first, select the candidate reset instruction with a smaller actual set of component domains; if they are still tied, select the candidate reset instruction with a smaller impact on the managed component domain set; if they are still tied, select the candidate reset instruction with a shorter execution latency.

[0132] In detail, the method for verifying coverage relationships is as follows: After generating instruction coverage identifiers, the coverage result of each component domain should be compared one by one with the actual set of active component domains, and the coverage result should be verified through historical reset playback or capability mapping rules. If there are discrepancies between the coverage identifiers and the actual set of active component domains, all coverage identifiers should be regenerated based on the actual set of active component domains.

[0133] Preferably, the fault record set is extracted and locally managed, and the hardware recovery order is determined according to the order of the power supply dependencies. The controlled hardware reset instruction is executed domain-by-domain release based on the condition that observable data is read for the first time. This includes: Combine to generate a collection of managed component domains : Extract the managed component domain set The subset of records in the component domains are merged to obtain the locally managed record set. Managed by the company: Extract the controlled hardware reset command The actual set of component domains The corresponding power supply dependency The hardware recovery order is obtained by performing topological sorting. : Issue the controlled hardware reset command and in accordance with the hardware recovery order. Release the component domain, for the currently released component domain Calculate its first observable time And upon reaching that moment, the release of the next said component domain will be triggered: in, For the set of managed component domains, and For the component domain, and For the aforementioned independent monitoring level, To perform the maximum value operation, and The maximum recorded mutation boundary The component domains at both ends; For the locally hosted record set, This is the subset of records before the reset. This refers to the subset of records accessible after the reset. This is a union operation; This is the controlled hardware reset command. The set of component fields to which the controlled hardware reset command actually operates. To limit the power supply dependencies within the set of component domains where the controlled hardware reset command actually operates, This is a topological sorting function. The hardware recovery sequence; For the currently released component domain, The first observable time, To perform the bounding operation, For continuous time variables, The time when the instruction is issued. For the component domain Real-time generation of record sets, It is an empty set.

[0134] The managed component domain set is an evidence-priority retention set composed of the component domain with the largest independent monitoring level and the component domains at both ends of the largest record mutation boundary.

[0135] The local managed record set is a local evidence set formed by combining the subset of records before the reset of each component domain within the managed component domain set with the subset of records accessible after the reset.

[0136] The power supply dependency after limiting the scope is the local power supply dependency obtained by restricting the power supply dependency to the actual influence range of the controlled hardware reset instruction.

[0137] The hardware recovery order is the release order obtained by topologically sorting the power supply dependencies corresponding to the set of component domains that are actually affected by the controlled hardware reset command.

[0138] The component domain being released is the one that was released from holding or resumed operation at the current moment during the domain-by-domain release process.

[0139] The first observable moment is the moment when a non-empty record first appears in the real-time generated record set of the currently released component domain.

[0140] The command issuance time is the actual point in time when the controlled hardware reset command is sent by the management plane or execution module. It can be obtained through the management controller command log, execution module timing records, and system audit timestamps.

[0141] The real-time generated recordset is a collection of records that a component domain continuously generates over time during the recovery process and can be read by the system. It can be obtained through real-time log subscription, out-of-band polling, event stream acquisition, bus status monitoring, and local cache reading.

[0142] In detail, because the component domain with the largest independent monitoring level is usually the most likely to retain observable clues after a failure, and the component domains at both ends of the largest recorded mutation boundary are precisely where the evidence break is most obvious, this application includes both types of component domains in the managed component domain set. This allows for priority protection of the most critical diagnostic clues before performing a second controlled reset.

[0143] In detail, because a controlled hardware reset could erase previously accessible traces, this application requires that critical evidence within the managed component domain set be extracted and merged into a locally managed recordset before the reset is performed. This ensures that even if some component domains are subsequently reset again, important original records for review and verification will not be lost.

[0144] In detail, because the recovery of different component domains cannot succeed in any arbitrary order, many component domains must wait for the upstream power supply domain, management domain, or interconnect domain to recover before they can become operational. Therefore, this application performs topological sorting of the power supply dependencies within the scope of the controlled hardware reset to obtain the hardware recovery order. This can avoid recovery failure or the generation of new accompanying anomalies due to incorrect release order.

[0145] In detail, because releasing all component domains at once would obscure key observation points during the recovery process and make it difficult to confirm which component domain has been restored to an observable state, this application adopts a domain-by-domain release mechanism. After releasing each component domain, its recovery status is observed before deciding whether to release the next domain, thereby maintaining the verifiability and controllability of the recovery process.

[0146] In detail, because the time it takes for each component domain to recover to an observable state varies greatly under different platforms, temperatures, and interconnection conditions, a fixed waiting time would result in insufficient waiting on some platforms and excessive waiting on others. Therefore, this application does not use a fixed delay, but instead uses the first non-empty record set generated in real time as the condition for releasing the next domain. This allows the recovery rhythm to adapt to the actual hardware state.

[0147] In detail, the local managed record set is stored as follows: Priority is given to writing to the local persistent storage domain, which is unaffected by controlled hardware reset commands. If the local persistent storage domain is also within the reset range, then priority should be given to writing to the independent storage area of ​​the out-of-band management controller. Writing employs a combination of append-only writing and integrity verification, and a managed completion flag is generated upon completion of the writing process.

[0148] In detail, the topology sorting method for handling parallel nodes is as follows: when multiple component domains simultaneously satisfy the condition that the in-degree is zero, the component domain with the higher upstream power supply level is released first; if they are still parallel, the out-of-band management related component domains are released first; if they are still parallel, they are released in the order of component domain number.

[0149] In detail, the method for determining first observable data is as follows: only the smallest record unit that can be mapped to the currently released component domain and can prove that the component domain has entered an operational or queryable state can be considered valid observable data. This smallest record unit includes at least one of the following: state refresh event, link establishment event, registration event, storage index readable event, or error cleanup event.

[0150] In detail, the release monitoring cycle is set as follows: after the current released component domain enters the recovery phase, the real-time generated record set of that component domain is continuously read at fixed short intervals until the first observable data is detected or the maximum allowable recovery time is reached. The maximum allowable recovery time should be determined based on the historical recovery time distribution of that component domain, and a uniform fixed value unrelated to the component domain type should not be used.

[0151] In detail, the hardware operation mapping method for release by domain is as follows: For each component domain to be released, the corresponding control action for that component domain is invoked, such as releasing hold, restoring power supply, releasing reset line, or restoring link enable. After executing the control action for each component domain, the observability monitoring phase for that component domain must be entered immediately; the monitoring phase must not be skipped consecutively to release the next domain directly.

[0152] In detail, the recovery failure handling method is as follows: if the currently released component domain does not produce the first observable data within the maximum allowed recovery time, the release of subsequent component domains will be immediately stopped, and the current component domain will be marked as a recovery failure domain. At the same time, the managed record set and the current monitoring record will be retained for subsequent fault review and manual intervention.

[0153] In detail, before performing topology sorting on power supply dependencies after limiting the scope, it is necessary to check for the existence of directed loops. If a directed loop exists, the component domains within the loop are shrunk into the same synchronous release group according to strongly connected components, and this synchronous release group is treated as a supernode to participate in subsequent topology sorting. When the platform controller does not support strict synchronous release, the component domains within the loop are released serially within the same release group according to the rule of out-of-band management domain priority, upstream power supply priority, and component domain number order. However, all release actions within this release group must be completed in the same group step and cannot be interleaved across groups.

[0154] In detail, the maximum allowable recovery time for the currently released component domain is preferentially taken as the 95th percentile of the historical recovery time distribution of that component domain. When the number of historical samples for that component domain is lower than the preset minimum sample threshold, the 95th percentile of samples of the same hardware model, firmware version, and component domain type is taken instead. If there are still insufficient samples, the largest of the following three values ​​is used as a fallback value: the platform firmware power-on timing table, the device datasheet, or the default waiting upper limit of the management controller. The minimum sample threshold, percentile, and fallback value remain fixed within the same diagnostic and reset closed loop, and the minimum sample threshold is preferably 20.

[0155] Preferably, after the release is performed, the drop convergence ratio of the maximum recorded mutation boundary and the residual anomaly amount within the anomaly propagation component set are calculated to generate and store a reset effectiveness index, including: After all domains of the controlled hardware reset command have reached the first observable moment, the data retention gap between adjacent component domains is recalculated to obtain an updated data retention gap value. And calculate the drop convergence ratio. : Recalculate the residual reset abnormal data for each of the component domains. And calculate the abnormal propagation component set. The residual anomaly within : The drop convergence ratio With the residual anomaly Add them together to generate the reset effectiveness index. And store: in, To retain the difference value for the updated data, This is an operation to find the maximum value in the set of connected edges in the component association graph; The magnitude of the mutation, For the prevention of zero minimum value, The drop convergence ratio is the ratio of the elevation difference. For the set of abnormal propagation components, The component domain within the set of abnormal propagation components; The residual abnormal data is the degree of reset. The residual abnormal data before the reset, The residual anomaly; This is the indicator of the effectiveness of the reset.

[0156] The updated data retention gap value is the data retention gap value between adjacent component domains that is recalculated after the controlled hardware is reset and the domain-by-domain release is completed.

[0157] The drop convergence ratio is a ratio of the maximum update data retention drop value to the initial mutation magnitude.

[0158] Residual abnormal data after reset is an indicator of the amount of residual abnormal information recalculated by each component domain after controlled hardware reset.

[0159] The residual anomaly quantity is the total amount of residual anomalies accumulated across all component domains within the anomaly propagation component set.

[0160] The repositioning effectiveness index is a comprehensive repositioning effect evaluation index formed by adding the drop convergence ratio and the residual anomaly.

[0161] In detail, because the original maximum recorded mutation boundary may have changed after the controlled hardware is reset, only by recalculating the data retention gap between adjacent component domains can it be determined whether the structural evidence break has truly converged. Therefore, this application requires a second calculation after all domains are made observable again.

[0162] In detail, because it is necessary to quantitatively represent how much the maximum recorded mutation boundary has converged relative to the initial state after the controlled hardware reset, this application compares the maximum drop after reset with the initial mutation amplitude, defining it as the drop convergence ratio. The smaller this ratio, the more significantly the original strongest evidence cliff has been compressed.

[0163] In detail, even if the maximum record mutation boundary converges, it does not mean that there is no residual fault information inside the abnormal propagation component set. Therefore, this application recalculates the residual abnormal data of each component domain after reset to measure whether the abnormal information still remains in the system.

[0164] In detail, because a comprehensive evaluation of residual anomalies within the anomaly propagation component set is required, and the baseline of anomaly information content differs across component domains, this application first normalizes the residual anomaly data degree of each component domain relative to its pre-reset anomaly data degree. Then, before summing, it multiplies this by the node proportion weight of that component domain in the set, and finally sums it within the anomaly propagation component set to obtain a residual anomaly quantity with controlled magnitude. This avoids scale expansion and prevents a small number of high-baseline domains from masking residual problems in other low-baseline domains.

[0165] In detail, because boundary convergence alone cannot reflect whether root cause propagation has been completely attenuated, and residual anomalies alone cannot reflect whether structural cliffs have been repaired, this application adds the drop convergence ratio to the residual anomalies to form a reset effectiveness index. This index simultaneously characterizes the repair status of structural evidence breaks and the residual degree of anomaly propagation closure, and can be used for subsequent write-back, review, and strategy optimization.

[0166] In detail, the secondary calculation window is determined as follows: the starting point of the window is the moment when the controlled hardware reset is actually completed and all actually functioning component domains reach their first observable state, and the ending point of the window is the moment when the system re-enters a stable monitoring state or reaches the preset verification duration. Records collected within this window are used to recalculate and update the data retention gap and the residual abnormal data after reset.

[0167] In detail, the method for calculating the residual abnormality data after reset is as follows: It adopts the same caliber as summing the summations of the negative natural logarithms of the probability of occurrence under normal operating conditions, except that the input record set is changed to the residual abnormality record set after reset within the secondary calculation window. This ensures that the three indicators—before reset, after reset, and in the residual stage—are comparable on the same scale.

[0168] In detail, the method for determining whether all scopes have reached an observable state is as follows: all scopes are considered to have reached their first observable moment only when every component field in the set of component fields actually affected by the controlled hardware reset instruction has been recorded at its first observable moment, and there are no recovery failure fields. If any component field fails to recover, the final index calculation stage cannot proceed.

[0169] In detail, the method for storing the reset validity index is as follows: the reset validity index, along with the fault node identifier, root cause component domain identifier, maximum record mutation boundary identifier, controlled hardware reset command identifier, and the timestamp of this diagnosis, are written into the local operation and maintenance knowledge base and synchronized to the cloud-side analysis library after the cloud link is restored.

[0170] In detail, the reset effectiveness metric is used as a feedback item for the historical effect of the next round of similar fault diagnosis, and is used to update the health history model, reset command priority, and anomaly propagation component set coverage strategy. The lower the metric, the more effective the controlled hardware reset in this round, and the more likely its corresponding parameter combination should be retained.

[0171] In detail, the processing method when updating the maximum drop is parallel is as follows: prioritize the connection edge located near the original maximum record mutation boundary as the reference edge after the update; if the parallel edges are not near the original boundary, select the connection edge inside the abnormal propagation component set; if they are still parallel, determine the connection edge according to the connection edge number order.

[0172] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A fault root cause localization and hardware reset system for cloud-edge collaborative architecture, characterized in that, include: The graph construction module is used to construct a component association graph of the fault node. The component association graph includes power supply dependencies and calculates the independent monitoring level and reset sweep span of each component domain in the component association graph. The mapping module is used to collect fault record sets before and after protective reset, and convert the fault record sets into abnormal data residue and data retention ratio of each component domain. The boundary extraction module is used to calculate the data retention difference value between adjacent component domains based on the independent monitoring level, the reset wave and span and the data retention ratio, so as to extract the maximum record mutation boundary and mutation amplitude. The root cause localization module is used to determine the set of abnormally exposed component domains based on the fault record set, calculate the fault explanatory power of each component domain to the set of abnormally exposed component domains, and determine the root cause component domain from each component domain by combining the node distance from each component domain to the boundary of the maximum record mutation. The reset selection module is used to construct the abnormal propagation component set of the root cause component domain, and select a controlled hardware reset instruction for the abnormal propagation component set based on the hardware restart cost evaluation function. The reset execution module is used to extract the fault record set for local management, determine the hardware recovery order according to the order of the power supply dependencies, and execute the domain-by-domain release of the controlled hardware reset instruction based on the condition that observable data is read for the first time. The verification write-back module is used to calculate the drop convergence ratio of the maximum record mutation boundary and the residual abnormality in the abnormal propagation component set after the release is executed, generate a reset validity index and store it.

2. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 1, characterized in that, Construct a component association graph for the faulty node, the component association graph including power supply dependencies, and calculate the independent monitoring level and reset sweep span of each component domain within the component association graph, including: Extract the component list and connection topology of the faulty node, and generate the component association graph including data dependency, control dependency, reset propagation relationship and power supply dependency; A reset reachability matrix is ​​constructed based on the reset propagation relationship to characterize whether any component domain in the component association graph can be reached by reset propagation. For each component domain in the component association graph, the total number of monitoring observation surfaces that can still be observed when a fault occurs is counted, which is used as the independent monitoring level. The monitoring observation surfaces include out-of-band management observation surfaces and edge local autonomous observation surfaces. The initiating component domain corresponding to the protective reset is determined, and the shortest path from the initiating component domain to each component domain is found in the reset propagation relationship. The length of the shortest path is taken as the reset sweep span.

3. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 2, characterized in that, Collect fault record sets before and after protective reset, and convert the fault record sets into abnormal data residual degree and data retention ratio for each component domain, including: For each component domain, extract the pre-reset record subset and the post-reset accessible record subset corresponding to the protective reset from the fault record set; For any fault record element in the pre-reset record subset and the post-reset accessible record subset, divide its health history frequency in the full health history record set by the sum of the health history frequencies of all record elements in the full health history record set to obtain the probability of normal operation. Calculate the negative of the natural logarithm of the probability of normal operation of each fault record element in the pre-reset record subset and the post-reset accessible record subset, and sum the calculation results in the corresponding subsets to obtain the pre-reset abnormal data residual degree and the post-reset abnormal data residual degree corresponding to each component domain. Divide the residual abnormal data after reset by the sum of the residual abnormal data before reset and the minimum value of zero prevention, to obtain the data retention ratio corresponding to each component domain.

4. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 3, characterized in that, Based on the independent monitoring level, the reset wave and span, and the data retention ratio, the data retention gap between adjacent component domains is calculated to extract the maximum record mutation boundary and mutation magnitude, including: For any two adjacent component domains in the component association diagram, calculate the difference between the independent monitoring level and the reset wave span of the two component domains respectively to obtain the attribute difference between the two component domains; Calculate the absolute value of the difference between the two attribute differences, and divide the absolute value of the difference by the sum of the shortest distance between the two component domains in the component association graph and one, to obtain the difference value between the monitoring and reset attributes; Calculate the natural logarithm of the sum of the data retention ratio and the minimum value of zero for each of the two component domains, and then calculate the absolute value of the difference between the two natural logarithms. Multiply the difference value between the monitoring and reset attributes by the absolute value of the difference to obtain the data retention gap value between adjacent component domains; In the component association graph, the connection edge of the adjacent component domain corresponding to the largest data retention gap value is selected as the maximum record mutation boundary, and the largest data retention gap value is taken as the mutation amplitude.

5. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 4, characterized in that, Based on the fault record set, a set of abnormally exposed component domains is determined. The fault explanatory power of each component domain to the set of abnormally exposed component domains is calculated. Combining the node distances from each component domain to the boundary of the largest recorded mutation, the root cause component domain is determined from each component domain, including: The component domains that have experienced an anomaly are grouped into the set of anomaly-exposed component domains; For the candidate component domains, calculate the difference between the abnormal data residual before reset and the abnormal data residual after reset for each abnormal exposed component domain in the set of abnormal exposed component domains. Divide the difference by the sum of the directed path length from the candidate component domain to the abnormal exposed component domain along the data dependency, the control dependency, and the power supply dependency, and obtain the explanation score. Sum the results to obtain the fault explanation degree. The fault explanation is divided by the sum of the node distances from the candidate component domain to the boundary of the maximum record mutation and one, and multiplied by one by the difference between the data retention ratio of the candidate component domain and one, to obtain the root cause candidate comprehensive score. The component domain corresponding to the one with the highest comprehensive score among the root cause candidates is selected as the root cause component domain.

6. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 5, characterized in that, Constructing the set of anomalous propagation components in the root cause component domain, and selecting a controlled hardware reset instruction for the set of anomalous propagation components based on a hardware restart cost evaluation function, including: The reachable domains of the root cause component domain are found along the data dependencies, the control dependencies, and the power supply dependencies, and then combined with the root cause component domain to form the anomaly propagation component set. Generate an instruction overwrite flag for candidate reset instructions; The first cost is obtained by summing the product of the instruction overwrite identifier of the component domain that is not in the set of abnormal propagation components and the residual abnormal data after reset; The second cost is obtained by summing the difference between the residual abnormal data before reset and the residual abnormal data after reset in the component domains that are not covered within the abnormal propagation component set. The sum of the first cost and the second cost is used as the result of the hardware restart cost evaluation function; The candidate reset instruction with the smallest result is selected as the controlled hardware reset instruction.

7. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 6, characterized in that, Extract the fault record set for local hosting, determine the hardware recovery order based on the order of the power supply dependencies, and execute the controlled hardware reset instruction's domain-by-domain release based on the condition that observable data is read for the first time, including: The component domain with the largest independent monitoring level and the component domains at both ends of the largest record mutation boundary are combined to obtain a managed component domain set; Extract the pre-reset record subset and the post-reset accessible record subset of the component domains within the managed component domain set, and merge them to obtain a local managed record set for management; Extract the set of actual active component domains corresponding to the controlled hardware reset command, and perform topological sorting on the power supply dependency relationships corresponding to the set of actual active component domains to obtain the hardware recovery order; Issue the controlled hardware reset command and release each component domain in the actual active component domain set according to the hardware recovery order; After the instruction is issued, monitor the real-time generation record set of the currently released component domain, take the moment when the real-time generation record set is not empty for the first time as the first observable moment, and trigger the release of the next component domain after the first observable moment is reached.

8. The fault root cause localization and hardware reset system for cloud-edge collaborative architecture according to claim 7, characterized in that, After the release is executed, the drop convergence ratio of the maximum recorded mutation boundary and the residual anomaly amount within the anomaly propagation component set are calculated to generate and store a reset effectiveness index, including: After all the component domains in the actual active component domain set corresponding to the controlled hardware reset command have reached the first observable time, the data retention gap value between adjacent component domains is recalculated to obtain an updated data retention gap value. The largest gap in the updated data retention is divided by the sum of the mutation amplitude and the minimum value of the zero-prevention method to obtain the gap convergence ratio; Recalculate the residual abnormal data of each component domain after reset; For each component domain within the abnormal propagation component set, the residual abnormal data degree after reset is divided by the sum of the corresponding residual abnormal data degree before reset and the minimum value of zero prevention, and then the results are summed to obtain the residual abnormal amount. The drop convergence ratio is added to the residual anomaly to generate the reset effectiveness index, which is then stored.

Images