A device fault automatic recovery method, device, apparatus and storage medium

Abstract

The application provides a device fault automatic recovery method and device, equipment and storage medium, the method comprises the following steps: collecting device operation index to generate real-time state data; according to real-time state data, the device operation index is predicted, and prediction data is obtained, when the prediction data exceeds the preset early warning threshold, an early warning signal is generated, the corresponding prediction data is matched with the feature of the fault knowledge graph, and business early warning data is generated; based on the early warning signal and the business early warning data, the fault node of the data transmission path is inquired reversely through link tracking identification, the business early warning data is compared with the fault node to determine the root cause information corresponding to the fault node; according to the root cause information, the recovery strategy is matched from the strategy library, and the recovery instruction is sent to the fault node according to the recovery strategy. The application can accurately predict before the fault occurs, and the fault node can be recovered according to the corresponding root cause information, so that the loss of the device after the fault occurs is prevented.

Description

Technical Field

[0001] This application relates to the field of equipment management technology, and in particular to an automated method, apparatus, device and storage medium for equipment fault recovery. Background Technology

[0002] In existing system fault management technologies, early warning processing solutions generally adopt a passive mode of early warning after a fault occurs, that is, the early warning process is only triggered when obvious faults such as equipment downtime or service interruption have occurred. This post-event processing mechanism means that fault location and recovery depend entirely on the fact that the fault has occurred, and the system cannot intervene in advance when the fault has not yet caused actual business impact. Summary of the Invention

[0003] This application provides a method, apparatus, device, and storage medium for automated equipment fault recovery, which addresses the problem in related technologies that equipment cannot intervene in advance before a fault causes actual business impact.

[0004] The first aspect of this application provides an automated equipment fault recovery method, the automated equipment fault recovery method comprising: Collect device operation metrics from both end-side and edge-side devices, and combine them with distributed system logs to generate real-time status data; Based on the real-time status data, the equipment operation indicators are predicted. When at least one predicted data exceeds a preset warning threshold, a warning signal is generated. The predicted data corresponding to the warning signal is then matched with the fault knowledge graph to generate business warning data. Based on the warning signal and the business warning data, the fault node of the data transmission path is queried in reverse through the preset link tracing identifier, and the business warning data is compared with the fault node to determine the root cause information corresponding to the fault node. Based on the root cause information, a recovery strategy is matched from the strategy library, and a recovery command is sent to the faulty node according to the recovery strategy.

[0005] Optionally, in the first implementation of the first aspect of this application, before the step of performing feature matching between the predicted data corresponding to the warning signal and the fault knowledge graph to generate business warning data, the method further includes: Obtain the abnormal indicator values ​​corresponding to the warning signal and the key log entries in the distributed system log; The abnormal indicator values ​​and the key log entries are mapped to graph nodes respectively, and edge connections are constructed based on the time correlation between the abnormal indicator values ​​and the key log entries to generate a real-time fault subgraph.

[0006] Optionally, in the second implementation of the first aspect of this application, the step of performing feature matching between the predicted data corresponding to the warning signal and the fault knowledge graph to generate business warning data includes: Align the nodes of the real-time fault subgraph with the pre-constructed fault knowledge graph to determine the feature similarity between each node in the real-time fault subgraph and each historical fault node in the fault knowledge graph. Based on the feature similarity, query the historical fault nodes and their adjacent nodes that match the real-time fault subgraph in the fault knowledge graph; The feature vector of the matching node is weighted and summed with the feature vectors of the neighboring nodes, and the weighted sum is subjected to a nonlinear transformation to output the matching probability corresponding to each fault type. When the matching probability of any fault type exceeds a preset probability threshold, corresponding business warning data is generated.

[0007] Optionally, in the third implementation of the first aspect of this application, after the step of performing feature matching between the predicted data corresponding to the warning signal and the fault knowledge graph to generate business warning data, the method further includes: Based on the fault type and prediction data in the business early warning data, the corresponding early warning title and risk description are matched from the preset early warning text library, and the abnormal indicator value and predicted occurrence time in the prediction data are filled into the corresponding position in the risk description to generate a pop-up prompt text. Based on the fault type, query the corresponding operation instruction information from the preset suggestion strategy library. The operation instruction information includes at least one executable operation instruction or configuration modification guide. The pop-up notification text and the operation instruction information are combined and encapsulated into warning data, and then sent to the receiving end via the communication bus.

[0008] Optionally, in the fourth implementation of the first aspect of this application, the step of reversely querying the fault node of the data transmission path based on the warning signal and the service warning data through a preset link tracing identifier, and comparing the service warning data with the fault node to determine the root cause information corresponding to the fault node includes: Based on the warning signal and the business warning data, the forwarding records of each hop node in the data transmission path are traversed in reverse according to the link tracing identifier to locate the last node that successfully forwarded the data, and the next hop node of that node is identified as the fault node. Obtain the real-time logs of the faulty node and the status snapshot within a preset time period before the fault occurred, extract the abnormal event sequence from the real-time logs, and extract the abnormal time period of the indicators from the status snapshots. Align the fault types in the business early warning data with the abnormal event sequence and the abnormal time period of the indicator on the time axis to determine the correlation confidence between the business early warning data and the abnormal event sequence and the abnormal time period of the indicator; The abnormal event with the highest correlation confidence is selected as the root cause event, and root cause information is generated based on the event type of the root cause event and the associated device parameters.

[0009] Optionally, in a fifth implementation of the first aspect of this application, the step of matching a recovery strategy from the strategy library based on the root cause information and sending a recovery instruction to the faulty node according to the recovery strategy includes: Based on the fault type and root cause event type in the root cause information, a corresponding recovery action sequence is matched from the preset recovery strategy library. The recovery action sequence includes one or more basic recovery actions arranged in execution order. When there are multiple fault nodes, obtain the hierarchical relationship of each fault node in the device topology, assign priority values ​​to each fault node according to the hierarchical relationship, and merge the recovery action sequence of each fault node into an ordered task queue according to the priority values ​​from high to low. According to the order in the ordered task queue, recovery commands are sent one by one to the faulty node through the soft bus command channel.

[0010] A second aspect of this application provides an automated equipment fault recovery device, which is used to implement an automated equipment fault recovery method. The automated equipment fault recovery device includes: The data acquisition module is used to collect the device operation indicators of the end-side devices and the edge-side devices, and combine them with the distributed system logs to generate real-time status data. The early warning module is used to predict the equipment operation indicators based on the real-time status data. When at least one predicted data exceeds a preset early warning threshold, an early warning signal is generated, and the predicted data corresponding to the early warning signal is matched with the fault knowledge graph to generate business early warning data. The determination module is used to, based on the warning signal and the business warning data, reverse query the fault node of the data transmission path through a preset link tracing identifier, and compare the business warning data with the fault node to determine the root cause information corresponding to the fault node; The recovery module is used to match a recovery strategy from the strategy library based on the root cause information, and send a recovery command to the faulty node according to the recovery strategy.

[0011] Optionally, the early warning module includes: The node alignment unit is used to align the real-time fault subgraph with the pre-constructed fault knowledge graph, and determine the feature similarity between each node in the real-time fault subgraph and each historical fault node in the fault knowledge graph. The query unit is used to query historical fault nodes and their adjacent nodes that match the real-time fault subgraph in the fault knowledge graph based on the feature similarity. The output unit is used to perform a weighted summation of the feature vector of the matching node and the feature vector of the neighboring node, and to perform a nonlinear transformation on the weighted summation result to output the matching probability corresponding to each fault type. The generation unit is used to generate corresponding business early warning data when the matching probability of any fault type exceeds a preset probability threshold.

[0012] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the automated device fault recovery method provided in the first aspect of this application.

[0013] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps of the automated equipment fault recovery method provided in the first aspect of this application.

[0014] In summary, the automated equipment fault recovery method, apparatus, device, and storage medium provided in this application collect equipment operation indicators from end-side and edge-side devices and combine them with distributed system logs to generate real-time status data. Based on the real-time status data, the device operation indicators are predicted. When at least one predicted data point exceeds a preset warning threshold, a warning signal is generated. The predicted data corresponding to the warning signal is then matched with a fault knowledge graph to generate business warning data. Based on the warning signal and the business warning data, a fault node in the data transmission path is queried in reverse using a preset link tracing identifier. The business warning data and the fault node are compared for root causes to determine the root cause information corresponding to the fault node. A recovery strategy is matched from the strategy library based on the root cause information, and a recovery command is sent to the fault node according to the recovery strategy. This application, by combining the predicted warning signal with the fault knowledge graph feature matching to generate business warning data containing fault type information, and by using this business warning data to query the link tracing identifier in reverse to determine the fault node, can accurately predict faults before they occur and recover fault nodes based on the corresponding root cause information, preventing equipment losses after a fault occurs. Attached Figure Description

[0015] Figure 1 A flowchart illustrating the automated equipment fault recovery method provided in this application embodiment; Figure 2 yes Figure 1 A flowchart of an embodiment of step S120; Figure 3 yes Figure 1 A flowchart of an embodiment of step S150; Figure 4 A schematic diagram of the program modules of the automated equipment fault recovery device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] To address the issue in related technologies where equipment cannot intervene proactively before a fault causes actual business disruption, embodiments of this application provide an automated equipment fault recovery method, such as... Figure 1 This is a flowchart illustrating the automated equipment fault recovery method provided in this embodiment. The automated equipment fault recovery method includes the following steps: Step 110: Collect the device operation indicators of the end-side device and the edge-side device, and combine them with the distributed system logs to generate real-time status data.

[0018] Specifically, operational metrics of both end-side and edge-side devices are collected via a distributed communication bus, along with distributed system logs. These logs are then uniformly time-aligned to generate structured real-time status data. During this process, embedded acquisition components on the device side periodically read metrics such as CPU load, memory usage, and network throughput, uploading them to edge nodes via low-latency communication. Simultaneously, the distributed log system continuously records operation logs and anomaly information during device operation, sorting them by timestamp. Subsequently, a time synchronization mechanism aligns and merges the metric data and log data, constructing an information set containing multi-dimensional features to obtain the corresponding real-time status data, ensuring data consistency and comparability during analysis.

[0019] Step 120: Predict equipment operation indicators based on real-time status data. When at least one predicted data exceeds a preset warning threshold, generate a warning signal and perform feature matching between the predicted data corresponding to the warning signal and the fault knowledge graph to generate business warning data.

[0020] Specifically, after obtaining real-time status data, a time-series prediction model is constructed to analyze the trends of equipment operation indicators, thereby obtaining predicted data for a preset time period. An early warning signal is generated when the prediction results meet the warning conditions. In this process, firstly, continuous time series data is extracted from the real-time status data as model input, and normalization is used to eliminate the influence between different units. Then, the processed sequence is input into a multi-layered status update structure, which models the equipment operation trend by memorizing and updating historical status information. Furthermore, based on the model output, it is determined whether the indicators will show a continuous increase or abnormal fluctuation in the future. When the predicted value exceeds a preset threshold and shows a continuous abnormal trend, an early warning signal is generated to achieve early identification of potential faults. After the early warning signal is generated, the corresponding predicted data is matched with a pre-constructed fault knowledge graph to form business early warning data with a clear fault indication. In this process, firstly, abnormal indicator features are extracted from the predicted data, and key log events are filtered from the distributed logs, mapping the above information into nodes in a graph structure. Then, connections between nodes are established based on time correlations to form a real-time fault subgraph reflecting the current system state. Next, through node alignment and feature similarity calculation, this subgraph is matched with historical fault patterns in the knowledge graph, and the matching probability of various faults is calculated through feature aggregation and nonlinear transformation. Finally, when the matching probability of a certain fault type exceeds a set threshold, the fault type is combined with an early warning signal to generate business early warning data, thereby improving the accuracy and interpretability of the early warning.

[0021] Step 130: Based on the early warning signal and business early warning data, reverse query the fault node of the data transmission path through the preset link tracing identifier, and compare the business early warning data with the fault node to determine the root cause information corresponding to the fault node.

[0022] Specifically, after obtaining the early warning signal and business early warning data, the data transmission path is analyzed in reverse based on the link tracing identifier to locate the fault node and determine the corresponding root cause information. In this process, firstly, the forwarding records of data between each device node are traced back based on the link tracing identifier, analyzing the node status step by step during data transmission. Then, by identifying the last node that successfully completed data forwarding and combining it with the communication status of its next-hop node, the fault node that may have an anomaly is determined. Next, the node's operating logs and status snapshots before and after the fault occur are obtained, and abnormal event sequences and abnormal indicator intervals are extracted. Then, the fault types contained in the business early warning data are aligned with the above-mentioned abnormal information on a timeline, and the confidence level of each candidate event is calculated through correlation analysis. Finally, the abnormal event with the highest confidence level is selected as the root cause event, and complete root cause information is generated by combining it with relevant device parameters.

[0023] Step 140: Match a recovery strategy from the strategy library based on the root cause information, and send a recovery command to the faulty node according to the recovery strategy.

[0024] Specifically, after determining the root cause information, the system retrieves and matches corresponding recovery strategies from a pre-defined strategy library and sends recovery commands to the faulty nodes to achieve automated processing. In this process, firstly, based on the fault type and event characteristics in the root cause information, a corresponding recovery action sequence is selected from the strategy library. This sequence consists of multiple basic operations in a predetermined order. Then, in the case of multiple faulty nodes, the nodes are prioritized according to their hierarchical relationship in the system topology, and their respective recovery action sequences are integrated into a unified task execution queue. Next, a distributed communication mechanism is used to issue recovery commands one by one according to the queue order, causing each node to execute the corresponding operation sequentially. Finally, after the recovery actions are completed, the system returns to normal operation, thereby achieving automated handling of equipment faults and improving system stability.

[0025] In one optional implementation, the step of performing time-series prediction of equipment operation indicators based on real-time status data to obtain predicted data within a preset future time period includes: extracting the indicator values ​​of the most recent K time points from the real-time status data, constructing a fixed-length input sequence, and normalizing each value in the input sequence; sequentially inputting the normalized input sequence into a three-layer serial state update unit, determining the retention ratio, discard ratio, and output ratio of the current input based on the current input value and the short-term state value passed from the previous unit; wherein the short-term state value of the first-layer state update unit is zero; updating the long-term memory state value according to the retention ratio, attenuating the updated long-term memory state value according to the discard ratio, and modulating the attenuated long-term memory state value according to the output ratio to generate the short-term state output value of the current state update unit; after undergoing three layers of state updates, weighting and summing the short-term state output value output by the third-layer state update unit through a linear mapping matrix, and restoring the predicted data for the next P time points through reverse mapping.

[0026] In this embodiment, after obtaining real-time status data, the device operation indicators of the most recent K time points are extracted from the continuous time series to construct a fixed-length input sequence, where K is an integer greater than or equal to 1, and K represents the length of the historical time window used for analysis. For example, in the scenario of monitoring the operation status of a smart home gateway, CPU load and memory usage data collected per second within the last 60 seconds can be selected as components of the input sequence. Since different indicators differ in their numerical ranges—for example, CPU load is usually expressed as a percentage while network throughput is expressed in Mbps—normalization processing is required to avoid the impact of numerical scale differences on subsequent status calculations. This involves mapping each indicator to a unified numerical range through linear transformation, thereby ensuring the comparability of features across dimensions during calculation. The sequence obtained after normalization not only retains the trend of the original data but also reduces the interference of extreme values ​​on the overall calculation. Based on this, the normalized input sequence is sequentially fed into a three-layer serial state update structure. Each layer contains two types of internal variables: short-term state values ​​and long-term memory state values. Short-term state values ​​represent the immediate characteristic response at the current moment, while long-term memory state values ​​accumulate historical information to reflect trend changes. For each input value, it is first fused with the short-term state value passed from the previous moment to obtain three proportional parameters for regulating information flow: the retention ratio controls the degree of retention of historical information at the current moment, the discard ratio suppresses the influence of invalid or outdated information on the current judgment, and the output ratio adjusts the intensity of information output at the current moment. Since the first layer does not have short-term state input from the previous layer, its initial short-term state value is set to a zero vector to ensure consistency of the calculation starting point. During the operation of the smart home gateway, for example, when the CPU load continues to rise, the retention ratio will gradually increase, thereby enhancing the memory of historical high-load states, while the discard ratio weakens sudden short-term fluctuations. Subsequently, the long-term memory state value is dynamically updated according to the aforementioned proportional parameters. First, the long-term memory state value from the previous moment is merged with the current input information based on the retention ratio, thus achieving the continuation of historical trends and the introduction of new information. Next, the updated long-term memory state value is attenuated using a discard ratio to reduce the cumulative effect of irrelevant information. Then, the processed long-term memory state value is modulated using an output ratio to generate the short-term state output value for the current moment. This output value includes both the immediate characteristics of the current input and the evolutionary information of historical states. For example, before the gateway experiences port occupancy anomalies, its CPU load and memory usage show a gradual upward trend. Through the above state update process, the strengthening characteristics of the abnormal trend can be gradually reflected in the short-term state output.After the input sequence passes through a three-layer state update structure, the short-term state values ​​output by the third layer have completed multi-level feature extraction and time-dependent modeling. At this point, the output vector is input into a linear mapping matrix for weighted summation. This linear mapping matrix is ​​a set of parameters used to convert the high-dimensional state representation into the target prediction space, establishing a mapping relationship between the internal state and the indicator values ​​at future time points. Subsequently, a reverse mapping process restores the calculation results to the numerical range of the actual physical quantity, thus obtaining the predicted data for P future time points. For example, in the smart home gateway scenario mentioned above, a predicted curve of the CPU load within the next 10 seconds can be output. When the prediction result shows that the load will approach the upper limit of the threshold in a short time, it can provide a basis for subsequent early warning and fault handling, thereby achieving forward-looking analysis and intervention of the device's operating status.

[0027] In one optional implementation of this embodiment, the normalized input sequence is sequentially input to a three-layer serial state update unit. The steps for determining the retention ratio, discard ratio, and output ratio of the current input based on the current input value and the short-term state value passed from the previous unit include: vector fusion of the current input value and the short-term state value passed from the previous unit, followed by adjustment of the forgetting weight parameter and nonlinear interval compression to generate the retention ratio; vector fusion of the current input value and the short-term state value passed from the previous unit, followed by adjustment of the input weight parameter and nonlinear interval compression to generate the input ratio; and vector fusion of the current input value and the short-term state value passed from the previous unit, followed by adjustment of the output weight parameter and nonlinear interval compression to generate the discard ratio.

[0028] In this embodiment, as Figure 2As shown, during the state update process, the current input value and the short-term state value passed from the previous unit need to undergo vector fusion processing to form a comprehensive representation that can simultaneously reflect the current observation information and historical state characteristics. Vector fusion refers to combining two multi-dimensional numerical sequences according to their corresponding dimensions to express them in the same feature space. Common implementations include concatenation or weighted summation of dimensions. In this scheme, a weight matrix is ​​introduced to perform a linear transformation on the two data before superposition, thereby achieving deep coupling at the feature level. The linear transformation can be understood as a process of weighted reorganization of the original data, that is, using a set of adjustable parameters to characterize the importance of different dimensions, making the fused result more prominent in terms of features related to the current state change. For example, in the smart home gateway operation scenario, the current input value contains the latest CPU load and memory usage information, while the short-term state value records the trend change of the previous moment. After fusion, both the current load level and the load change trend can be reflected simultaneously. After completing the vector fusion, three different sets of weight parameters need to be introduced to adjust the fusion result, thereby generating the corresponding control ratio. The forgetting weight parameter adjusts the degree to which historical information is retained at the current moment. Essentially, it's a set of coefficients with the same dimension as the fusion vector. Through element-wise multiplication with the fusion result, different features have differentiated effects in subsequent calculations. The input weight parameter characterizes the contribution of the current input information to the overall state update, strengthening or weakening the influence of new input features in the fusion result. The output weight parameter controls the intensity of information transmission outward, determining the range of influence of the current state on subsequent levels. All three types of weight parameters are implemented through a linear transformation with the fusion vector, resulting in a numerical sequence with the same dimension as the original vector. After obtaining the weighted result, further nonlinear interval compression is needed to generate the corresponding scaling parameters. Nonlinear interval compression maps values ​​from any range to a defined interval. Here, it's used to restrict the calculation result to between 0 and 1, allowing it to participate as a scaling factor in subsequent state updates. This process is achieved by introducing a nonlinear function with saturation characteristics. Its feature is that the output tends to stabilize when the input value is large or small, thus avoiding imbalance caused by extreme values. For example, when the value of a certain dimension in the fusion result is high, the corresponding ratio after compression is close to 1, indicating that the feature has a high retention or input weight at the current time; conversely, when the value is low, the output is close to 0, indicating that the feature has a weak influence.During the operation of the smart home gateway, when the system detects a continuous increase in CPU load that already showed an upward trend in the previous moment, the value of the corresponding dimension in the fusion result will increase significantly. After adjustment by the forgetting weight parameter, the value of this dimension in the retention ratio approaches the upper limit, thus reinforcing the historical high-load state in long-term memory. Simultaneously, the input weight parameter adjusts the input ratio according to the change in the current input value, ensuring that new load changes are reflected in the state update in a timely manner. The output weight parameter generates a discard ratio based on the overall state change, suppressing irrelevant or low-fluctuation features. For example, when memory usage changes relatively steadily, the corresponding fusion result, after compression, has a lower discard ratio, thus reducing the impact of this feature on subsequent outputs. Through this continuous calculation process, proportional parameters matching the system's operating state can be dynamically generated at each moment, enabling the state update to both remember historical trends and respond sensitively to current changes, thereby providing stable and discriminative feature representations for the prediction results.

[0029] It should be noted that in the time series prediction part, the training process revolves around the temporal evolution of device operation indicators. Historical state data of multiple devices during long-term operation are collected from a distributed environment. This data is arranged in chronological order and includes continuously sampled indicators such as CPU load, memory usage, and network throughput, with precise timestamps to ensure the temporal consistency of the sequence. The raw data is then preprocessed, with missing values ​​imputed to ensure sequence integrity and abnormal noise smoothed to avoid interfering with trend judgment. Normalization is used to unify indicators of different dimensions to the same numerical range, making each indicator comparable in subsequent calculations. Based on this, training samples are constructed using a sliding window approach, taking fixed-length historical time segments as input and the actual indicator values ​​of subsequent time segments as output, thus forming a mapping relationship between input and target. The constructed samples are input into a prediction unit composed of a multi-layered state update structure. Forward computation yields the prediction results, and the error between the prediction results and the actual values ​​is used as a feedback signal to adjust internal parameters, gradually bringing the prediction results closer to the actual trend. During training, parameters are continuously updated through multiple iterations, and the prediction performance is validated on independent subsets of data to avoid over-reliance on specific data features. Once training is complete, the distribution of prediction errors in the validation data is analyzed to determine the range of warning thresholds. This allows the system to assess the potential for abnormal trends in actual operation based on the prediction results. For example, in the operation of a smart home gateway, if historical data shows repeated instances of gradually increasing CPU usage leading to service anomalies, the trained prediction unit can identify similar trends before the load reaches its limit, thus issuing a risk signal in advance.

[0030] In one optional implementation, before the step of performing feature matching between the predicted data corresponding to the warning signal and the fault knowledge graph to generate business warning data, the method further includes: obtaining the abnormal indicator value corresponding to the warning signal and the key log entries in the distributed system log; mapping the abnormal indicator value and the key log entries to graph nodes respectively, and constructing edge connections based on the time correlation between the abnormal indicator value and the key log entries to generate a real-time fault subgraph.

[0031] In this embodiment, after the warning signal is generated, abnormal indicator values ​​and key log entries from the distributed system log are extracted from the corresponding time window. Abnormal indicator values ​​refer to operational indicators that have exceeded preset thresholds or exhibit abnormal trends, such as CPU load continuously approaching its limit or memory usage rapidly increasing in a smart home gateway scenario. Key log entries refer to abnormal records occurring within the same time frame, such as port occupancy, abnormal process exits, or resource allocation failures. Event tags with clear semantics are extracted by parsing the log text and aligned with the abnormal indicators in the time dimension, allowing both types of information to reflect different aspects of the same abnormal process. Based on this, abnormal indicator values ​​and key log entries are mapped to nodes in a graph structure. Graph nodes are a unified representation of different types of information, and each node contains a numerical description of its corresponding characteristics. For example, indicator nodes may contain indicator type, numerical value, and rate of change, while log nodes contain event type and occurrence time. Edge connections are constructed based on the temporal correlation between abnormal indicators and log entries. Edge connections are used to describe the degree of correlation between nodes. The core of edge connections is to determine the strength of causal relationship through time intervals and order of occurrence. For example, if the increase in CPU load occurs before the port occupation log, a directed connection from the indicator node to the log node can be established, and different weight values ​​can be assigned according to the length of the time interval, thereby forming a real-time fault subgraph that can reflect the current abnormal evolution process.

[0032] In one optional implementation, the step of generating business early warning data by performing feature matching between the predicted data corresponding to the early warning signal and the fault knowledge graph includes: obtaining the abnormal indicator value corresponding to the early warning signal and the key log entries in the distributed system log; mapping the abnormal indicator value and the key log entries to graph nodes respectively, and constructing edge connections based on the time correlation between the abnormal indicator value and the key log entries to generate a real-time fault subgraph; aligning the real-time fault subgraph with the pre-constructed fault knowledge graph to determine the feature similarity between each node in the real-time fault subgraph and each historical fault node in the fault knowledge graph; querying the historical fault nodes and their adjacent nodes that match the real-time fault subgraph from the fault knowledge graph based on the feature similarity; performing a weighted summation of the feature vectors of the matching nodes and the feature vectors of the adjacent nodes, and performing a nonlinear transformation on the weighted summation result to output the matching probability corresponding to each fault type; when the matching probability of any fault type exceeds a preset probability threshold, combining the fault type and the early warning signal to encapsulate business early warning data.

[0033] In this embodiment, the real-time fault subgraph is aligned with a pre-constructed fault knowledge graph. The fault knowledge graph is a structured relationship network built from historical fault data, containing various types of nodes and their relationships, including devices, metrics, log events, and fault types. The node alignment process maps nodes in the real-time subgraph to semantically similar or identical nodes in the knowledge graph. Matching is achieved by comparing node feature vectors, which are numerical representations of node attributes, such as converting high CPU load or port occupancy into multi-dimensional numerical representations. By calculating the feature similarity between real-time subgraph nodes and historical nodes in the knowledge graph, the proximity between the current abnormal state and historical fault patterns can be quantified. A higher similarity score indicates greater consistency in their manifestation and evolutionary characteristics. After obtaining the similarity results, historical fault nodes with high matching scores to the real-time fault subgraph are queried from the knowledge graph, and their neighboring nodes are extracted simultaneously. Neighboring nodes refer to nodes in the knowledge graph that are directly associated with the fault node, such as metric anomalies and log events associated with configuration conflicts. The matching node and its neighboring nodes' feature vectors are weighted and summed. Weighted summation assigns different weights based on the strength of the connections between nodes, linearly combining multiple feature vectors to obtain a comprehensive representation of the fault characteristics. This result is then subjected to a nonlinear transformation, mapping the linear combination to a finite interval to enhance feature discriminative power and suppress noise interference, ultimately yielding the matching probability for each type of fault. Taking a smart home gateway as an example, when two nodes—one with continuously increasing CPU load and the other with port occupancy—exist in the real-time subgraph, and there is a strong temporal correlation between them, they will have a high similarity to the set of nodes in the knowledge graph that configure conflicting fault modes. After feature aggregation and nonlinear transformation, the matching probability for this fault type will be significantly higher than other types. When the matching probability of a certain fault type exceeds a preset threshold, this fault type is combined and encapsulated with the original warning signal to form business warning data containing fault category information. This ensures that the warning results not only reflect abnormal trends but also clearly indicate the specific type of potential fault, providing a direct basis for subsequent location and recovery.

[0034] It should be noted that in the fault feature matching part, the training process constructs structured relationships based on historical fault cases and achieves fault type identification by learning the relationships between nodes. First, a large amount of fault data is collected. Each data record includes changes in operational indicators before the fault occurred, log events within the corresponding time period, and the finally confirmed fault type label. Then, the data is parsed, abstracting equipment, indicator status, log events, and fault types into different categories of nodes. Connections between nodes are established based on actual causal relationships; for example, high CPU load and abnormal process exits are associated with resource contention faults, thus forming a relational structure with clear semantics. On this basis, the attributes of each node are transformed into numerical feature vectors to describe its state characteristics and semantic information. Substructures composed of multiple nodes and their connections are input as training samples into the feature matching unit. Through multi-layered information transmission and aggregation of node features and their adjacency relationships, nodes not only retain their own features during the update process but also integrate the contextual information of surrounding nodes, thus forming a comprehensive representation that reflects the overall fault mode. The resulting representation is compared with the corresponding fault type label, and the internal parameters are adjusted based on the classification error, allowing the system to gradually learn the differences between different fault modes. After training, when a new abnormal state is mapped as a graph structure input, the matching probability of each fault type can be calculated by comparing its similarity to historical structures.

[0035] In one optional implementation, after the step of performing feature matching between the predicted data corresponding to the warning signal and the fault knowledge graph to generate business warning data, the method further includes: matching the corresponding warning title and risk description from a preset warning text library based on the fault type and predicted data in the business warning data, and filling the abnormal indicator value and predicted occurrence time in the predicted data into the corresponding position in the risk description to generate a pop-up prompt text; querying the corresponding operation instruction information from a preset suggestion strategy library based on the fault type, wherein the operation instruction information contains at least one executable operation instruction or configuration modification guide; combining and encapsulating the pop-up prompt text and operation instruction information into warning data, and sending it to the receiving end through a communication bus.

[0036] In this embodiment, after obtaining the business early warning data, the corresponding early warning title and risk description are matched from a preset early warning text library based on the fault type and prediction data contained therein. The early warning text library is a structured collection of texts for various fault scenarios. Each record contains a title template and a description template corresponding to a specific fault type. The templates have reserved placeholder fields for dynamic data; for example, the time and threshold positions in "Equipment load will reach {threshold} at {time}" are replaceable content. During the matching process, the fault type in the business early warning data is identified and parsed, and used as an index to retrieve the corresponding template record in the text library. Based on the abnormal indicator value and predicted occurrence time provided in the prediction data, the aforementioned placeholder fields are replaced with specific values, thereby generating a pop-up notification text with clear semantics. The process of filling in placeholder fields needs to ensure consistency in values ​​and units. For example, in a smart home gateway scenario, when the prediction result shows that the CPU load will reach 90% in the next 10 minutes, the time field in the risk description template can be replaced with "10 minutes later," and the threshold field can be replaced with "90%" to form a complete and expressive prompt, allowing users to intuitively understand the potential risk. After generating the pop-up prompt text, based on the fault type in the business warning data, the corresponding operation instructions are queried from the preset suggestion strategy library. The suggestion strategy library is a collection of feasible handling measures under different fault scenarios. Each record contains operation instructions corresponding to the fault type. The operation instructions are used to guide security personnel in handling warning faults, and their form can be specific operation instructions or configuration adjustment instructions. Operation instructions refer to execution content that can directly affect the device's operating status, such as closing a specified process, restarting a service, or releasing resources; configuration modification instructions are used to explain the parameters that need to be adjusted and how to modify them, such as changing port configurations or adjusting resource allocation strategies. During the query process, the fault type is used as a key index to locate the corresponding strategy record, and the operation instructions matching the current fault scenario are extracted. For example, in the event of a port conflict at the gateway, operation instructions such as "check port usage and terminate abnormal processes" and "reconfigure port listening parameters" can be obtained, providing clear guidance for the handling process. The generated pop-up prompt text and operation instructions are combined and encapsulated to form structured warning data. The encapsulation process integrates different types of information according to a unified data format, enabling transmission and parsing within the system or between devices. This structured data typically includes a title field, a description field, and an operation suggestion field, each carrying different dimensions of information content, ensuring that the receiving end can accurately parse and display the corresponding information. After encapsulation, the warning data is sent to the receiving end via a communication bus, which is a unified channel for information transmission between devices, supporting low-latency and highly reliable data distribution.For example, in the application scenario of smart home gateways, when an abnormal increase in CPU load is detected and matched with the corresponding fault type, warning data is generated, including the title "Gateway Load Risk Warning", the description "CPU load will reach 90% in 10 minutes" and operation instructions such as "Terminate abnormal process and optimize resource configuration". This data is then sent to the user terminal device via the communication bus, allowing the receiving end to display the complete information in the form of a pop-up window, thereby enabling timely prompts and handling guidance for potential risks.

[0037] In one optional implementation, the steps of determining the root cause information corresponding to the fault node by reverse querying the data transmission path based on the warning signal and business warning data, using a preset link tracing identifier, and comparing the business warning data with the fault node to identify the root cause information of the fault node include: based on the warning signal and business warning data, traversing the forwarding records of each hop node in the data transmission path in reverse according to the link tracing identifier, locating the node that successfully forwarded the last data, and determining the next hop node of that node as the fault node; obtaining the real-time log of the fault node and the state snapshot within a preset time period before the fault occurred, extracting the abnormal event sequence from the real-time log, and extracting the abnormal time period of the indicator from the state snapshot; aligning the fault type in the business warning data with the abnormal event sequence and the abnormal time period of the indicator on the time axis to determine the correlation confidence between the business warning data and the abnormal event sequence and the abnormal time period of the indicator; selecting the abnormal event with the highest correlation confidence as the root cause event, and generating root cause information based on the event type of the root cause event and the associated device parameters.

[0038] In this embodiment, after obtaining the early warning signal and business early warning data, the link tracing identifier is used to perform reverse analysis on the data transmission path between devices. The link tracing identifier is a unique marker assigned to each data transmission process, used to record the forwarding path and status information of the data between nodes. By reading the forwarding record corresponding to this identifier, the data can be traversed step-by-step in reverse chronological order through each hop node. Each node's forwarding record contains information such as data reception time, forwarding time, and forwarding result status. By comparing the records of adjacent nodes, it can be determined whether the data transmission was interrupted. When a node's forwarding record is found to be successful, but its next-hop node does not have a corresponding successful reception record, the successful node can be determined as the last normal forwarding node, and its successor node is the faulty node that may be abnormal. For example, in the communication process between a smart home gateway and a lighting controller, if the gateway shows that it has successfully sent a control command, but the controller does not record any reception behavior, the controller can be identified as a faulty node. After locating the faulty node, the real-time log of the node and a status snapshot for a period of time before the fault occurred are further obtained. The status snapshot is a continuous record of the device's operating indicators within a certain time interval, used to reflect the device's status change process. By parsing real-time logs, abnormal event sequences can be extracted, which are sets of abnormal behaviors arranged in chronological order, such as port occupancy, thread blocking, and resource allocation failures. Simultaneously, abnormal time periods for metrics can be identified from state snapshots, i.e., time intervals during which operational metrics deviate from normal ranges, such as periods where CPU load consistently exceeds thresholds or memory usage rises rapidly. These two types of information describe the fault occurrence process from the behavioral and state levels, respectively, providing a foundation for subsequent correlation analysis. Based on this, the fault type information contained in the business warning data is time-aligned with the abnormal event sequences and abnormal time periods for metrics. Time-alignment means arranging data from different sources according to a unified time base to analyze their chronological relationships and overlap. By comparing the time matching degree between the typical event characteristics corresponding to the fault type and the actually observed abnormal events, the correlation confidence between various abnormal events and warning information can be calculated. The correlation confidence measures the likelihood that a particular abnormal event is the root cause, and its calculation is based on factors such as the degree of overlap between the event occurrence time and the abnormal metric interval, the degree of matching between the event type and the fault type, and the event duration. For example, in a scenario where a gateway configuration conflict occurs, if a port occupancy event occurs in the early stages of a CPU load increase and persists, the confidence level of its correlation with the warning information will be significantly higher than that of other short-lived abnormal events.Finally, based on the comparison of the correlation confidence scores of each abnormal event, the abnormal event with the highest value is selected as the root cause event. Root cause information is then generated by combining the event type of this event with relevant device parameters. These device parameters include specific attributes such as port number, process identifier, and resource usage, used to further describe the specific manifestation of the fault. For example, in the scenario above, when the correlation confidence score corresponding to port 23 being occupied by an abnormal process is the highest, this event can be identified as the root cause event. Furthermore, by combining the port number and process information, root cause information for the port 23 configuration conflict is generated, thus providing a clear basis for matching subsequent recovery strategies and achieving precise localization from abnormal phenomena to specific fault causes.

[0039] In one optional implementation, the steps of matching a recovery strategy from a strategy library based on root cause information and sending recovery instructions to the faulty node according to the recovery strategy include: matching a corresponding recovery action sequence from a preset recovery strategy library based on the fault type and root cause event type in the root cause information, wherein the recovery action sequence includes one or more basic recovery actions arranged in execution order; when there are multiple faulty nodes, obtaining the hierarchical relationship of each faulty node in the device topology, assigning priority values ​​to each faulty node according to the hierarchical relationship, and merging the recovery action sequences of each faulty node into an ordered task queue according to the priority values ​​from high to low; and issuing recovery instructions to the faulty node one by one through the soft bus instruction channel according to the order in the ordered task queue.

[0040] In this embodiment, as Figure 3As shown, after obtaining the root cause information, a matching process is performed in a preset recovery strategy library based on the fault type and root cause event type contained therein to determine the corresponding recovery action sequence. The recovery strategy library is a set of rules formed by structuring and organizing historical processing experience. Each strategy is indexed by fault type and event characteristics and associated with a set of basic recovery actions arranged in execution order. Basic recovery actions refer to the smallest operational unit that can directly affect the device's operating state, such as closing abnormal processes, releasing occupied ports, restarting services, or reloading configurations. Each action includes execution conditions, target objects, and execution instructions. During the matching process, the event attributes in the root cause information are parsed, such as port number, process identifier, or resource usage, and compared with the characteristic conditions recorded in the strategy library to find the recovery action sequence that best matches the current fault characteristics. These are then organized into a set of operations to be executed consecutively in a preset order. For example, in a scenario where a smart home gateway experiences port conflicts, an action sequence containing actions such as terminating abnormal processes, releasing port resources, and restarting the listening service can be matched, giving the recovery process a clear execution path. When multiple faulty nodes exist in a system, recovery tasks need to be prioritized based on the devices' positions within the overall network structure. Device topology is a structural model describing the connections and data flows between devices, typically represented hierarchically. For example, gateways are at the core layer, controllers at the access layer, and sensors at the edge layer. By analyzing the hierarchical position of each faulty node within the topology, a corresponding priority value can be assigned. This priority value indicates the order in which recovery operations are performed; a higher value indicates a higher priority. In smart home scenarios, the gateway acts as the data hub, and its failure affects multiple downstream devices; therefore, its priority is higher than that of the terminal controller. When multiple nodes fail simultaneously, the recovery action sequences for each node are sorted from highest to lowest priority, and the action sequences from different nodes are merged into a unified, ordered task queue. Each task in this queue includes not only the specific action to be performed but also the corresponding target device identifier, thus forming a cross-device collaborative recovery execution structure. After the task queue is constructed, recovery commands are sent to the target nodes one by one according to the queue order via the soft bus command channel. The soft bus command channel is a low-latency command transmission path built on a distributed communication mechanism, which enables fast and reliable distribution of control commands between different devices. Each recovery command contains specific execution parameters and the target device address. During the transmission process, the system schedules commands one by one according to the task queue order, and triggers the transmission of the next command only after each command is executed, so as to ensure the orderliness of the recovery process and the correctness of the dependencies.For example, in a scenario where both the gateway and the lighting controller malfunction simultaneously, the system first sends a command to the gateway to restart the service and release the port. After confirming that the gateway has recovered, it then sends a configuration reset and restart command to the controller. This prevents downstream operations from failing due to the upstream node's failure to recover. Through this execution process, orderly recovery in a multi-device environment can be achieved, allowing each node to gradually return to a normal state according to its topological dependencies, thereby improving the overall system's stability and recovery efficiency.

[0041] In one optional implementation, when multiple devices are detected to have associated faults, a device association graph is constructed based on the data dependencies between the fault nodes, and the state constraints of each fault node are extracted. Based on the root cause information of each fault node, corresponding recovery actions are matched from a preset recovery strategy library, and trigger conditions are configured for each recovery action. The trigger conditions are used to characterize the states of other devices on which the recovery action depends. Each recovery action and its trigger conditions are mapped to state-driven execution units, and trigger relationships between execution units are established based on the device association graph. When any execution unit meets its trigger conditions, a recovery command is sent to the corresponding fault node through a distributed communication channel, and the state information of the corresponding device is updated after execution. Based on the real-time update results of the device state, it is dynamically determined whether the trigger conditions of other execution units are met, thereby triggering the execution of subsequent recovery actions to achieve collaborative automatic recovery among multiple devices.

[0042] In this embodiment, after detecting multiple interconnected faults in devices, a device relationship graph is constructed based on the data interaction relationships between the devices. This graph describes the dependencies between different devices in the data transmission and control process, where nodes represent specific devices and edges represent dependency constraints between devices. For example, the data processing of one device depends on the output of another device. In a smart home scenario, the gateway is responsible for data forwarding and control command distribution, while the lighting controller relies on the gateway to receive commands. Therefore, a dependency relationship is formed in the relationship graph, pointing from the gateway to the controller. Simultaneously, combining the operating status and root cause information of each fault node, state constraints describing the recovery conditions are extracted. These state constraints limit the execution prerequisites for a device's recovery operation, such as gateway communication returning to normal or port resources being released, thus providing a basis for triggering subsequent recovery actions. Based on this, according to the root cause information of each fault node, corresponding recovery actions are matched from a preset recovery strategy library, and trigger conditions are configured for each recovery action. The trigger conditions refer to the states of other devices or the previous states of the device on which the recovery action depends. When the trigger conditions are not met, the corresponding recovery action remains in a waiting state; when the conditions are met, the recovery action can be activated. For example, in a scenario where both the gateway and the lighting controller fail simultaneously, the lighting controller's recovery action can be configured to depend on the gateway's communication recovery status, thus avoiding premature execution of invalid operations before the gateway has recovered. This transforms the recovery action from a simple sequential execution to condition-triggered execution dependent on state changes. Subsequently, each recovery action and its triggering condition are mapped to an independent execution unit. Each execution unit is a structured entity containing action instructions, triggering conditions, and target device information. Triggering relationships between execution units are established based on a device association graph, enabling different execution units to interact through state changes. For instance, when the gateway's recovery action is completed and its status is updated to "communication normal," the execution unit associated with this status will be activated, triggering the lighting controller's recovery action. This process does not rely on a fixed order but rather on dynamic scheduling based on state changes. During execution, when the triggering condition of any execution unit is met, a recovery instruction is sent to the corresponding device through a distributed communication channel. After execution, the status information fed back by the device is obtained, and the device status is updated. The status update not only reflects whether the current device has recovered but also determines whether the triggering conditions of other execution units have been met, thus forming a linkage mechanism based on state propagation. For example, after the gateway recovers, its status change will immediately trigger the controller's recovery operation without waiting for a preset execution sequence. As the status of each device continues to update, the system dynamically determines whether the recovery actions that have not yet been executed meet the triggering conditions and gradually activates the corresponding execution units, thereby forming a collaborative recovery process among multiple devices.In smart home scenarios, this mechanism can automatically trigger controller recovery after the gateway recovers, and then further trigger terminal devices to update synchronously, so that the entire system recovers step by step according to the dependency relationship, without relying on a fixed priority queue, thereby improving the flexibility and response efficiency of the recovery process.

[0043] According to the automated equipment fault recovery method provided in this application, by combining the early warning signal triggered by time-series prediction with the feature matching of fault knowledge graph, business early warning data containing fault type information is generated, and the link tracing identifier is queried in reverse based on the business early warning data to determine the fault node. It can accurately predict before the fault occurs and recover the fault node according to the corresponding root cause information to prevent the equipment from being damaged after the fault occurs.

[0044] Figure 4 This application provides an automated equipment fault recovery device, which can be used to implement the automated equipment fault recovery method described in the foregoing embodiments. For example... Figure 4 As shown, the automated fault recovery device for this equipment mainly includes: The data acquisition module is used to collect the device operation indicators of the end-side devices and the edge-side devices, and combine them with the distributed system logs to generate real-time status data. The early warning module is used to predict the equipment operation indicators based on the real-time status data. When at least one predicted data exceeds a preset early warning threshold, an early warning signal is generated, and the predicted data corresponding to the early warning signal is matched with the fault knowledge graph to generate business early warning data. The determination module is used to, based on the warning signal and the business warning data, reverse query the fault node of the data transmission path through a preset link tracing identifier, and compare the business warning data with the fault node to determine the root cause information corresponding to the fault node; The recovery module is used to match a recovery strategy from the strategy library based on the root cause information, and send a recovery command to the faulty node according to the recovery strategy.

[0045] In an optional implementation of this embodiment, the early warning module is further configured to: obtain the abnormal indicator value corresponding to the early warning signal and the key log entries in the distributed system log; map the abnormal indicator value and the key log entries to graph nodes respectively, and construct edge connections according to the time correlation between the abnormal indicator value and the key log entries to generate a real-time fault subgraph.

[0046] In an optional implementation of this embodiment, the early warning module includes: a node alignment unit, used to align the real-time fault subgraph with a pre-constructed fault knowledge graph, and determine the feature similarity between each node in the real-time fault subgraph and each historical fault node in the fault knowledge graph; The query unit is used to query historical fault nodes and their adjacent nodes that match the real-time fault subgraph in the fault knowledge graph based on the feature similarity. The output unit is used to perform a weighted summation of the feature vector of the matching node and the feature vector of the neighboring node, and to perform a nonlinear transformation on the weighted summation result to output the matching probability corresponding to each fault type. The generation unit is used to generate corresponding business early warning data when the matching probability of any fault type exceeds a preset probability threshold.

[0047] In an optional implementation of this embodiment, the early warning module is further configured to: match the corresponding early warning title and risk description from a preset early warning text library based on the fault type and prediction data in the business early warning data, and fill the abnormal indicator value and predicted occurrence time in the prediction data into the corresponding position in the risk description to generate a pop-up prompt text; query the corresponding operation instruction information from a preset suggestion strategy library based on the fault type, wherein the operation instruction information contains at least one executable operation instruction or configuration modification guide; combine and encapsulate the pop-up prompt text and operation instruction information into early warning data, and send it to the receiving end through the communication bus.

[0048] In one optional implementation of this embodiment, the determining module is specifically used for: based on the warning signal and business warning data, traversing the forwarding records of each hop node in the data transmission path in reverse according to the link tracing identifier, locating the node that successfully forwarded the last data, and determining the next hop node of that node as the fault node; obtaining the real-time log of the fault node and the status snapshot within a preset time period before the fault occurred, extracting the abnormal event sequence from the real-time log, and extracting the abnormal time period of the indicator from the status snapshot; aligning the fault type in the business warning data with the abnormal event sequence and the abnormal time period of the indicator on the time axis, and determining the correlation confidence between the business warning data and the abnormal event sequence and the abnormal time period of the indicator; selecting the abnormal event with the highest correlation confidence as the root cause event, and generating root cause information according to the event type of the root cause event and the associated device parameters.

[0049] In one optional implementation of this embodiment, the recovery module is specifically used to: match the corresponding recovery action sequence from a preset recovery strategy library according to the fault type and root cause event type in the root cause information, wherein the recovery action sequence includes one or more basic recovery actions arranged in execution order; when there are multiple fault nodes, obtain the hierarchical relationship of each fault node in the device topology, assign priority values ​​to each fault node according to the hierarchical relationship, and merge the recovery action sequences of each fault node into an ordered task queue according to the priority values ​​from high to low; and issue recovery instructions to the fault nodes one by one through the soft bus instruction channel according to the order in the ordered task queue.

[0050] According to the proposed solution, an automated equipment fault recovery device generates business early warning data containing fault type information by combining early warning signals triggered by time-series prediction with fault knowledge graph feature matching. Based on the business early warning data, the device reversely queries the link tracing identifier to determine the fault node. This allows for accurate prediction before a fault occurs and recovery of the fault node based on the corresponding root cause information, preventing losses to the equipment after a fault occurs.

[0051] According to the scheme provided in this application Figure 5 An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the automated device fault recovery method described in the foregoing embodiments, and mainly includes: The system includes a memory 501, a processor 502, and a computer program 503 stored on the memory 501 and executable on the processor 502. The memory 501 and the processor 502 are connected via communication. When the processor 502 executes the computer program 503, it implements the automated device fault recovery method described in the foregoing embodiments. The number of processors can be one or more.

[0052] Memory 501 can be high-speed random access memory (RAM) or non-volatile memory, such as disk storage. Memory 501 is used to store executable program code, and processor 502 is coupled to memory 501.

[0053] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 5 The memory in the illustrated embodiment.

[0054] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the automated device failure recovery method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0055] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0056] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An automated equipment fault recovery method, characterized in that, include: Collect device operation metrics from both end-side and edge-side devices, and combine them with distributed system logs to generate real-time status data; Based on the real-time status data, the equipment operation indicators are predicted. When at least one predicted data exceeds a preset warning threshold, a warning signal is generated. The predicted data corresponding to the warning signal is then matched with the fault knowledge graph to generate business warning data. Based on the warning signal and the service warning data, the forwarding records of each hop node in the data transmission path are traversed in reverse according to the link tracing identifier to locate the last node that successfully forwarded the data, and the next hop node of that node is identified as the fault node. The real-time log of the fault node and the state snapshot within a preset time period before the fault occurred are obtained. An abnormal event sequence is extracted from the real-time log, and an abnormal time period of the indicator is extracted from the state snapshot. The fault type in the service warning data is aligned with the abnormal event sequence and the abnormal time period of the indicator on the time axis to determine the correlation confidence between the service warning data and the abnormal event sequence and the abnormal time period of the indicator. The abnormal event with the highest correlation confidence is selected as the root cause event, and root cause information is generated based on the event type of the root cause event and the associated device parameters. Based on the root cause information, a recovery strategy is matched from the strategy library, and a recovery command is sent to the faulty node according to the recovery strategy.

2. The automated equipment fault recovery method according to claim 1, characterized in that, Before the step of performing feature matching between the predicted data corresponding to the early warning signal and the fault knowledge graph to generate business early warning data, the method further includes: Obtain the abnormal indicator values ​​corresponding to the warning signal and the key log entries in the distributed system log; The abnormal indicator values ​​and the key log entries are mapped to graph nodes respectively, and edge connections are constructed based on the time correlation between the abnormal indicator values ​​and the key log entries to generate a real-time fault subgraph.

3. The automated equipment fault recovery method according to claim 2, characterized in that, The step of performing feature matching between the predicted data corresponding to the early warning signal and the fault knowledge graph to generate business early warning data includes: The real-time fault subgraph is aligned with the pre-constructed fault knowledge graph to determine the feature similarity between each node in the real-time fault subgraph and each historical fault node in the fault knowledge graph. Based on the feature similarity, query the historical fault nodes and their adjacent nodes that match the real-time fault subgraph in the fault knowledge graph; The feature vector of the matching node is weighted and summed with the feature vectors of the neighboring nodes, and the weighted sum is subjected to a nonlinear transformation to output the matching probability corresponding to each fault type. When the matching probability of any fault type exceeds a preset probability threshold, corresponding business warning data is generated.

4. The automated equipment fault recovery method according to claim 3, characterized in that, After the step of performing feature matching between the predicted data corresponding to the early warning signal and the fault knowledge graph to generate business early warning data, the method further includes: Based on the fault type and prediction data in the business early warning data, the corresponding early warning title and risk description are matched from the preset early warning text library, and the abnormal indicator value and predicted occurrence time in the prediction data are filled into the corresponding position in the risk description to generate a pop-up prompt text. Based on the fault type, query the corresponding operation instruction information from the preset suggestion strategy library. The operation instruction information includes at least one executable operation instruction or configuration modification guide. The pop-up notification text and the operation instruction information are combined and encapsulated into warning data, and then sent to the receiving end via the communication bus.

5. The automated equipment fault recovery method according to claim 1, characterized in that, The step of matching a recovery strategy from the strategy library based on the root cause information and sending a recovery command to the faulty node according to the recovery strategy includes: Based on the fault type and root cause event type in the root cause information, a corresponding recovery action sequence is matched from the preset recovery strategy library. The recovery action sequence includes one or more basic recovery actions arranged in execution order. When there are multiple fault nodes, obtain the hierarchical relationship of each fault node in the device topology, assign priority values ​​to each fault node according to the hierarchical relationship, and merge the recovery action sequence of each fault node into an ordered task queue according to the priority values ​​from high to low. According to the order in the ordered task queue, recovery commands are sent one by one to the faulty node through the soft bus command channel.

6. An automated equipment fault recovery device, characterized in that, The automated equipment fault recovery device is used to implement the automated equipment fault recovery method according to claim 1, and the automated equipment fault recovery device includes: The data acquisition module is used to collect the device operation indicators of the end-side devices and the edge-side devices, and combine them with the distributed system logs to generate real-time status data. The early warning module is used to predict the equipment operation indicators based on the real-time status data. When at least one predicted data exceeds a preset early warning threshold, an early warning signal is generated, and the predicted data corresponding to the early warning signal is matched with the fault knowledge graph to generate business early warning data. The determination module is used to locate the last node that successfully forwarded data, based on the warning signal and the service warning data, by traversing the forwarding records of each hop node in the data transmission path in reverse according to the link tracing identifier, and determining the next hop node of that node as the fault node. It then obtains the real-time logs of the fault node and a state snapshot within a preset time period before the fault occurred. The module extracts an abnormal event sequence from the real-time logs and an abnormal time period for indicators from the state snapshots. It aligns the fault type in the service warning data with the abnormal event sequence and the abnormal time period for indicators on the time axis to determine the correlation confidence between the service warning data and the abnormal event sequence and the abnormal time period for indicators. Finally, it selects the abnormal event with the highest correlation confidence as the root cause event and generates root cause information based on the event type of the root cause event and the associated device parameters. The recovery module is used to match a recovery strategy from the strategy library based on the root cause information, and send a recovery command to the faulty node according to the recovery strategy.

7. The automated equipment fault recovery device according to claim 6, characterized in that, The early warning module includes: The node alignment unit is used to align the real-time fault subgraph with the pre-constructed fault knowledge graph, and determine the feature similarity between each node in the real-time fault subgraph and each historical fault node in the fault knowledge graph. The query unit is used to query historical fault nodes and their adjacent nodes that match the real-time fault subgraph in the fault knowledge graph based on the feature similarity. The output unit is used to perform a weighted summation of the feature vector of the matching node and the feature vector of the neighboring node, and to perform a nonlinear transformation on the weighted summation result to output the matching probability corresponding to each fault type. The generation unit is used to generate corresponding business early warning data when the matching probability of any fault type exceeds a preset probability threshold.

8. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the automated equipment fault recovery method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the automated equipment fault recovery method according to any one of claims 1 to 6.

Images