Fault processing method, electronic equipment and storage medium

Abstract

The invention discloses a fault processing method, electronic equipment and a storage medium, relates to the technical field of computers, and constructs a training sample with a real tag in order to effectively improve the accuracy of fault detection. And training the local fault detection large model by using the training sample. And knowledge distillation is carried out on the trained local fault detection large model, so that a lightweight fault detection model is obtained. After the cloud server sends the fault detection model to the edge server, the edge server can perform fault detection on newly collected log data by using the fault detection model, so that the fault can be detected more accurately.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault handling method, an electronic device, and a storage medium. Background Art

[0002] Traditional data center operations and maintenance mainly rely on manual inspections and static rule-based alarm systems. However, manual inspections are inefficient and difficult to detect instantaneous faults in a timely manner. In addition, static rule-based alarm systems have high false alarm and missed alarm rates.

[0003] Currently, there are operation and maintenance solutions that combine edge computing and federated learning. However, there is still a problem that the single training data is insufficient to cover diverse fault scenarios, resulting in insufficient generalization ability of the model and difficulty in accurately detecting faults.

[0004] How to improve the accuracy of fault detection in data centers is a technical problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0005] This application provides a fault handling method, electronic equipment and storage medium. This application can form a closed-loop learning mechanism of detection-dispatching-feedback-optimization to continuously improve the accuracy of fault detection.

[0006] A fault handling method, applied to a cloud server, comprising: Receive fault log data reported by the edge server; the fault log data includes log data and fault detection results, the fault detection results are the results obtained by the edge server using the latest fault detection model to detect the log data; After sending a fault work order matching the fault log data to the operation and maintenance terminal, receiving the actual fault data fed back by the operation and maintenance terminal; Using the real fault data, the log data and the fault detection results, construct training samples with real labels; When the training conditions are met, the local fault detection model is trained using the training samples, and knowledge distillation is performed on the trained local fault detection model to obtain a new fault detection model; The new fault detection model is sent to the edge server so that the edge server performs fault detection on the newly collected log data using the latest fault detection model.

[0007] A fault handling method, applied to an edge server, comprising: Collect log data; Detect the log data using the latest local fault detection model to obtain a fault detection result; splicing the log data and the fault detection result to obtain fault log data; The fault log data is reported to the cloud server so that the cloud server executes the steps of the fault handling method as described above.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned fault handling methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned fault handling methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned fault handling methods when executed by a processor.

[0011] The cloud server in this application can receive fault log data reported by the edge server. The fault log data includes the log data collected by the edge server that implements edge computing, as well as the fault detection results obtained by the edge server after detection using a local fault detection model. The cloud server can send a fault work order that matches the fault log data to the operation and maintenance terminal. And receive real fault data fed back by the operation and maintenance terminal. In order to effectively improve the accuracy of fault detection, this application combines real fault data, log data and fault detection results to construct training samples with real labels. That is, the training samples can feedback real fault conditions. When the training conditions are met, the local fault detection model is trained using the training samples. In order to enable the edge server to carry the detection model, a lightweight fault detection model can be obtained by performing knowledge distillation on the trained local fault detection model. After the cloud server sends the fault detection model to the edge server, the edge server can use the fault detection model to perform fault detection on the newly collected log data.

[0012] In other words, this application constructs training samples with real labels that reflect the actual fault conditions by combining the feedback data corresponding to the fault work tickets. Based on these training samples, a large local fault detection model is trained on the cloud server. Through knowledge distillation, a lightweight fault detection model that can be deployed on edge servers is obtained, allowing the edge servers to perform fault detection and thus more accurately detect faults.

[0013] This application has the technical effect of forming a closed-loop learning mechanism of detection-dispatching-feedback-optimization, and continuously improving the accuracy of fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 A flowchart of a fault handling method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an intelligent operation and maintenance and work order coordination system based on federated learning and edge computing provided in an embodiment of the present application; Figure 3 A flowchart of another fault handling method provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 5 A schematic diagram of the specific structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0018] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0019] Please refer to Figure 1 , Figure 1 This is a flowchart of a fault handling method in an embodiment of the present application, which is applied to a cloud server. The cloud server can be Figure 2 The server corresponding to the cloud layer in the system shown.

[0020] For details, please refer to Figure 2 , Figure 2 This application provides an intelligent operation and maintenance and work order coordination system based on federated learning and edge computing. In this system, it is mainly divided into the cloud and the edge, where the cloud has a cloud server and the edge has an edge server. The edge servers are distributed in customer data rooms in various places, and can collect hardware logs and system logs of computing equipment, network equipment, storage equipment, power supply equipment and refrigeration equipment in the room in real time. A fault detection model can be run in the edge server, and the fault detection model can be used to perform fault detection on the collected log data, thereby obtaining fault detection results, and filtering out the data to be reported based on the fault detection results. The reported data includes fault detection results with a confidence level greater than a dynamic threshold and the corresponding log data. In this article, the data reported in this case is referred to as fault log data.

[0021] After receiving the fault log data, the cloud server can perform fault detection on the fault log data. If an anomaly is detected, a fault ticket is generated and dispatched to the operation and maintenance terminal. The operation and maintenance personnel associated with the operation and maintenance terminal can then process the fault ticket and provide feedback on the actual fault data.

[0022] The cloud server constructs training samples with real labels based on real fault data, log data and fault detection results, and uses the training samples to train the local fault detection model. After the training is completed, it is compressed into a lightweight fault detection model and distributed to the edge server, thereby updating the fault detection model in the edge server to improve the accuracy of fault detection.

[0023] Specifically, the troubleshooting method provided in this application includes the following steps.

[0024] S101: Receive fault log data reported by an edge server.

[0025] The fault log data includes log data and fault detection results. The fault detection results are the results obtained by the edge server using the latest fault detection model to detect the log data.

[0026] Edge servers can collect logs from devices within their computer rooms. Logs include, but are not limited to, hardware logs and system logs. The devices collected include, but are not limited to, computing devices, network devices, storage devices, power supply equipment, and cooling equipment.

[0027] When collecting logs, edge servers can set differentiated collection cycles based on device type and log characteristics. For example, they can collect BMC (Baseboard Management Controller) and accelerator card logs every 5 minutes, RAID (Redundant Array of Independent Disks) logs every hour, and SMART (Self-Monitoring, Analysis and Reporting Technology) logs daily. For network devices, hardware and system logs can be collected every 10 minutes using SNMP (Simple Network Management Protocol).

[0028] After collecting log data, the edge server can desensitize it, removing any information irrelevant to fault prediction, such as user privacy data. The edge server then uses the latest local fault detection model to perform fault detection on the log data, generating a fault detection result.

[0029] In order to reduce the pressure on the cloud, after obtaining the fault detection results, these data can be filtered according to the confidence level, and only the fault detection results with a confidence level greater than the dynamic threshold and their corresponding log data are selected and reported to the cloud server.

[0030] When reporting fault log data, the edge server can encrypt and upload the data. When uploading, it can also compress and upload it in blocks. Combined with the hash algorithm, the integrity of the blocks can be verified and the order of the blocks can be determined.

[0031] After the cloud server receives the fault log data, it can re-detect the fault in the fault log data. If an abnormality is detected, it indicates that a fault problem exists and needs to be processed. At this time, a corresponding fault work order can be generated.

[0032] In a specific embodiment of the present application, receiving fault log data reported by the edge server includes: using an encrypted channel with the edge server to obtain compressed blocks corresponding to the fault log data; using the ID corresponding to the compressed blocks, assembling the compressed blocks, and then decompressing them to obtain the fault log data. In other words, when the edge server reports the fault log data, it can protect the data by encryption, or it can combine compression technology to reduce the amount of data transmission. Specifically, Protocol Buffers can be used to define the log upload service interface, the protoc tool can be used to generate the gRPC (remote procedure call framework) client and server code, and OpenSSL can be used to generate the root certificate, cloud server certificate and edge server certificate. The cloud server certificate is deployed in the cloud, the edge server certificate is deployed at the edge, and the root certificate is pre-set at the edge and cloud for two-way authentication. TLS 1.3 (Transport Layer Security 1.3) is enabled on both the server and edge, using cipher suites such as TLS_AES_128_GCM_SHA256. Logs from module 2 with a confidence level greater than a threshold are compressed using Gzip (a data compression algorithm). The compressed logs are then chunked, with each chunk generating a unique chunk_id, which records the chunk's order within the complete log. The edge server records the chunk upload status in a database and queues pending upload chunks. These chunks are processed asynchronously by a separate thread pool to avoid blocking the main thread. Before each transfer, the database is queried, skipping successfully uploaded chunks. If a transfer is interrupted, the failed chunk is logged and retried after a delay. If uploads fail three times in a row, the transfer is paused and an alert is logged. Upon receiving the chunks, the cloud server verifies the hash value (comparing the chunk_id with the calculated value). If the verification fails, an error code is returned requesting retransmission. The complete log file is reassembled in the order of the sequence field. Upon completion of the reassembly, decompression (Gzip) and cloud-based inference are triggered.

[0033] S102: After sending the fault work order that matches the fault log data to the operation and maintenance terminal, receive the actual fault data fed back by the operation and maintenance terminal.

[0034] In this embodiment, the operation and maintenance terminal can be a mobile device bound to the operation and maintenance personnel. Fault tickets are sent to the operation and maintenance terminal so that the operation and maintenance personnel can process the fault tickets, that is, identify and troubleshoot the fault. During and after processing the fault ticket, the operation and maintenance personnel can report the actual fault situation to the cloud server. In this embodiment, the data / information corresponding to the actual fault situation reported by the operation and maintenance terminal is referred to as true fault data. True fault data may include data such as whether the fault is true (i.e., whether it is a false alarm), fault type, faulty device, and repair measures.

[0035] In a specific implementation of the present application, a fault work order that matches the fault log data is sent to an operation and maintenance terminal, including: using the latest local fault detection large model to perform anomaly detection on the fault log data; when an anomaly is detected, generating a fault work order corresponding to the anomaly; obtaining real-time status information and skill information of the operation and maintenance personnel; determining the adapted target operation and maintenance personnel in combination with the real-time status information and skill information; and sending the fault work order to an operation and maintenance terminal that has a binding relationship with the target operation and maintenance personnel.

[0036] Specifically, after receiving logs reported by each data center (i.e., fault log data reported by the edge server), the cloud server uses a local fault detection model to detect anomalies. When an anomaly is detected, a work order is generated. For example, a description template is matched from a preset template library based on the fault type and device type. For example, the template: Fault Description: {Fault Type} detected on {Device Model} device {Device Location} in the {Data Center Location}, Recommended Action: {Recommended Action}, Associated Skill Tag: {Skill Tag}. Then, the database is queried for the device's historical maintenance records and a work order template is generated, such as: {"Device ID": "Device 1","Fault Type": "RAID Card Abnormal","Fault Description": "A RAID card abnormality was detected on a Dell device in a certain data center","Recommended Action": "Please check the physical connection status of the RAID card. If no improvement is seen, replace the card","Skill Tag": ["Storage Device Repair", "RAID Configuration"],"Additional Information": {"Historical Maintenance Record": ["Record 1"]}}.

[0037] After a fault work order is generated, the database is checked for the existence of an unprocessed work order based on the fault type and device ID of the newly generated work order. If it already exists, it will not be reassigned. If it does not exist, the work order will be dynamically assigned based on the real-time status information (such as location, load) and technical information (such as the types of skills mastered in the notes) of the operation and maintenance personnel. Specifically, the formula Dynamically assign work orders. The higher the Score, the more suitable the operator is for handling the current work order. 1 / Distance: The reciprocal of the geographic distance between the operator's current location and the equipment room where the faulty device is located. The closer the distance, the larger the value and the higher the score. Skill_Match: The degree of match between the skill tags required for the work order and the operator's skill tags. 1 / Workload: The reciprocal of the operator's current workload. The fewer work orders, the larger the value and the higher the score.

[0038] θ, η, ξ are weight coefficients. The default values of θ, η, ξ are 0.5, 0.3, and 0.2, respectively. For example, for an emergency work order, θ=0.7, η=0.2, ξ=0.1; for a complex fault, θ=0.3, η=0.5, ξ=0.2.

[0039] Use the Haversine formula to calculate the straight-line distance between the GPS coordinates of the maintenance personnel and the GPS coordinates of the computer room. Calculate the Jaccard similarity between the skill tags required for the work order (such as ["storage device maintenance", "RAID configuration"]) and the maintenance personnel's skill tags (such as ["storage device maintenance", "network equipment maintenance"]): The reciprocal of the number of work orders currently assigned by the operator (e.g., 3) is used as the load score. The operator with the highest score is assigned a work order, and a push notification is sent via the mobile app (i.e., the operator's terminal). The work order status is updated in real time to the database (e.g., assigned, processing). When the number of work orders exceeds the system's processing capacity, they are temporarily stored in a buffer queue and assigned based on priority.

[0040] After processing the work order, the operation and maintenance personnel report the actual fault type, repair measures, whether it is a false alarm, and other information (i.e., real fault data) through the mobile terminal.

[0041] S103: Use real fault data, log data and fault detection results to construct training samples with real labels.

[0042] After obtaining the real fault data, we can use the real fault data, log data and fault detection results to construct training samples that can reflect the real fault situation.

[0043] Specifically, after collecting logs, the edge server records the input features and model outputs (logits or soft labels), encapsulating the logs into {input feature hash value, fault type logits, confidence level, timestamp}. After handling the fault ticket, the operations and maintenance personnel report the actual fault type, remediation measures, and whether it is a false alarm to the cloud via their mobile device. The cloud then associates the actual fault data with the original logits to generate training samples with true labels.

[0044] The cloud receives logits and associated feature hash values uploaded by each edge server, clusters logits by fault type and device type, and builds a global knowledge base to train the local fault detection model.

[0045] In one specific embodiment of the present application, the method further includes: obtaining false positive samples from training samples; adjusting a dynamic threshold using the false positive samples; and transmitting the dynamic threshold to an edge server so that the edge server only reports fault log data corresponding to fault detection results with a confidence level greater than the dynamic threshold. The false positive samples can be used to adjust the dynamic threshold corresponding to the fault detection model of the edge server.

[0046] S104. When the training conditions are met, the local fault detection model is trained using the training samples, and knowledge distillation is performed on the trained local fault detection model to obtain a new fault detection model.

[0047] In the present application, training conditions may be set, such as periodic cycles, or specific conditions for conditional triggering of training, such as at least one of a false alarm rate being lower than a false alarm threshold and a false alarm rate being higher than a false alarm threshold.

[0048] When the training conditions are met, the training samples can be used to train the large local fault detection model, enabling it to achieve more accurate fault detection accuracy. To enable the large local fault detection model to run on edge servers, in this embodiment, knowledge distillation can be performed on the large local fault detection model to obtain a new, lightweight fault detection model.

[0049] Among them, in a specific implementation of the present application, the local fault detection large model is trained using training samples, including: obtaining the divergence of the training samples; if the divergence exceeds the divergence threshold, adding a distribution alignment term to the loss function to enhance the loss function; using the training samples and combining the enhanced loss function to train the local fault detection large model.

[0050] Regarding the loss function, cross entropy loss (based on the hard labels of the work order feedback): ,in, : Cross entropy loss value, which measures the difference between the cloud model prediction result and the actual fault label. The smaller the value, the more accurate the model prediction.

[0051] : The one-hot encoding of the true label, indicating that the sample belongs to the true label of the i-th type of fault.

[0052] : Model prediction probability, the cloud model predicts the probability value of the sample belonging to the i-th type of fault, the range is [0,1] and satisfies .

[0053] is the logarithmic probability, the predicted probability Take the natural logarithm, when When it is close to the true label, Losses are reduced.

[0054] KL divergence loss (soft labels based on marginal model logits): ;in, Divergence loss value, which measures the prediction distribution of the cloud model Soft label distribution with marginal model difference.

[0055] : Sum all fault categories, C = total number of fault types.

[0056] : The soft label probability output by the edge model, which represents the predicted probability of the edge model for the i-th type of fault.

[0057] : Cloud model prediction probability, cloud prediction probability for the same sample.

[0058] : Probability distribution difference measure that penalizes the situation where the cloud model distribution deviates from the edge model distribution.

[0059] Total loss: (ρ is dynamically adjusted with the training rounds, initially ρ=0.3, later ρ=0.7), where Ltotal: the total mixed loss value, the final optimization target of cloud model training. Lhard: hard label loss, equivalent to the cross entropy loss L CE Lsoft: soft label loss, equivalent to KL divergence loss L KL ρ: hard label loss weight; 1-ρ: soft label loss weight.

[0060] The clustered logits can be used as soft labels to guide BERT-Large (Bidirectional EncoderRepresentations from Transformers, a natural language processing model, corresponding to the large local fault detection model) to learn the knowledge of the edge model. Combined with the actual fault labels of the work order feedback (such as the repair results reported by the operation and maintenance personnel), supervised training is performed. FedAvg aggregation is performed on the first few layers of parameters of each edge model regularly (such as weekly), the cloud shared layer parameters are updated, and the shared layer parameters are sent to the edge. After the operation and maintenance personnel handle the work order, they report the actual fault type, repair measures, whether it is a false alarm, and other information through the mobile terminal. The cloud will associate the feedback data with the original logits to generate training samples with real labels. For false alarm samples: adjust the edge model classification threshold; for missed samples: increase the loss weight of the category during cloud training. Trigger global model retraining regularly (for example, weekly) and calculate the Jensen-Shannon divergence (JSD): , ;in, : Jensen-Shannon divergence of the high / low confidence sample distribution, which measures the degree of difference in the probability distribution of the two types of samples. The larger the value, the more significant the distribution difference. The probability distribution of fault categories for high-confidence samples. When the edge model infers local logs, the probability distribution of fault categories for samples whose confidence scores exceed the dynamic threshold. : The probability distribution of fault categories for low-confidence samples, the probability distribution of fault categories for samples whose edge model confidence scores are less than the dynamic threshold. M: The intermediate distribution between high / low confidence distributions, and The arithmetic mean distribution of .

[0061] The KL divergence of the distribution relative to M measures the degree of difference between the high confidence distribution and the intermediate distribution.

[0062] The KL divergence of the distribution relative to M measures how different the low confidence distribution is from the intermediate distribution.

[0063] is the weight coefficient, and the two KL divergences are averaged to ensure the symmetry of JSD.

[0064] When the JSD exceeds a threshold (e.g., 0.3), distribution alignment is triggered.

[0065] Loss function enhancement, that is, in the original On this basis, add distribution alignment items: ;in: is the distribution alignment loss value; is the weight coefficient of MMD loss; The weight coefficient of loss; Maximum mean difference, which measures the distribution distance of high / low confidence samples in the feature space; is the attention cosine similarity, which measures the consistency of the model's attention focus on high / low confidence samples; MMD (maximum mean difference) measures the difference in feature space distribution of high / low confidence samples, and is calculated as follows: Where Xhigh is the set of high-confidence samples, i.e., samples whose edge model confidence is greater than the threshold. Xlow is the set of low-confidence samples, i.e., samples whose edge model confidence is less than or equal to the threshold. Nhigh and Nlow are the number of samples, Nhigh = m and Nlow = n. Kernel function mapping, maps the input features to the Reproducing Kernel Hilbert Space (RKHS). : Mean embedding of high confidence samples in RKHS. : Mean embedding of low confidence samples in RKHS. : Hilbert space norm, calculates the distance between two mean embeddings.

[0066] CosSim (cosine similarity) calculates the attention similarity of BERT-Large for high / low confidence samples ;in, : The attention feature vector of the sample, extracted from the cloud model; Vector dot product, which measures directional similarity; Norm, normalization factor, constrains the output range. Weight coefficients such as λ=0.2 and μ=0.1.

[0067] Final loss function: .

[0068] Distilling knowledge from the large local fault detection model to create a lightweight model involves using the updated BERT-Large model as a teacher model and re-distilling it to generate a lightweight edge model. For example, a progressive distillation approach can be used: in the first round, distilling to BERT-Medium (12 layers) preserves more teacher knowledge, and in the second round, further compressing to BERT-Tiny (6 layers) for edge servers.

[0069] S105: Send the new fault detection model to the edge server, so that the edge server can use the latest fault detection model to perform fault detection on the newly collected log data.

[0070] In a specific implementation of the present application, the teacher model parameters on the cloud side are compared with the current model parameters on the edge side, the changed parameters are extracted, the difference data is compressed using Gzip, and the transmission protocol is defined in combination with Protocol Buffers and sent to the edge server. In other words, when sending the fault detection model to the edge server, there is no need to send the complete fault detection model. It is only necessary to compare the latest fault detection model currently corresponding to the edge server with the newly obtained fault detection model on the cloud side (i.e., the teacher model parameters on the cloud side) to determine the changed parameters. The changed parameters are compressed and sent to the edge server to reduce the amount of data transmission, reduce the occupation of system resources, and ensure the effective operation of the business. The edge server adjusts the local fault detection model based on these changed parameters to complete the update of the fault detection model, so that the edge server can use the latest fault detection model to perform fault detection on the newly collected log data.

[0071] The cloud server in this application can receive fault log data reported by the edge server. The fault log data includes the log data collected by the edge server that implements edge computing, as well as the fault detection results obtained by the edge server after detection using a local fault detection model. The cloud server can send a fault work order that matches the fault log data to the operation and maintenance terminal. And receive real fault data fed back by the operation and maintenance terminal. In order to effectively improve the accuracy of fault detection, this application combines real fault data, log data and fault detection results to construct training samples with real labels. That is, the training samples can feedback real fault conditions. When the training conditions are met, the local fault detection model is trained using the training samples. In order to enable the edge server to carry the detection model, a lightweight fault detection model can be obtained by performing knowledge distillation on the trained local fault detection model. After the cloud server sends the fault detection model to the edge server, the edge server can use the fault detection model to perform fault detection on the newly collected log data.

[0072] In other words, this application constructs training samples with real labels that reflect the actual fault conditions by combining the feedback data corresponding to the fault work tickets. Based on these training samples, a large local fault detection model is trained on the cloud server. Through knowledge distillation, a lightweight fault detection model that can be deployed on edge servers is obtained, allowing the edge servers to perform fault detection and thus more accurately detect faults.

[0073] Please refer to Figure 3 , Figure 3 This is a flowchart of another fault handling method in the embodiment of the present application, which can be applied to Figure 2 The edge server in the edge layer of the system shown in the figure comprises the following steps.

[0074] S201: Collect log data.

[0075] Log data refers to the logs corresponding to several devices monitored by the edge server, including but not limited to hardware logs and system logs.

[0076] In this embodiment, different collection frequencies may be used for collection of different types of logs.

[0077] In one specific embodiment of the present application, collecting log data includes: collecting hardware logs and system logs from target devices within the computer room where the edge server is located; searching for content to be hidden in the hardware logs and system logs using a business-sensitive vocabulary and a matching algorithm; replacing the content to be hidden in the hardware logs and system logs; and formatting the hardware logs and system logs to obtain log data. The content to be hidden can be data requiring privacy protection and unrelated to fault prediction.

[0078] Edge servers are deployed in customer data centers across various locations to collect real-time hardware and system logs from computing devices, network devices, storage devices, power supply equipment, and cooling equipment within the data centers. Differentiated collection cycles are set based on device type and log characteristics (for example, servers collect BMC and accelerator logs every 5 minutes, RAID logs every hour, and SMART logs daily; network devices collect hardware and system logs every 10 minutes via SNMP). Logs required for fault prediction generally do not involve customer applications or business data. To ensure customer data security, collected logs are filtered for sensitive content using a multi-pattern matching algorithm (Aho-Corasick) based on a pre-set database of business-sensitive terms (such as customer names, internal database table names, and business paths).

[0079] Specifically, the keywords are broken down into characters to construct a dictionary tree (Trie tree). The breadth-first search (BFS) is used to traverse the dictionary tree to calculate the failure pointer for each node. The log content is traversed for matching. For the matched sensitive content (that is, a specific type of content to be hidden), the starting position start and ending position end of the sensitive word in the text are obtained. The characters at the start-1 and end+1 positions are checked to see if they are boundary characters (such as spaces, punctuation marks, and special symbols). If the boundary conditions are met, it is marked as a valid match, the valid sensitive content is replaced, and the filtered log is formatted.

[0080] The log format can be as follows: { "device_id": "Server_01", "log_type": "BMC SEL", "timestamp": "2023-10-01 14:30:00", "log_detail": "3ef9 | Processor #0xff | IERR | Asserted " }.

[0081] The corresponding meaning of the log structure is as follows: { "Device ID": "Server_01", "Log Type": "BMC System Event Log", "Timestamp": "2023-10-01 14:30:00", "Log Details": "3ef9 | Processor #0xff | IERR (Internal Error) | Asserted"}.

[0082] In actual applications, log collection and processing can be separated into independent threads and executed in batches through asynchronous thread pools to improve efficiency.

[0083] S202: Use the latest local fault detection model to detect log data and obtain a fault detection result.

[0084] A fault detection model is deployed in the edge server. After being trained for the cloud server, the pre-trained BERT-Large model is converted into a lightweight model through knowledge distillation and deployed on the edge server. When device log information is collected, the log information is inferred to detect anomalies.

[0085] S203: Combine the log data and the fault detection result to obtain fault log data.

[0086] Log data and fault detection results are combined to obtain fault log data. This fault log data contains not only log data but also fault detection results.

[0087] In a specific embodiment of the present application, log data and fault detection results are spliced to obtain fault log data, including: if the confidence of the fault detection result is greater than a dynamic threshold, the log data and the fault detection result are spliced to obtain fault log data; wherein, the process of determining the dynamic threshold includes: obtaining local omission rate, false alarm rate and alarm statistics, and using the omission rate, false alarm rate and alarm statistics to adjust the old threshold to obtain a new threshold; or, in the case of receiving a dynamic threshold sent by a cloud server, determining the received dynamic threshold as the new threshold.

[0088] When the confidence level of the fault detection model output exceeds the dynamic threshold, the log data and inference results are uploaded to the cloud for collaborative processing; otherwise, an alarm notification is pushed to the computer room contact through the message queue, and the feature hash is uploaded to the cloud.

[0089] Regarding the generation of dynamic thresholds: Dynamic thresholds are generated based on device type, fault severity (fatal / critical / warning / general), and criticality (core / critical / important / general). The initial value is calculated by adding a baseline value (70%) to the criticality modifier (-10% for core devices, +5% for general devices) and a fault severity modifier (-10% for fatal, +5% for general). The calculation results are used to create a mapping table. Every 24 hours, the dynamic threshold is adjusted based on the false positive rate (FP rate) and false negative rate (FN rate) according to the following threshold adjustment formula: Where FP is the number of false positives and FP Rate is the false positive rate; FN is the number of false negatives and FN Rate is the false negative rate. The target FP Rate and target FN Rate are preset ideal false positive and false negative rates, respectively. They are used to adjust dynamic thresholds to achieve a balance between false positives and false negatives in the system.

[0090] FP and FN are used to calculate the actual indicators FP Rate and FN Rate. These actual values are then compared with the target values to generate threshold adjustment signals.

[0091] Adaptive optimization is performed based on dynamic thresholds, where A is the total number of logs for which the model triggers alarms each day (i.e., the number of logs with a confidence level greater than or equal to the threshold), B is the total number of logs for which the model does not trigger alarms each day (i.e., the number of logs with a confidence level less than or equal to the threshold), γ is the proportion of actual faults among the logs that do not trigger alarms (default γ = 5%), and the adjustment coefficients α and β control the sensitivity to false positives and false negatives, respectively (default α = 0.3 and β = 0.5). A threshold range is set (e.g., 50% to 80%) to prevent over-adjustment.

[0092] S204, reporting the fault log data to the cloud server so that the cloud server can execute the following Figure 1 The steps of the described troubleshooting method.

[0093] After collecting logs, the edge server records the log's input features and model output (logits or soft labels), encapsulates the log's content into a package consisting of {input feature hash value, fault type logits, confidence level, and timestamp}, and uploads it. Sparse coding is used to compress the logits (for example, retaining the top-3 probability values) to reduce transmission volume, and the logits are uploaded to the cloud via a gRPC+TLS encrypted channel. The cloud receives the logits and associated feature hash values uploaded by each edge server, clusters the logits by fault type and device type, and builds a global knowledge base.

[0094] The cloud server pre-trains BERT-Large using historical global fault log data (such as public datasets or early desensitized data) as the initial teacher model. The training objective is fault classification (e.g., fault type identification). Knowledge distillation is performed on BERT-Large to generate a lightweight edge server model. The cloud and edge models are required to maintain consistent structure in the first four layers (word embeddings + first three Transformer layers), while subsequent layers remain independent. The edge server fine-tunes the lightweight model using locally collected logs. The training objective is to minimize the local fault detection loss function (e.g., cross-entropy loss) and update the model parameters. The training frequency is adjusted based on the device log collection cycle, for example, triggering fine-tuning hourly for high-frequency logs and daily for low-frequency logs.

[0095] Among them, fine-tuning refers to conducting small-scale training for fault detection tasks based on the fault detection model sent by the cloud server, making slight adjustments to the fault detection model parameters, and finally obtaining a fault detection model that is adapted to the fault detection task and data.

[0096] It should be noted that the technical content and technical effects related to the fault handling method provided in this embodiment are Figure 1 The corresponding troubleshooting methods can be referenced by each other and will not be described in detail here.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0098] Corresponding to the above method embodiment, an embodiment of the present application further provides an electronic device. The electronic device described below and the fault handling method described above can refer to each other.

[0099] See also Figure 4 As shown, the electronic device includes: Memory 332, for storing computer programs; The processor 322 is configured to implement the steps of the fault handling method of the above method embodiment when executing a computer program.

[0100] For details, please refer to Figure 5 , Figure 5 This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. This electronic device may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) (for example, one or more processors) and memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 may be temporary storage or permanent storage. The program stored in the memory 332 may include one or more modules (not shown), each of which may include a series of instruction operations in the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 to execute the series of instruction operations in the memory 332 on the electronic device 301.

[0101] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input / output interfaces 358 , and / or one or more operating systems 341 .

[0102] The steps in the fault handling method described above can be implemented by the structure of the electronic device.

[0103] Specifically, the electronic device may be a server. When the server is deployed at the edge layer, the server may implement the following Figure 1 When the server is deployed in the cloud layer, the server can implement the following steps: Figure 3 The troubleshooting steps are shown.

[0104] At the same time, there are edge servers deployed in the edge layer and cloud servers deployed in the cloud layer, and corresponding fault handling methods are implemented respectively, which is Figure 2 The system shown.

[0105] That is to say, in Figure 2 In the system shown in FIG. 1 , the edge server may implement the following Figure 3 At the same time, the following steps are implemented in the cloud server: Figure 1The steps of the fault handling method shown in the figure are shown. This improves the real-time and accuracy of fault detection. By deploying a lightweight fault detection model on edge nodes and combining it with a dynamic threshold optimization algorithm, local real-time anomaly detection is achieved, significantly reducing the latency of traditional cloud-based centralized processing. Furthermore, knowledge distillation technology is used to compress large cloud-based models into lightweight models suitable for edge devices, ensuring detection accuracy while reducing computing resource consumption, effectively addressing the need for rapid response to transient faults. Enhanced data security and privacy protection: Multi-pattern matching algorithms (such as Aho-Corasick) combined with boundary character validation accurately identify and desensitize sensitive content in logs, avoiding the misjudgment or omission problems of traditional regular expression matching. Furthermore, block transmission, asynchronous retry, and optimized encrypted communication significantly reduce the computing and communication overhead of edge devices while ensuring data transmission security, ensuring efficient system operation in resource-constrained environments. Collaborative learning and model generalization optimization are achieved across data centers. Through a federated learning framework, the cloud aggregates feature hashes and model outputs (logits) uploaded by each edge node. A hybrid loss function (cross entropy + KL divergence + distribution alignment) is then used to dynamically optimize the global model, significantly improving its generalization capabilities. Sparse coding compression technology is also used to reduce data transmission, enabling efficient knowledge sharing and collaborative evolution while protecting edge node data privacy. Optimizing work order assignment strategies: Based on a dynamic scoring algorithm, the system integrates the real-time location, skill matching, and workload of maintenance personnel to accurately assign work orders, significantly reducing mean time to repair (MTTR) and improving customer satisfaction. A closed-loop learning mechanism is implemented to continuously optimize the system. Feedback from work order processing results (such as false positive / false negative flags) is used to dynamically adjust edge model thresholds and increase loss weighting during cloud-based training. This creates a closed-loop learning mechanism of detection-assignment-feedback-optimization. This mechanism continuously improves model accuracy, prevents recurrence of similar failures, and ensures system self-improvement and long-term stability.

[0106] Corresponding to the above method embodiments, embodiments of the present application further provide a readable storage medium. The readable storage medium described below and the fault handling method described above can be referenced in correspondence with each other. Embodiments of the present application further provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above fault handling method embodiments when executed.

[0107] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0108] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned fault handling method embodiments are implemented.

[0109] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned fault handling method embodiments are implemented.

[0110] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art may make several improvements and modifications to this application without departing from the principles of this application, and such improvements and modifications also fall within the scope of protection of this application.

Claims

1. A fault handling method, characterized in that: Applied to cloud servers, including: Receive fault log data reported by the edge server; the fault log data includes log data and fault detection results, the fault detection results are the results obtained by the edge server using the latest fault detection model to detect the log data; After sending a fault work order matching the fault log data to the operation and maintenance terminal, receiving the actual fault data fed back by the operation and maintenance terminal; Using the real fault data, the log data and the fault detection results, construct training samples with real labels; When the training conditions are met, the local fault detection model is trained using the training samples, and knowledge distillation is performed on the trained local fault detection model to obtain a new fault detection model; The new fault detection model is sent to the edge server so that the edge server performs fault detection on the newly collected log data using the latest fault detection model.

2. The method according to claim 1, characterized in that Sending a fault work order that matches the fault log data to the operation and maintenance terminal includes: Performing anomaly detection on the fault log data using the latest local fault detection model; When an anomaly is detected, a troubleshooting ticket corresponding to the anomaly is generated; Obtain real-time status and skill information of operation and maintenance personnel; Determining a suitable target operation and maintenance personnel based on the real-time status information and the skill information; The fault work order is sent to an operation and maintenance terminal that has a binding relationship with the target operation and maintenance personnel.

3. The method according to claim 1, characterized in that The local fault detection model is trained using the training samples, including: Obtaining the divergence of the training sample; If the divergence exceeds a divergence threshold, adding a distribution alignment term to the loss function to enhance the loss function; The local fault detection large model is trained using the training samples in combination with the enhanced loss function.

4. The method according to claim 1, wherein Receive fault log data reported by edge servers, including: utilizing an encrypted channel with the edge server to obtain compressed blocks corresponding to the fault log data; The compressed blocks are assembled using the IDs corresponding to the compressed blocks, and then decompressed to obtain the fault log data.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Obtaining false positive samples from the training samples; adjusting a dynamic threshold using the false positive sample; The dynamic threshold is sent to the edge server so that the edge server only reports fault log data corresponding to fault detection results with a confidence level greater than the dynamic threshold.

6. A fault handling method, characterized in that: Applicable to edge servers, including: Collect log data; Detect the log data using the latest local fault detection model to obtain a fault detection result; splicing the log data and the fault detection result to obtain fault log data; The fault log data is reported to a cloud server so that the cloud server executes the steps of the fault handling method according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that Collect log data, including: Collecting hardware logs and system logs of target devices in the computer room where the edge server is located; Combining a business-sensitive vocabulary and a matching algorithm, searching for content to be hidden in the hardware log and the system log; Replacing the content to be hidden in the hardware log and the system log; The hardware log and the system log are formatted to obtain the log data.

8. The method according to claim 6 or 7, characterized in that The log data and the fault detection result are combined to obtain fault log data, including: If the confidence level of the fault detection result is greater than a dynamic threshold, concatenating the log data and the fault detection result to obtain the fault log data; The process of determining the dynamic threshold includes: Obtaining local missed alarm rate, false alarm rate, and alarm statistics, and adjusting the old threshold using the missed alarm rate, the false alarm rate, and the alarm statistics to obtain a new threshold; Or, in the case of receiving a dynamic threshold sent by the cloud server, the received dynamic threshold is determined as the new threshold.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the fault handling method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the fault handling method according to any one of claims 1 to 8.

Images