Server fault detection methods, devices, systems, and baseboard management controllers

Abstract

This invention discloses a server fault detection method, device, system, and baseboard management controller, relating to the field of server technology. The method, applied to a baseboard management controller, includes: collecting status data of various components in the server to be monitored; using a locally trained model for each local detection service to perform fault detection on the status data of each local detection service, obtaining fault detection results for each local detection service, and sending the fault detection results to a remote server; and sending the status data of each peer detection service to a low-latency server. This invention enables fault identification of privacy-related and time-sensitive data-related services on the baseboard management controller (BMC), improving the timeliness, reliability, and security of fault warnings. Furthermore, by utilizing edge servers or cloud servers with relatively low communication latency for fault prediction of corresponding services, it can meet the fault warning requirements of services with high real-time requirements and large computational demands.

Description

Technical Field

[0001] This invention relates to the field of server technology, and in particular to a server fault detection method, apparatus, system, baseboard management controller, and computer-readable storage medium. Background Technology

[0002] As the underlying hardware infrastructure architecture in fields such as the Internet and cloud computing, servers support critical business operations and facilitate the iterative optimization of various technologies. This inevitably places exceptional demands on overall performance, maintainability standards far exceeding those of home PCs, and operational stability requirements far more stringent. Therefore, general-purpose servers need to possess high performance, high availability, and high reliability. Even though servers undergo early data simulations, multiple PCB (printed circuit board) verifications, and rigorous testing under various extreme scenarios during the development phase, absolute reliability cannot be guaranteed. To ensure effective server management and timely fault diagnosis during operation, this relies on a key component for managing and monitoring servers: the Baseboard Management Controller (BMC).

[0003] In related technologies, traditional BMC management systems monitor and trigger alarms on real-time collected data through a remote monitoring and management server; for example... Figure 1 As shown, the BMC firmware is only responsible for status data collection. A monitoring and management module is deployed on the remote server to check the collected and reported status data in real time and to monitor and alarm for abnormal states that exceed preset thresholds. However, as the number of servers increases, a single remote server can hardly handle the large amount of status data and fault detection operations. In some solutions, the BMC device collects status data and reports it separately to edge servers and cloud servers for fault detection to reduce the workload of the remote server. However, with the continuous and rapid growth in the number of servers that need to be monitored, the data reporting by the BMC device in these solutions increases the computing and network transmission pressure on the edge servers and cloud servers, resulting in serious fault warning latency. This fails to meet the fault warning requirements of services with high real-time requirements and large computational loads, and the data reporting also poses a certain risk of privacy leakage.

[0004] Therefore, how to provide low-latency server fault monitoring and early warning services, and improve the reliability and security of fault early warning, is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a server fault detection method, apparatus, system, baseboard management controller, and computer-readable storage medium to provide low-latency server fault monitoring and early warning services, thereby improving the reliability and security of fault early warning.

[0006] To address the aforementioned technical problems, this invention provides a server fault detection method, applied to a baseboard management controller, comprising:

[0007] Collect status data of each component in the server to be monitored;

[0008] The system utilizes the local trained models of each local detection service to perform fault detection on the status data of each local detection service, obtains the fault detection results of each local detection service, and sends the fault detection results to a remote server; wherein, the local detection services include privacy data services and / or time-sensitive services;

[0009] The status data of each peer detection service is sent to a low-latency server, so that the trained models of each peer detection service in the low-latency server can be used to perform fault detection on the status data of each peer detection service, and the fault detection results of each peer detection service are sent to the remote server; wherein, the low-latency server is a part of the edge server and cloud server that are communicatively connected to the baseboard management controller, and the communication latency between the low-latency server and the baseboard management controller in the edge server and the cloud server is less than that of other servers.

[0010] On the other hand, sending the fault detection result to the remote server includes:

[0011] Based on the confidence level of the fault detection results of each of the aforementioned local detection services, local feedback services are detected within the local detection services; wherein, the local feedback services include local detection services whose confidence level of the fault detection results is greater than the corresponding confidence level threshold.

[0012] The fault detection results of the local feedback service are sent to the remote server; wherein, the remote detection service includes all the services to be detected on the server to be monitored, excluding the local feedback service.

[0013] On the other hand, the step of using the local trained model of each local detection service to perform fault detection on the status data of each local feedback service, obtaining the fault detection result of each local detection service, and sending the fault detection result to the remote server includes:

[0014] Using the local trained models of each of the services to be tested, fault detection is performed on the status data of each service to be tested, and the fault detection results of each service to be tested are obtained.

[0015] If the service to be detected is the privacy data service or the time-sensitive service, then determine whether the confidence level of the fault detection result of the service to be detected is greater than the first confidence level threshold; wherein, the service to be detected is any of the services to be detected.

[0016] If the confidence level is greater than the first confidence threshold, then the current service to be detected is determined to be a local feedback service.

[0017] If the confidence level is not greater than the first confidence threshold, then the current service to be detected is determined to be the peer detection service;

[0018] If the service to be detected is not the privacy data service or the time-sensitive service, then determine whether the confidence level of the fault detection result of the service to be detected is greater than the second confidence threshold; wherein, the second confidence threshold is greater than the first confidence threshold.

[0019] If the confidence level is greater than the second confidence threshold, then the current service to be detected is determined to be the local feedback service.

[0020] If the value is not greater than the second confidence threshold, then the current service to be detected is determined to be the peer detection service;

[0021] The fault detection results of the local feedback service in all the services to be tested are sent to the remote server.

[0022] On the other hand, both the local trained model and the trained model in the edge server are models that were trained by the cloud server and then distributed.

[0023] On the other hand, before using the local trained model of each local detection service to perform fault detection on the status data of each local detection service and obtain the fault detection result of each local detection service, the method further includes:

[0024] The cloud server receives the initial trained model corresponding to the current local detection service; wherein the current local detection service is any of the local trained models, and the initial trained model is a model trained by the cloud server using sample data from at least two service types.

[0025] Using the sample data from the current local detection service, the initially trained model is transferred to obtain the local trained model for the current local detection service.

[0026] On the other hand, the method also includes:

[0027] Receive a model adjustment instruction sent by the cloud server; wherein, the model adjustment instruction includes the neuron parameters to be adjusted of the target local trained model, and the neuron parameters to be adjusted are the neuron parameters whose change is greater than the update threshold, determined by the cloud server based on the fault detection results and user feedback information collected by the remote server;

[0028] Based on the neuron parameters to be adjusted, the corresponding neuron parameters in the target local trained model are adjusted.

[0029] The present invention also provides a server fault detection device, applied to a baseboard management controller, comprising:

[0030] The data acquisition module is used to collect status data of various components in the server to be monitored.

[0031] The local detection module is used to perform fault detection on the status data of each local detection service using the local trained model of each local detection service, obtain the fault detection result of each local detection service, and send the fault detection result to the remote server; wherein, the local detection service includes privacy data service and / or time-sensitive service;

[0032] A transmission detection module is used to send the status data of each peer detection service to a low-latency server, so as to use the trained models of each peer detection service in the low-latency server to perform fault detection on the status data of each peer detection service, and send the detected fault detection results of each peer detection service to the remote server; wherein, the low-latency server is a portion of the edge server and cloud server that are communicatively connected to the baseboard management controller, and the communication latency between the low-latency server and the baseboard management controller in the edge server and the cloud server is lower than that of the other servers.

[0033] The present invention also provides a baseboard management controller, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor, used to execute the computer program to implement the steps of the server fault detection method as described above.

[0036] The present invention also provides a server fault detection system, comprising: an edge server, a cloud server, and a baseboard management controller as described above;

[0037] The baseboard management controller is communicatively connected to the edge server and the cloud server. The edge server and the cloud server are each used to perform fault detection on the status data of each peer detection service sent by the baseboard management controller using their respective trained models of peer detection services, and send the detected fault detection results of each peer detection service to the remote server.

[0038] In addition, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the server fault detection method as described above.

[0039] This invention provides a server fault detection method applied to a baseboard management controller, comprising: collecting status data of each component in the server to be monitored; performing fault detection on the status data of each local detection service using a local trained model of each local detection service, obtaining fault detection results for each local detection service, and sending the fault detection results to a remote server; wherein the local detection services include privacy data services and / or time-sensitive services; sending the status data of each peer detection service to a low-latency server, so as to perform fault detection on the status data of each peer detection service using a trained model of each peer detection service in the low-latency server, and sending the obtained fault detection results of each peer detection service to a remote server; wherein the low-latency server is a portion of the edge servers and cloud servers that are communicatively connected to the baseboard management controller, and the communication latency between the low-latency server and the baseboard management controller in the edge servers and cloud servers is lower than that of other servers.

[0040] As can be seen, this invention utilizes the trained models of each local detection service to perform fault detection on the status data of each local detection service, obtaining the fault detection results for each local detection service, and sending the fault detection results to a remote server. This enables fault identification of services related to privacy data and time-sensitive data on the BMC, minimizing the leakage of privacy information on the server and improving the efficiency of fault identification in time-sensitive services, thereby enhancing the timeliness, reliability, and security of fault warnings. Furthermore, by sending the status data of each peer detection service to a low-latency server, and utilizing edge servers or cloud servers with lower communication latency for fault prediction of the corresponding services, this invention can meet the fault warning requirements of services with high real-time requirements and computational demands. In addition, this invention also provides a server fault detection device, system, baseboard management controller, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of a traditional BMC management system;

[0043] Figure 2 A flowchart illustrating a server fault detection method provided in an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the system architecture of a server fault detection method provided in an embodiment of the present invention;

[0045] Figure 4 A structural block diagram of a server fault detection device provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of a baseboard management controller provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of a server fault detection system provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please refer to Figure 2 , Figure 2 A flowchart illustrating a server fault detection method provided in an embodiment of the present invention. This method, applied to a Baseboard Management Controller (BMC), may include:

[0050] Step 101: Collect status data of each component in the server to be monitored.

[0051] It is understood that the server to be monitored in this embodiment may be a server that needs to use BMC to detect and collect status data of each component for fault alarm, such as a data center server. In this embodiment, the BMC in the server to be monitored can monitor and collect status data of each component that needs to be monitored in the server to be monitored.

[0052] Correspondingly, such as Figure 3 As shown, in this embodiment, the BMC (BMC firmware) can communicate with edge servers, cloud servers, and remote servers. For example, the data center server monitored by the BMC can be networked with edge servers, cloud servers, and remote servers respectively, and the edge servers and cloud servers can be networked with remote servers respectively to establish communication connections for data transmission. Each BMC in this embodiment can be a SOC (System on Chip) system, divided into two levels: BMC chip and BMC firmware. The BMC can be embedded in the motherboard of the data center server (i.e., the server to be monitored). With the rise of the CPU (Central Processing Unit) + GPU (Graphics Processing Unit) + DPU (Data Processing Unit) cloud computing architecture in recent years, the embedding of BMC can also be extended from the motherboard level to the component level. A single server platform can be configured with multiple BMCs, that is, in the network, multiple BMCs can be distributed and parallel to collect the status data of each component of the server to be monitored.

[0053] Accordingly, the specific quantity and type of status data collected by the BMC in this embodiment can be set by the designer according to the practical scenario and user needs. For example, when the server to be monitored is a data center server, the status data collected by the BMC can include all hardware information and operating system (OS) level information of the data center server. For example, hardware information can include the voltage, temperature, fan speed and power status of each component. In order to provide timely warning of the failure of the core components of the server, so as to ensure the normal operation of the core components of the server and reduce the losses caused by the failure of the core components of the server, the voltage, temperature and fan speed of the above-mentioned components can at least include the motherboard voltage, power status, chassis temperature and chassis fan speed of the data center server. Among them, the motherboard voltage can include the power input voltage, the core chip input voltage and the standby voltage, and the chassis temperature can include the inlet temperature, the outlet temperature and the core chip temperature.

[0054] Step 102: Utilize the local trained models of each local detection service to perform fault detection on the status data of each local detection service, obtain the fault detection results of each local detection service, and send the fault detection results to the remote server; wherein, the local detection services include privacy data services and / or time-sensitive services.

[0055] Understandably, due to the continuous and rapid growth in the number of servers connected to the network, the number of servers and components that BMC needs to monitor is also increasing rapidly. The amount of data obtained by BMC after collecting status data of each component on the server is massive. If all the collected massive data is transmitted to edge servers and cloud servers for fault warning, it will inevitably cause serious latency due to the transmission of all data. Moreover, both edge servers and cloud servers need to perform fault warning on all data, resulting in repetitive processing of status data, wasting computing and communication resources, and also causing serious latency. This cannot meet the fault warning requirements of real-time and computationally intensive services (such as autonomous driving and interactive virtual reality or augmented reality games).

[0056] To reduce server fault warning latency and improve the security and reliability of fault warnings for systems involving privacy data (such as military information processing systems and high-precision technology R&D systems), this embodiment utilizes the BMC (Backend Management System) to perform fault detection on the status data of privacy data services and / or time-sensitive services locally, thereby ensuring the security and reliability of fault warnings. Privacy data services refer to services that require the use of privacy data for fault detection, and time-sensitive services refer to services that require the use of time-sensitive data for fault detection.

[0057] Accordingly, the local detection service in this embodiment can be a service that needs to perform fault detection in the BMC (i.e., the service to be detected). The local trained model in this embodiment can be a trained model required for fault prediction in the BMC (e.g., Figure 3 (Computational models in the context of BMC and edge servers). For example, to adapt to the computing power and memory requirements of BMC and edge servers, the trained models on this end can adopt efficient network models with fewer parameters and less computation, such as SquezeNet (a lightweight convolutional neural network model), MobileNet (a lightweight neural network model), and GhostNet (a lightweight deep learning model).

[0058] Correspondingly, the specific method by which BMC uses the local trained models of each local detection service to perform fault detection on the status data of each local detection service, obtains the fault detection results of each local detection service, and sends the fault detection results to the remote server can be set by the designers according to the practical scenario and user needs. For example, BMC can use the local trained models of each local detection service to perform fault detection on the status data of each local detection service, and directly send the detected fault detection results of each local detection service to the remote server. This allows BMC to use fault detection when the local detection service includes a small portion of the services to be detected in all the services to be monitored on the server (such as privacy data services and time-sensitive services), ensuring the security and reliability of fault warnings. In other words, the peer detection service in step 103 can be a service other than the local detection service in all the services to be detected.

[0059] Correspondingly, after BMC performs fault detection on the status data of each local detection service using the local trained model of each local detection service, it can also filter out local feedback services with confidence scores higher than the corresponding confidence thresholds based on the confidence scores of the fault detection results of each local detection service, and send the fault detection results of the local feedback services to the remote server, so that the fault detection results corresponding to the status data with high confidence scores can be directly output to the remote server; that is, the peer detection service in step 103 can be a service other than the local feedback service among all the services to be detected, so that the status data with low confidence scores can be sent to a low-latency server for more accurate fault detection.

[0060] For example, the process of sending the fault detection results to the remote server in this step may include: detecting the local feedback service in the local detection service according to the confidence level of the fault detection results of each local detection service; wherein, the local feedback service includes local detection services whose fault detection results have a confidence level greater than the corresponding confidence level threshold; sending the fault detection results of the local feedback service to the remote server; wherein, the remote detection service includes the detection services other than the local feedback service among all the detection services of the server to be monitored.

[0061] For example, the local detection service can include all services to be detected. In this step, the BMC can use the local trained models of each service to perform fault detection on the status data of each service to be detected, and obtain the fault detection results of each service. If the current service to be detected is a privacy data service or a time-sensitive service, then it is determined whether the confidence level of the fault detection result of the current service to be detected is greater than a first confidence threshold. Here, the current service to be detected is any service to be detected. If it is greater than the first confidence threshold, then the current service to be detected is determined to be a local feedback service. If it is not greater than the first confidence threshold, then... If the current service to be detected is determined to be a peer-to-peer detection service, then more accurate fault detection can be performed using the corresponding trained model in the low-latency server in step 103. If the current service to be detected is not a privacy data service or a time-sensitive service, then it is determined whether the confidence level of the fault detection result of the current service to be detected is greater than the second confidence threshold. If it is greater than the second confidence threshold, then the current service to be detected is determined to be a local feedback service. If it is not greater than the second confidence threshold, then the current service to be detected is determined to be a peer-to-peer detection service. The fault detection results of the local feedback services among all services to be detected are sent to the remote server. The second confidence threshold can be greater than the first confidence threshold, so that by setting a smaller first confidence threshold, the two types of services, privacy data services and time-sensitive services, can be detected at the BMC as much as possible, ensuring the timeliness, reliability and security of fault warnings.

[0062] This embodiment does not limit the specific content of the fault detection results. For example, the fault detection results may include any one or more of the following: fault type, fault occurrence time, faulty component, component location, and fault handling plan. For instance, in order to enable the remote server to quickly locate the fault location, understand the fault cause, and the handling plan, the fault detection results may include the fault type, fault occurrence time, faulty component, component location, and fault handling plan.

[0063] Step 103: Send the status data of each peer detection service to the low-latency server, so as to use the trained model of each peer detection service in the low-latency server to perform fault detection on the status data of each peer detection service, and send the fault detection results of each peer detection service to the remote server.

[0064] Among them, low-latency servers are some of the edge servers and cloud servers that communicate with BMC. The communication latency between low-latency servers and BMC is lower than that of other servers in edge servers and cloud servers.

[0065] It is understood that the peer detection service in this embodiment can be any service among all the services to be detected on the server to be monitored that requires fault detection on a low-latency server, such as services other than those for which the BMC sends corresponding fault detection results to a remote server (i.e., local feedback services). The low-latency server in this embodiment can be one of the edge servers and cloud servers with relatively low current communication latency that are connected to the BMC, such as the server with the lowest communication latency.

[0066] Correspondingly, in this step, the BMC sends the status data of each peer detection service to a low-latency server. This allows the low-latency server (such as an edge server or cloud server) to utilize its trained models for each peer detection service to perform fault detection on the status data of each peer detection service. The low-latency server then sends the detected fault detection results to the remote server. This allows the low-latency server, with its low communication latency with the BMC, to accurately detect faults in the status data of the peer detection services (such as status data where the BMC's fault detection results are unclear). Compared to related technologies where the BMC transmits all data to edge servers and cloud servers, this significantly reduces network transmission pressure and the computational load on the BMC during data processing. It fully utilizes the excellent computing power of edge servers or cloud servers, resulting in the optimal overall fault warning effect of the system. This provides low-latency server fault monitoring and warning services even with the continuous and rapid growth in the number of servers connected to the network, meeting the fault warning needs of services with high real-time requirements and computational demands.

[0067] Correspondingly, the method provided in this embodiment may also include the process of BMC detecting the communication latency with the edge server and the cloud server respectively; for example, network probing, path tracing and / or network diagnostic tools can be used to detect the communication latency with the edge server and the cloud server.

[0068] It should be noted that the trained models in the BMC (i.e., the locally trained models) as well as the trained models in the edge servers and cloud servers can all be models trained by the cloud server. In other words, both the locally trained models and the trained models in the edge servers can be models trained by the cloud server and then distributed, thus utilizing the cloud server to handle the model training process and reducing the computational load on the BMC and edge servers. For example, in some embodiments, the BMC can directly receive models trained by the cloud server and then distributed (i.e., locally trained models).

[0069] In other embodiments, the system receives an initial trained model corresponding to the current local detection service sent by the cloud server; it then uses sample data from the current local detection service to perform transfer training on the initial trained model to obtain a local trained model for the current local detection service. Here, the current local detection service is any local trained model, and the initial trained model is a model trained by the cloud server using sample data from at least two service types. In other words, since server fault warnings for different service types are all based on state data collected by the BMC, in the field of server fault detection based on state data collected by the BMC, it can be assumed that the state data of various server components collected by the BMC under different service types follow the same distribution and originate from the same feature space. Based on this, in this embodiment, the cloud server can perform multi-task learning based on state data from multiple (i.e., at least two) service types, enabling it to find and understand the correlation and hierarchy of different tasks, thereby improving the model's generalization ability.

[0070] Correspondingly, the process by which the edge server obtains the trained model corresponding to each peer detection service is similar to the process by which the BMC obtains the local trained model of the current local detection service, and will not be described in detail here.

[0071] For example, a cloud server can train a model using sample data from two different business types; the cloud server then distributes the trained model to the BMC. Since the status data of various server components collected by the BMC are correlated across different business types, to make the model learned by the cloud server through multi-task learning more adaptable to fault warnings under the current business type, transfer learning can be used to share the learned model parameters with the local model in the BMC. This accelerates and optimizes the model's learning efficiency, unlike most networks that learn from scratch. In other words, the BMC can use sample data from the current business type to perform transfer training on the model sent by the cloud server, obtaining a locally trained model.

[0072] Correspondingly, in this embodiment, it is considered that the feature extractor in the model trained on the cloud server may not have processed a certain part of the data in the current business, which may result in the output target feature not belonging to the source data distribution, so that the model cannot accurately predict this type of feature; in this embodiment, a business classifier is added to the model. For example, the cloud server can add a classifier after each intermediate layer of the initial model to obtain the model to be trained. During the model training process, the business classifier can determine the business to which the feature extracted by the feature extractor belongs, so that the feature extractor can extract the feature corresponding to the business from the sample data under the current business type, and realize the training of the initial trained model of each business to be detected.

[0073] The process of training a model using sample data from multiple business types on a cloud server can include: acquiring status data of various components of the data center server under different business types, as well as fault causes, fault locations, and fault handling solutions, as sample data to establish a multidimensional dataset; assigning corresponding labels to the data in the multidimensional dataset according to whether the components are normal or faulty; dividing the sample data in the multidimensional dataset to obtain a training set and a validation set; inputting the data from the training set into the model to be trained; verifying the generalization ability of the model to be trained using the data from the validation set, and adjusting the hyperparameters of the model to be trained based on the model performance to obtain the initial trained model.

[0074] It should be noted that in this embodiment, the remote server can display the fault detection results of each received service to the user, and the remote server can also support the user's confirmation and adjustment of the displayed fault warning results. To improve the accuracy of fault warnings, this embodiment can also collect user feedback information on fault monitoring results (i.e., fault detection result user feedback information) to dynamically update the model. Considering that adjusting the model using fault detection result user feedback information would generate a large amount of computation, the remote server can send the fault detection result user feedback information to the cloud server, utilizing the cloud server's powerful computing capabilities to dynamically update the model. For example, the cloud server can obtain the fault detection result user feedback information sent by the remote server; increase the input of the corresponding trained model to update and train the trained model.

[0075] Since the models of BMC, cloud servers, and edge servers can all be trained based on the same initial trained model and sample data from the same multiple business types, their model parameters are essentially the same, at least the parameters of neurons learning common features are essentially the same. Based on this, the cloud server can first filter the updated model parameters in the cloud server. Parameters of neurons with small changes (i.e., neuron parameters) are not sent to the BMC or edge servers; only the parameters of neurons with large changes are sent. This allows the BMC or edge servers to maintain the parameters of other neurons when updating their local models, adjusting only the corresponding neuron parameters based on the sent parameters. This reduces the communication overhead of sending parameters and lowers the training computational cost of the local model on the BMC or edge servers. In other words, the cloud server can filter parameters (i.e., neuron parameters) with changes greater than the update threshold from the updated trained model; and send the filtered parameters to the edge servers and BMC to adjust the corresponding parameters in their respective models.

[0076] Accordingly, the method provided in this embodiment may further include: the BMC receiving a model adjustment instruction sent by the cloud server; adjusting the corresponding neuron parameters in the target local trained model according to the neuron parameters to be adjusted; wherein, the model adjustment instruction includes the neuron parameters to be adjusted in the target local trained model, the neuron parameters to be adjusted are the neuron parameters whose change is greater than the update threshold determined by the cloud server based on the fault detection results and user feedback information collected by the remote server; the target local trained model can be a local trained model that needs to adjust its neuron parameters.

[0077] Based on this, the cloud server updates the model using user feedback information for each received fault detection result, and sends neuron parameters with changes exceeding the update threshold to the BMC or edge server. Due to the limited computing power of the BMC or edge server, if the cloud server receives multiple user feedback messages for fault detection results from the remote server within a preset time, to alleviate the gradient staleness problem of the local model on the BMC or edge server, the BMC or edge server may not immediately update its local model parameters after receiving the neuron parameters sent by the cloud server. Instead, it will update the model parameters only after a preset time has elapsed. During this period, the BMC or edge server's model continues to perform fault detection with the previous parameters. In other words, the BMC can adjust the corresponding neuron parameters in the target local trained model according to the neuron parameters to be adjusted after receiving the model adjustment instruction within a preset time.

[0078] In this embodiment, the present invention utilizes the trained models of each local detection service to perform fault detection on the status data of each local detection service, obtains the fault detection results of each local detection service, and sends the fault detection results to a remote server. This enables fault identification of services related to privacy data and time-sensitive data on the BMC, minimizing the leakage of privacy information on the server and improving the efficiency of fault identification in time-sensitive services, thereby improving the timeliness, reliability, and security of fault warnings. Furthermore, by sending the status data of each peer detection service to a low-latency server, and utilizing edge servers or cloud servers with lower current communication latency for fault prediction of the corresponding services, the fault warning requirements of services with high real-time requirements and large computational demands can be met.

[0079] Corresponding to the above method embodiments, this invention also provides a server fault detection device. The server fault detection device described below and the server fault detection method described above can be referred to each other.

[0080] Please refer to Figure 4 , Figure 4 This is a structural block diagram of a server fault detection device provided in an embodiment of the present invention. The device is applied to a BMC (Browser Control Center) and may include:

[0081] The acquisition module 10 is used to collect status data of each component in the server to be monitored;

[0082] The local detection module 20 is used to perform fault detection on the status data of each local detection service using the local trained model of each local detection service, obtain the fault detection results of each local detection service, and send the fault detection results to the remote server; wherein, the local detection services include privacy data services and / or time-sensitive services.

[0083] The sending detection module 30 is used to send the status data of each peer detection service to the low-latency server, so as to use the trained model of each peer detection service in the low-latency server to perform fault detection on the status data of each peer detection service, and send the fault detection results of each peer detection service to the remote server; wherein, the low-latency server is a part of the edge server and cloud server that are connected to the baseboard management controller, and the communication latency between the low-latency server and the baseboard management controller in the edge server and cloud server is lower than that of other servers.

[0084] In some embodiments, the sending detection module 30 may include:

[0085] The confidence level detection submodule is used to detect local feedback services in the local detection services based on the confidence level of the fault detection results of each local detection service; wherein, local feedback services include local detection services whose fault detection results have a confidence level greater than the corresponding confidence level threshold.

[0086] The result generation submodule is used to send the fault detection results of the local feedback service to the remote server; wherein, the remote detection service includes all the detection services of the server to be monitored, excluding the local feedback service.

[0087] In some embodiments, the sending detection module 30 may include:

[0088] The fault detection submodule is used to perform fault detection on the status data of each service under test by utilizing the local trained models of each service under test, and to obtain the fault detection results of each service under test.

[0089] The first judgment submodule is used to determine whether the confidence level of the fault detection result of the current service to be detected is greater than a first confidence threshold if the current service to be detected is a privacy data service or a time-sensitive service; wherein, the current service to be detected is any service to be detected; if it is greater than the first confidence threshold, the current service to be detected is determined to be a local feedback service; if it is not greater than the first confidence threshold, the current service to be detected is determined to be a peer detection service;

[0090] The second judgment submodule is used to determine whether the confidence level of the fault detection result of the current service to be detected is greater than a second confidence threshold if the current service to be detected is not a privacy data service or a time-sensitive service; wherein the second confidence threshold is greater than the first confidence threshold; if it is greater than the second confidence threshold, the current service to be detected is determined to be a local feedback service; if it is not greater than the second confidence threshold, the current service to be detected is determined to be a peer detection service.

[0091] The sending submodule is used to send the fault detection results of the local feedback service among all the services to be detected to the remote server.

[0092] In some embodiments, both the trained model on the local end and the trained model on the edge server are models trained on the cloud server and then distributed.

[0093] In some embodiments, the device may further include:

[0094] The model receiving module is used to receive the initial trained model corresponding to the current local detection service sent by the cloud server; wherein, the current local detection service is any local trained model, and the initial trained model is a model trained by the cloud server using sample data of at least two service types.

[0095] The transfer training module is used to transfer train the initially trained model using sample data from the current local detection service, so as to obtain the local trained model for the current local detection service.

[0096] In some embodiments, the device may further include:

[0097] The receiving module is adjusted to receive model adjustment instructions sent by the cloud server. The model adjustment instructions include the neuron parameters to be adjusted of the target local trained model. The neuron parameters to be adjusted are the neuron parameters whose changes are greater than the update threshold, determined by the cloud server based on the fault detection results and user feedback information collected by the remote server.

[0098] The model update module is used to adjust the corresponding neuron parameters in the target local trained model based on the neuron parameters to be adjusted.

[0099] In this embodiment, the local detection module 20 uses the trained models of each local detection service to perform fault detection on the status data of each local detection service, obtains the fault detection results of each local detection service, and sends the fault detection results to the remote server. This enables fault identification of services related to privacy data and time-sensitive data on the BMC, minimizing the leakage of privacy information on the server and improving the efficiency of fault identification in time-sensitive services, thereby improving the timeliness, reliability, and security of fault warning. Furthermore, the sending detection module 30 sends the status data of each peer detection service to a low-latency server, and uses edge servers or cloud servers with low current communication latency to perform fault prediction for the corresponding services, which can meet the fault warning requirements of services with high real-time requirements and large computational load.

[0100] Corresponding to the above method embodiments, this invention also provides a baseboard management controller. The baseboard management controller described below and the server fault detection method described above can be referred to in correspondence.

[0101] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a baseboard management controller provided in an embodiment of the present invention. The server may include:

[0102] Memory D1 is used to store computer programs;

[0103] The processor D2 is used to execute computer programs to implement the steps of the server fault detection method provided in the above embodiments.

[0104] Corresponding to the above method embodiments, this invention also provides a server fault detection system. The server fault detection system described below and the server fault detection method described above can be referred to each other.

[0105] Please refer to Figure 6 , Figure 6 This is a schematic diagram of a server fault detection system provided in an embodiment of the present invention. The system may include: an edge server 100, a cloud server 200, and a baseboard management controller 300 as provided in the above embodiment;

[0106] The baseboard management controller 300 is communicatively connected to the edge server 100 and the cloud server 200. The edge server 100 and the cloud server 200 are both used to perform fault detection on the status data of each peer detection service sent by the baseboard management controller 300 using their respective trained models of peer detection services, and send the fault detection results of each peer detection service to the remote server.

[0107] In other words, when the edge server 100 and the cloud server 200 act as low-latency servers in the above embodiments, they can perform fault detection on the status data of the peer detection service sent by the baseboard management controller 300.

[0108] In other embodiments, the cloud server 200 can also be used to train and send the local trained model to the baseboard management controller 300; and to train and send the trained model to the edge server.

[0109] In other embodiments, the cloud server 200 may be specifically used to train an initial trained model corresponding to the current local detection service using sample data from at least two service types; and send the initial trained model to the baseboard management controller 300 so that the baseboard management controller 300 can use the sample data of the current local detection service to perform transfer training on the initial trained model to obtain the local trained model of the current local detection service.

[0110] In other embodiments, the cloud server 200 can also be used to receive user feedback information on fault detection results collected by a remote server; to train and update the corresponding trained model based on the user feedback information on fault detection results; and if the change in the updated neuron parameters is greater than the update threshold, to generate corresponding model adjustment instructions using the updated neuron parameters.

[0111] Corresponding to the above method embodiments, this invention also provides a computer program product. The computer program product described below and the server fault detection method described above can be referred to each other.

[0112] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the server fault detection method provided in the above embodiments.

[0113] Corresponding to the above method embodiments, this invention also provides a computer-readable storage medium. The computer-readable storage medium described below and the server fault detection method described above can be referred to in correspondence.

[0114] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the server fault detection method provided in the above embodiments.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus, BMC, system, computer program product, and computer-readable storage medium disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant details can be found in the method section.

[0116] The foregoing has provided a detailed description of a server fault detection method, apparatus, system, baseboard management controller, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A failure detection method of a server, characterized by, The application is applied to a substrate management controller, comprising: Collecting state data of each component in a server to be monitored; Using a locally trained model of each local detection service to perform fault detection on the state data of each local detection service, obtaining fault detection results of each local detection service, and sending the fault detection results to a remote server; wherein the local detection service includes a privacy data service and / or a time-sensitive service; Sending the state data of each opposite-end detection service to a low-latency server to use a trained model of each opposite-end detection service in the low-latency server to perform fault detection on the state data of each opposite-end detection service, and sending the fault detection results of each opposite-end detection service obtained by detection to the remote server; wherein the low-latency server is part of servers in an edge server and a cloud server in communication connection with the substrate management controller, the communication delay of the low-latency server in the edge server and the cloud server to the substrate management controller is less than that of other servers; the opposite-end detection service includes services other than the local detection service in all services to be detected.

2. The failure detection method of a server according to claim 1, characterized by, The sending of the fault detection results to the remote server comprises: Detecting a local feedback service in the local detection service according to the confidence of the fault detection results of each local detection service; wherein the local feedback service includes a local detection service whose confidence is greater than a corresponding confidence threshold; Sending the fault detection results of the local feedback service to the remote server; wherein the opposite-end detection service includes services to be detected other than the local feedback service in all services to be detected of the server to be monitored.

3. The method of claim 1, wherein the server is a server of a network. The use of the locally trained model of each local detection service to perform fault detection on the state data of each local feedback service, to obtain the fault detection results of each local detection service, and to send the fault detection results to the remote server comprises: Using the locally trained model of each of all services to be detected to perform fault detection on the state data of each service to be detected, to obtain the fault detection results of each service to be detected; If the current service to be detected is the privacy data service or the time-sensitive service, determining whether the confidence of the fault detection results of the current service to be detected is greater than a first confidence threshold; wherein the current service to be detected is any of the services to be detected; If it is greater than the first confidence threshold, determining that the current service to be detected is a local feedback service; If it is not greater than the first confidence threshold, determining that the current service to be detected is the opposite-end detection service; If the current service to be detected is not the privacy data service or the time-sensitive service, determining whether the confidence of the fault detection results of the current service to be detected is greater than a second confidence threshold; wherein the second confidence threshold is greater than the first confidence threshold; If it is greater than the second confidence threshold, determining that the current service to be detected is the local feedback service; If it is not greater than the second confidence threshold, determining that the current service to be detected is the opposite-end detection service; Send the fault detection result of the local feedback service in all the to-be-detected services to the remote server.

4. The method of claim 1 to 3, wherein, The local trained model and the trained model in the edge server are both models issued by the cloud server after training.

5. The method of claim 4, wherein the server is a server of a network. Before the step of using the local trained model of each local detection service to perform fault detection on the state data of each local detection service to obtain the fault detection result of each local detection service, the method further includes: Receiving an initial trained model corresponding to a current local detection service sent by the cloud server; wherein the current local detection service is any local trained model, and the initial trained model is a model trained by the cloud server using sample data of at least two service types; Using sample data of the current local detection service to perform transfer training on the initial trained model to obtain a local trained model of the current local detection service.

6. The method of claim 4, wherein the server is a server of a network. Further comprising: Receiving a model adjustment instruction sent by the cloud server; wherein the model adjustment instruction includes to-be-adjusted neuron parameters of a target local trained model, and the to-be-adjusted neuron parameters are neuron parameters whose change amount is greater than an update threshold value, which are determined by the cloud server according to fault detection result user feedback information collected by the remote server; Adjusting the corresponding neuron parameters in the target local trained model according to the to-be-adjusted neuron parameters.

7. A failure detection apparatus of a server, characterized by comprising: Applied to a baseboard management controller, comprising: A collection module configured to collect state data of each component in a to-be-monitored server; A local detection module configured to use a local trained model of each local detection service to perform fault detection on state data of each local detection service to obtain a fault detection result of each local detection service, and send the fault detection result to a remote server; wherein the local detection service includes a privacy data service and / or a time-sensitive service; A sending detection module configured to send state data of each opposite detection service to a low-latency server to use a trained model of each opposite detection service in the low-latency server to perform fault detection on state data of each opposite detection service, and send a fault detection result of each opposite detection service obtained by detection to the remote server; wherein the low-latency server is part of servers in an edge server and a cloud server in communication connection with the baseboard management controller, the communication delay of the low-latency server in the edge server and the cloud server to the baseboard management controller is less than that of other servers; and the opposite detection service includes services other than the local detection service in all to-be-detected services.

8. A baseboard management controller, comprising: Comprising: A memory configured to store a computer program; A processor configured to implement the steps of the fault detection method of the server according to any one of claims 1 to 6 when executing the computer program.

9. A failure detection system of a server, characterized by, Comprising: An edge server, a cloud server, and a baseboard management controller according to claim 8; Wherein the baseboard management controller is in communication connection with the edge server and the cloud server; The edge server and the cloud server are both configured to detect faults of the state data of the peer detection service sent by the baseboard management controller by using a trained model of the peer detection service respectively, and send the detected fault detection results of the peer detection service to the remote server.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the fault detection method of the server according to any one of claims 1 to 6.

Images