Server fault prediction method, device and equipment based on multiple modes

Abstract

The invention provides a server fault prediction method, device and equipment based on multiple modes, which are applied to the technical field of fault prediction, and the method comprises the following steps: collecting multi-mode data of a node server; the multi-modal data includes at least two of thermograms, system logs, GPU utilization, power consumption, temperature, fan speed and fan audio. And respectively extracting a modal feature corresponding to each modal from the multi-modal data. And fusing modal features of all modals in the multi-modal data, and predicting fault information of the node server. And mapping the fault information to the thermogram to obtain a fault risk map of the node server. The fault risk map is used for displaying the fault information in the thermogram in an overlapping manner. The problem that a traditional fault prediction method based on a single data source is difficult to discover hidden problems generated by a server, so that the fault of the server cannot be predicted in time can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of fault prediction, and in particular to a multimodal-based server fault prediction method, apparatus, and device. Background Art

[0002] With the rapid development of artificial intelligence, big data, and high-performance computing, GPU servers, due to their superior parallel computing capabilities, have been widely used in data centers and high-performance computing clusters to handle high-load tasks such as large-scale model training, image rendering, and scientific computing. GPUs tend to generate a large amount of heat during continuous high-load operation. If heat is not dissipated promptly and effectively, the GPU core temperature may overheat, leading to serious consequences such as system frequency throttling, performance degradation, hardware damage, and even complete system downtime, affecting task continuity and system stability.

[0003] To ensure the reliable operation of GPU servers, related technologies generally use monitoring methods based on hardware sensors and system logs to monitor the operating status of GPUs. These methods usually rely on a single modal data source, such as relying solely on the temperature value fed back by the temperature sensor or the error information recorded in the system log to determine whether the GPU is abnormal. However, in actual applications, GPU failures are often complex and hidden, such as the gradual decline in fan performance, local failure of the cooling system, or abnormal changes in the heat conduction path. These problems may not immediately trigger obvious temperature mutations or system alarms, but will gradually accumulate, eventually leading to thermal runaway or equipment damage. Therefore, traditional fault prediction methods based on a single data source often lack sufficient early warning capabilities when faced with these slowly evolving, multi-factor coupled hidden faults, making it difficult to achieve early detection and timely intervention. Summary of the Invention

[0004] The purpose of this application is to provide a multimodal server fault prediction method, device and equipment to solve the problem that traditional fault prediction methods based on a single data source have difficulty in discovering hidden problems generated by the server and thus cannot predict server failures in a timely manner.

[0005] In the first aspect, an embodiment of the present application provides a server fault prediction method based on multimodality, which is applied to a server platform, and the server platform includes at least one node server. The method includes: collecting multimodal data of the node server. The multimodal data includes at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed and fan audio. The modal features corresponding to each mode are extracted from the multimodal data respectively. The modal features of all modes in the multimodal data are fused to predict the fault information of the node server. The fault information is mapped to the thermal image to obtain a fault risk map of the node server. The fault risk map is used to superimpose the fault information on the thermal image.

[0006] The multimodal server fault prediction method provided in the embodiment of the present application collects multimodal data of the node server, including at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed and fan audio, and extracts modal features corresponding to each mode, and then fuses these modal features to predict the fault information of the node server, and then maps the fault information to the thermal image to generate a fault risk map to display the fault information. This can effectively improve the accuracy and reliability of fault detection, reduce false alarms and missed alarms, help operation and maintenance personnel quickly locate the fault area, improve operation and maintenance efficiency, and reduce operation and maintenance costs.

[0007] In one possible implementation, fault information includes all fault types and the corresponding failure probabilities for each fault type. Modal features from all modes in the multimodal data are fused to predict fault information for the node server. This involves time-aligning and fusing the modal features of all modes to construct a feature vector. The trained fault prediction model is then used to classify and predict faults in the feature vector, determining all fault types for the node server and the corresponding failure probabilities for each type.

[0008] In one possible implementation, the method further includes obtaining a pre-built fault knowledge graph on the server platform. The fault knowledge graph includes multiple nodes. The nodes include modal feature nodes, fault type nodes, device type nodes, and feedback information nodes. Modal feature nodes correspond to multimodal data. Fault type nodes correspond to fault types. Device type nodes are used to represent the device model of devices in the node server. Devices include GPUs and fans. Feedback information nodes are used to represent feedback from operations and maintenance personnel on fault information. The fault knowledge graph is used to determine the confidence level of the fault information. The confidence level is used to verify the authenticity of the fault information.

[0009] One possible implementation involves temporally aligning and fusing the modal features of all modalities to construct a feature vector. This involves determining the fluctuation intensity of any target modality within the current acquisition window. The fluctuation intensity represents the degree of change in the modal features corresponding to the target modality within the current acquisition window. The fluctuation intensity is used to adjust the attention weight of the target modality to obtain the fluctuation weight corresponding to the target modality. A feature vector is then determined based on the modal features of all modalities and the corresponding fluctuation weights.

[0010] One possible implementation involves mapping fault information to a thermal image to obtain a fault risk map for the node server. This includes determining, based on the thermal image, a corresponding thermal channel activation heat map. The thermal channel activation heat map represents image regions within the thermal image that are associated with the fault information. The fault information is mapped to the thermal image using the thermal channel activation heat map to determine the fault risk map for the node server.

[0011] In one possible implementation, the method further includes: receiving feedback from operation and maintenance personnel regarding fault information. Binding the feedback information with multimodal data to determine labeled samples for the node server. Utilizing the device types and corresponding labeled samples of multiple node servers on the server platform, determining the node weight corresponding to each node server. Based on the node weight corresponding to each node server and the model parameters of the fault prediction model corresponding to each node server, adjusting the global parameters of the global fault prediction model of the server platform. The global fault prediction model is used to indicate adjustments to the fault prediction model of each node server.

[0012] A possible implementation method is that after receiving feedback information on fault information from operation and maintenance personnel, the method also includes: using the fault information of the node server, the modal features corresponding to each mode in the multimodal data, the device type of the node server, and the feedback information on the fault information from the operation and maintenance personnel, to update the fault knowledge graph of the server platform.

[0013] In one possible implementation, the method further includes: upon detecting that any modal data in the multimodal data is missing, reasoning using associations in a pre-built fault knowledge graph to determine the missing modal data, and filling the multimodal data with the missing modal data.

[0014] In a second aspect, an embodiment of the present application provides a multimodal-based server fault prediction device, which is applied to a server platform. The server platform includes at least one node server, and the device includes: an acquisition module, an extraction module, a prediction module and a mapping module.

[0015] The acquisition module is configured to collect multimodal data from the node server, including at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed, and fan audio.

[0016] The extraction module is used to extract the modal features corresponding to each modality from the multimodal data.

[0017] The prediction module is used to fuse the modal features of all modes in the multimodal data and predict the fault information of the node server.

[0018] The mapping module is used to map the fault information to the thermal image to obtain a fault risk map of the node server. The fault risk map is used to overlay the fault information on the thermal image.

[0019] In a third aspect, embodiments of the present application provide a multimodal server fault prediction device, which functions to implement the multimodal server fault prediction method described in the first aspect or any possible implementation of the first aspect. This function can be implemented via hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer can execute the multi-modal server fault prediction method of the above-mentioned first aspect or any possible implementation of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the multimodal-based server fault prediction method of the above-mentioned first aspect or any possible implementation method.

[0022] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by the first aspect or different possible implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 A method flow chart of a multi-modal server failure prediction method provided in an embodiment of the present application; Figure 2 A specific example diagram of a thermal image provided in an embodiment of the present application; Figure 3 A specific example diagram of a fault knowledge graph provided in an embodiment of the present application; Figure 4 A specific example diagram of a fault risk diagram provided in an embodiment of the present application; Figure 5A schematic structural diagram of a multi-modal server fault prediction device provided in an embodiment of the present application; Figure 6 A system architecture diagram of a multimodal server fault prediction system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0027] To ensure the reliable operation of GPU servers, related technologies generally use monitoring methods based on hardware sensors and system logs to monitor the operating status of GPUs. These methods usually rely on a single modal data source, such as relying solely on the temperature value fed back by the temperature sensor or the error information recorded in the system log to determine whether the GPU is abnormal. However, in actual applications, GPU failures are often complex and hidden, such as the gradual decline in fan performance, local failure of the cooling system, or abnormal changes in the heat conduction path. These problems may not immediately trigger obvious temperature mutations or system alarms, but will gradually accumulate, eventually leading to thermal runaway or equipment damage. Therefore, traditional fault prediction methods based on a single data source often lack sufficient early warning capabilities when faced with these slowly evolving, multi-factor coupled hidden faults, making it difficult to achieve early detection and timely intervention.

[0028] Based on this, an embodiment of the present application provides a server fault prediction method based on multimodality, which is applied to a server platform, and the server platform includes at least one node server. The method includes: collecting multimodal data of the node server. The multimodal data includes at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed and fan audio. The modal features corresponding to each mode are extracted from the multimodal data respectively. The modal features of all modes in the multimodal data are fused to predict the fault information of the node server. The fault information is mapped to the thermal image to obtain a fault risk map of the node server. The fault risk map is used to superimpose the fault information on the thermal image.

[0029] The multimodal server fault prediction method provided in the embodiment of the present application collects multimodal data of the node server, including at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed and fan audio, and extracts modal features corresponding to each mode, and then fuses these modal features to predict the fault information of the node server, and then maps the fault information to the thermal image to generate a fault risk map to display the fault information. This can effectively improve the accuracy and reliability of fault detection, reduce false alarms and missed alarms, help operation and maintenance personnel quickly locate the fault area, improve operation and maintenance efficiency, and reduce operation and maintenance costs.

[0030] The method provided in the embodiments of the present application will be described below with reference to specific drawings.

[0031] On the one hand, an embodiment of the present application provides a multimodal-based server failure prediction method, which may include the following steps.

[0032] S101, collecting multimodal data from the node server.

[0033] The multimodal data includes at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed, and fan audio.

[0034] One possible implementation method is to use a thermal infrared camera to collect thermal images of the GPU chip or the entire server.

[0035] For example, a thermal imager such as FLIR A700 or Seek ShotPro is used to perform high-frequency sampling on the GPU chip or the entire server to capture its thermal distribution and obtain a thermal image. Figure 2 As shown, Figure 2 This is a specific example of a thermal image provided in an embodiment of the present application. The image resolution of this thermal image is 640×480 @ 30Hz. The pixel values ​​in the thermal image represent specific temperature values ​​in degrees Celsius. Brighter areas in the thermal image indicate higher temperatures.

[0036] Another possible implementation method is to collect log information at multiple levels of the server.

[0037] For example, log information may include GPU driver logs (such as logs generated by NVIDIA's nvidia-smi tool), Kubernetes container logs, and system syslog, etc. The log information records various warnings and error messages generated by the server during operation.

[0038] For example, a log message such as "2025-06-10 10:12:44 [WARN] GPU0 Core Temp 95°C (overheating)" indicates that the GPU0 core temperature is too high, reaching 95°C, posing an overheating risk. Another log message such as "2025-06-10 10:13:01 [ERROR] GPU0 Fan 0 RPM (stopped)" indicates that the fan speed of GPU0 is 0, meaning the fan has stopped.

[0039] Another possible implementation method is to collect the GPU utilization, power consumption, temperature, and fan speed of the server and convert them into independent timing vectors.

[0040] For example, taking power consumption as an example, the collected power consumption can be converted into a time series vector: P =( P 1, P 2, ..., Pt ,...).in, Pt Represents the power value at time t.

[0041] Another possible implementation is to use a microphone to collect the fan audio of the server.

[0042] For example, a miniature stationary microphone (such as a MEMS array) is placed on the GPU server or rack side, capturing audio signals at a 44.1kHz sampling rate, with either single or dual channels. Each capture is a 3-5 second audio clip, updated using a sliding window to continuously monitor the fan audio generated by the server during operation.

[0043] S102: extracting modal features corresponding to each modality from the multimodal data.

[0044] Specifically, we extract thermal distribution spatial features from thermal images, semantic features related to faults from system logs, trend change features from GPU utilization, power consumption, temperature, and fan speed, and abnormal noise frequency band features from fan audio.

[0045] One possible implementation method is to use a pre-trained deep learning model, such as a convolutional neural network (CNN) or Vision Transformer, to extract thermal distribution spatial features from thermal images. The deep learning model can automatically learn complex patterns and structures in the image and capture subtle changes in thermal distribution, thereby providing rich spatial information for fault detection. For example, after processing the thermal image through CNN or Vision Transformer, we get an image feature vector Imagefeature = [0.23, -0.12, ..., 0.09], whose dimension is 512. These eigenvalues ​​can be used to represent the feature intensity of the image at different spatial positions.

[0046] Another possible implementation method is to use BERT encoder to extract semantic features related to faults from system logs. BERT encoder can convert system logs into high-dimensional semantic feature vectors by understanding the context and semantic information in the system log text. For example, after BERT encoding the system log, a text feature vector Log feature = [0.01, 0.88, ..., 0.44], with a dimension of 768. These feature values ​​can be used to represent key semantic information in system logs.

[0047] Another possible implementation method is to use a temporal convolutional network (TCN) or a long short-term memory network (LSTM) to capture the trend changes and time dependencies in the data for temperature, power consumption, and fan speed. This process is based on processing the long-term dependencies in time series data and extracting features that reflect changes in system operating status from temperature, power consumption, and fan speed. For example, after processing time series data through TCN or LSTM, a time series feature vector TS can be obtained. feature = [0.25, 0.35, ..., 0.12], with a dimension of 128. These eigenvalues ​​can be used to represent the trend changes of various time series indicators at different time points.

[0048] Another possible implementation method is to use Mel-Frequency Cepstral Coefficients (MFCC) to characterize the fan speed change and abnormal sound frequency band characteristics. MFCC can capture the frequency and energy distribution information in the sound signal and can identify the normal operation sound and abnormal sound of the fan. For example, after MFCC processing the fan audio signal, a sound feature vector AD feature = [0.34, 0.43, ..., 0.23], whose dimension is 256. These eigenvalues ​​can be used to represent the energy distribution of fan sound in different frequency bands.

[0049] S103: Fusing the modal features of all modes in the multimodal data to predict the fault information of the node server.

[0050] The fault information includes: all fault types and the fault probability corresponding to each fault type.

[0051] One possible implementation involves time-aligning and fusing the modal features of all modes to construct a feature vector. The trained fault prediction model is then used to classify and predict faults within the feature vector, determining all fault types for the node server and the corresponding failure probability for each type.

[0052] Specifically, after extracting the modal features of all modalities in the multimodal data, the modal features of all modalities are first aligned in the time dimension to ensure their consistency in the temporal dimension. After completing the temporal alignment, the modal features of all modalities are integrated using a Transformer or a multimodal fusion network (such as the MM-Transformer) to construct the fused input tensor, i.e., the feature vector.

[0053] The trained fault prediction model is then used to classify and predict faults on the fused feature vectors to determine all fault types of the node server and the corresponding fault probability of each fault type.

[0054] For example, a multi-layer perceptron (MLP) or Transformer decoder is used to predict the fused vector and output the fault type (e.g., "Fan Failure," "GPU Overheat," "Power Anomaly," or "Normal") and its corresponding fault probability. For example, the output result may be: [Fan Failure: 0.72, GPU Overheat: 0.11, Power Anomaly: 0.03, Normal: 0.14].

[0055] Furthermore, in order to improve the accuracy and sensitivity of fault prediction, the embodiments of the present application dynamically guide the distribution of attention weights by using the modal attention mechanism and the modal feature volatility weighting mechanism in the process of feature fusion of the modal features of all modals to construct a feature vector, so that the attention distribution is based not only on the modal features, but also on the abnormal fluctuations of the modal data, making the fault prediction particularly sensitive to sudden events in the GPU server, such as sudden abnormal sounds and abnormal power consumption jitter.

[0056] One possible implementation involves determining the fluctuation intensity of any target modality within the current acquisition window for all modalities. Fluctuation intensity represents the degree of change in the modal features corresponding to the target modality within the current acquisition window. The fluctuation intensity is used to adjust the attention weight of the target modality to obtain the fluctuation weight corresponding to the target modality. A feature vector is then determined based on the modal features of all modalities and the fluctuation weights corresponding to each modality.

[0057] Specifically, for any target mode among all modes, the fluctuation intensity of the target mode in the current acquisition window, that is, the fluctuation coefficient, is first determined. The fluctuation intensity is used to indicate the degree of change of the modal characteristics corresponding to the target mode in the current acquisition window. It can be determined by the following formula.

[0058]

[0059] in, is the standard deviation of the mode in the sliding window, is the mean, is a small constant, and i is the target mode.

[0060] Then use the fluctuation coefficient to adjust the attention weight of the target modality to obtain the fluctuation weight corresponding to the target modality .

[0061]

[0062] Among them, W i and W j is the weight matrix, h i is the eigenvector, h j is the sum of the eigenvectors, i is the index of the eigenvector, and j is the sum index of the i eigenvectors.

[0063] The embodiment of the present application uses a modal attention mechanism and a modal feature volatility weighting mechanism to dynamically guide the distribution of attention weights, so that the attention distribution is not only based on the modal content, but also refers to the abnormal modal fluctuations, thereby making the model more sensitive to situations in the GPU server such as sudden abnormal sounds, abnormal power consumption jitter, etc., and can capture potential faults in the server earlier.

[0064] Furthermore, in order to improve the accuracy of the fault information predicted by the fault prediction model, the present application also determines the confidence level of the fault information by utilizing a pre-built fault knowledge graph to determine the true accuracy of the fault information prediction.

[0065] The fault knowledge graph can be constructed based on the modal features, fault types, device types, and feedback information from the historical fault prediction process. The modal features, fault types, device types, and feedback information are organized into multiple nodes and edges to form a directed fault knowledge graph.

[0066] This node can include a modal feature node, a fault type node, a device type node, and a feedback information node. The modal feature node corresponds to multimodal data. The fault type node corresponds to the fault type. The device type node represents the device model of the node server. Devices include GPUs and fans. The feedback information node represents feedback from maintenance personnel regarding fault information.

[0067] For example, a fault type node might be "abnormal fan noise" or "GPU overheating." A modal feature node might be "high-frequency noise peak" or "heatmap asymmetry." A device type node might be "GPU model" or "fan specifications." A feedback information node might be "confirmation" or "false alarm."

[0068] Edges are used to represent the relationship between connected nodes.

[0069] For example, the edge type may be “cause_of”, “has_feature”, “validated_by”, “similar_to”, etc.

[0070] After the graph modeling is completed, the modal characteristics, fault types, equipment types and feedback information in the historical fault prediction process are constructed and the corresponding representations are filled into the graph to build a complete fault knowledge graph.

[0071] For example, Figure 3 As shown, Figure 3 This is a specific example diagram of a fault knowledge graph provided in an embodiment of the present application. In this fault knowledge graph, yellow nodes are modal feature nodes, pink nodes are fault type nodes, green nodes are device type nodes, and purple nodes are feedback information nodes. The device type determined during the historical fault prediction process is a GPU model of Mthead S4000 and a fan model of Fan S23. The extracted modal features are a fan peak of 3kHz and a GPU throttling problem, the fault type is a fan abnormality, and the feedback information is true.

[0072] Specifically, when using the fault knowledge graph to determine the confidence level of fault information, a pre-built fault knowledge graph can be obtained from the server platform. The fault knowledge graph is used to determine the confidence level of the fault information. The confidence level is used to verify the authenticity of the fault information.

[0073] Specifically, based on the predicted fault information and the device type and modal characteristics determined during the fault prediction process, the fault knowledge graph is used to determine a corresponding directed graph within the fault knowledge graph. If the feedback information indicated in the directed graph is true, the confidence level of the fault prediction is increased.

[0074] For example, the collected fan peak is 3.2kHz, and the corresponding modal feature extracted is fan abnormality. Based on the fault knowledge graph, the corresponding feedback information can be determined to be true, thereby improving the confidence of the fault information prediction.

[0075] It should be noted that the fault knowledge graph can also be used as a training sample generator. Soft label data can be constructed through the nodes in the fault knowledge graph and the corresponding relationships between nodes. Using soft label data for knowledge distillation and enhanced training can improve the reliability of fault information predicted by the fault prediction model.

[0076] S104: Map the fault information to the thermal image to obtain a fault risk map of the node server.

[0077] Among them, the fault risk map is used to overlay the fault information on the thermal image.

[0078] One possible implementation involves determining a corresponding thermal channel activation heat map based on the thermal image. The thermal channel activation heat map represents the image area in the thermal image associated with the fault information. The fault information is mapped to the thermal image using the thermal channel activation heat map to determine a fault risk map for the node server.

[0079] For example, a Grad-CAM operation is performed on the thermal image. First, the thermal image is forward propagated through the CNN until the last convolutional layer, and the output of the feature map of the last convolutional layer is recorded. The forward propagation continues until the network output layer, and the score of the target category is calculated. The target category score is backpropagated to calculate the gradient of the feature map of the last convolutional layer. The gradient of the feature map of this convolutional layer is used to represent the contribution of each position on the feature map to the target category score. Global average pooling is performed on the gradient of each channel of the last convolutional layer to obtain the weight of each channel. The feature map of each channel is multiplied by its corresponding weight, and then the results of all channels are added to obtain a single heat map. The heat map is normalized and mapped to the thermal image. By adjusting the transparency adjustment factor (β) and applying the pseudo-color conversion function (colormap), the fault information is superimposed on the thermal image in a color-coded form to obtain a fault risk map. The fault risk map can be generated using the following formula.

[0080]

[0081] Among them, I risk (x, y) is the pixel value of the fault risk map at the coordinate (x, y), I original (x, y) is the pixel value of the thermal image at the coordinate (x, y), β is the transparency adjustment factor, colormap is the pseudo color conversion function, G cam(x, y) is the pixel value of the hot channel activation heat map at coordinate (x, y).

[0082] The process generates a failure risk map showing the failure risk areas by superimposing the failure information onto the thermal image, e.g. Figure 4 The figure shows a specific example of a fault risk map provided by an embodiment of the present application. Operation and maintenance personnel can use the fault risk map to quickly identify and locate the area where the fault occurs, for example, the area where the fault occurs is the G1 module and the G5 module.

[0083] Furthermore, feedback from operations and maintenance personnel regarding fault information is received. This feedback information is combined with multimodal data to determine labeled samples for the node servers. The device types and corresponding labeled samples of multiple node servers on the server platform are used to determine the node weight corresponding to each node server. Based on the node weight corresponding to each node server and the model parameters of the fault prediction model corresponding to each node server, the global parameters of the server platform's global fault prediction model are adjusted. The global fault prediction model is used to indicate adjustments to the fault prediction model for each node server.

[0084] Specifically, after completing the fault information prediction, detailed prediction information can be automatically provided to the operation and maintenance personnel. For example, the prediction information may include all the fault types predicted during the fault information prediction process, the corresponding probability of each fault type, and the contribution of key modes. Among them, the key mode is the mode whose fluctuation weight exceeds the threshold, such as the sound mode accounting for 78%. The operation and maintenance personnel need to confirm and feedback these prediction results. The feedback options may include "Confirm Fault", "False Alarm", "Ignore", etc.

[0085] The system then records the feedback and binds it to the corresponding modal features to form annotated samples. Using the device types of multiple node servers on the server platform and the corresponding annotated samples, the node weight corresponding to each node server is determined. This node weight can be determined using the following formula.

[0086]

[0087] Among them, ω k is the weight of the kth server, λ1, λ2, λ3 are weight coefficients, n k is the local sample size of the kth server, n is the sum of the sample sizes of all servers, Var k is the prediction loss variance of the kth server, GPUTypeFactor k is the sensitivity factor of the device type of the kth server to the target failure (different GPU models have different sensitivity factors to failures, for example, the sensitivity of the L40 GPU model is greater than that of the T4 GPU model).

[0088] Finally, based on the node weights and the model parameters of the fault prediction model corresponding to each node server, the global parameters of the server platform's global fault prediction model are adjusted. The global parameters can be determined using the following formula.

[0089]

[0090] Among them, θ is a global parameter, θ k is the model parameter of the kth server.

[0091] Furthermore, when adjusting the global parameters of the server platform's global fault prediction model, if a server detects a preset number of consecutive alarms (for example, more than three fan failures), the model parameters for that server can be uploaded to the server platform in advance, and the server platform adjusts the global parameters accordingly. This process allows the global model to quickly adapt to sudden fault information, improving the real-time and accuracy of fault prediction.

[0092] Furthermore, the fault knowledge graph of the server platform is updated by using the fault information of the node server, the modal features corresponding to each mode in the multimodal data, the device type of the node server, and the feedback information on the fault information from the operation and maintenance personnel.

[0093] Specifically, the nodes in the fault knowledge graph and the types of edges between nodes are updated by utilizing the fault information of the node server, the modal features corresponding to each mode in the multimodal data, the equipment type of the node server, and the feedback information on the fault information from the operation and maintenance personnel.

[0094] When any modal data in the multimodal data is missing, the missing modal data is determined by reasoning based on the association relationship in the pre-built fault knowledge graph. The missing modal data is then added to the multimodal data.

[0095] Specifically, during the data collection process, it is also possible to detect whether any modal data is missing, such as a missing thermal image due to a thermal imager failure or a problem with the data transmission process. If missing modal data is detected, reasoning can be performed using the associations in the fault knowledge graph. For example, if a thermal image is missing, modal data from other modalities such as fan speed, temperature, and power consumption, as well as their known associations with the thermal image, can be used to infer the missing thermal image. This padded multimodal data is then fed into the fault prediction model as input to fill in the reasoning blind spots in the missing modal scenario.

[0096] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the working principle of the device. It can be understood that in order to realize the above functions, the server fault prediction device based on multimodality includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0097] In the embodiments of the present application, the multimodal server fault prediction device can be divided into functional modules according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The integrated modules can be implemented in the form of hardware or software functional modules.

[0098] It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods. Figure 5 FIG. 1 shows a possible schematic diagram of the composition of the multi-modal server fault prediction device involved in the above and embodiments. Figure 5 As shown, the multimodal-based server fault prediction device 500 may include: a collection module 501 , an extraction module 502 , a prediction module 503 and a mapping module 504 .

[0099] The acquisition module 501 is used to support the server fault prediction device 500 based on multi-modality to execute Figure 1 S201 in the server failure prediction method based on multi-modality is shown.

[0100] Extraction module 502, used to support the server fault prediction device 500 based on multi-modality to execute Figure 1 S202 in the server failure prediction method based on multi-modality is shown.

[0101] Prediction module 503, used to support the server failure prediction device 500 based on multi-modality to execute Figure 1 S203 in the server failure prediction method based on multi-modality is shown.

[0102] Mapping module 504, used to support the server fault prediction device 500 based on multi-modality to execute Figure 1S204 in the server failure prediction method based on multi-modality is shown.

[0103] In one possible implementation, fault information includes all fault types and the corresponding failure probabilities for each fault type. A multimodal server fault prediction device specifically performs time alignment and feature fusion on the modal features of all modalities to construct a feature vector. The trained fault prediction model is then used to classify and predict faults in the feature vector, determining all fault types for the node server and the corresponding failure probabilities for each type.

[0104] In one possible implementation, a multimodal server fault prediction device is specifically used to obtain a pre-built fault knowledge graph for the server platform. The fault knowledge graph includes multiple nodes. These nodes include modal feature nodes, fault type nodes, device type nodes, and feedback information nodes. Modal feature nodes correspond to multimodal data. Fault type nodes correspond to fault types. Device type nodes are used to represent the device model of devices in the node server. Devices include GPUs and fans. Feedback information nodes are used to represent feedback from operations and maintenance personnel on fault information. The fault knowledge graph is used to determine the confidence level of the fault information. The confidence level is used to verify the authenticity of the fault information.

[0105] In one possible implementation, a multimodal server fault prediction device is specifically configured to determine the fluctuation intensity of any target modality within the current acquisition window for all modalities. The fluctuation intensity represents the degree of change in the modal features corresponding to the target modality within the current acquisition window. The attention weight of the target modality is adjusted using the fluctuation intensity to obtain the fluctuation weight corresponding to the target modality. A feature vector is then determined based on the modal features of all modalities and the fluctuation weights corresponding to the modalities.

[0106] In one possible implementation, a multimodal server fault prediction device is specifically configured to determine, based on a thermal image, a corresponding thermal channel activation heat map. The thermal channel activation heat map represents image regions within the thermal image associated with fault information. Fault information is mapped onto the thermal image using the thermal channel activation heat map to determine a fault risk map for the node server.

[0107] In one possible implementation, a multimodal server fault prediction device is specifically used to receive feedback from operations and maintenance personnel regarding fault information. This feedback information is bound to multimodal data to determine labeled samples for the node servers. The device types and corresponding labeled samples of multiple node servers on the server platform are used to determine the node weight corresponding to each node server. Based on the node weight corresponding to each node server and the model parameters of the fault prediction model corresponding to each node server, the global parameters of the server platform's global fault prediction model are adjusted. The global fault prediction model is used to indicate adjustments to the fault prediction model of each node server.

[0108] A possible implementation method is that the multimodal server fault prediction device is specifically used to update the fault knowledge graph of the server platform by utilizing the fault information of the node server, the modal features corresponding to each mode in the multimodal data, the equipment type of the node server, and the feedback information of the operation and maintenance personnel on the fault information.

[0109] In one possible implementation, a multimodal server fault prediction device is specifically configured to detect missing data in any modal data, use the associations in a pre-built fault knowledge graph to infer the missing modal data, and then fill in the missing modal data with the corresponding data in the multimodal data.

[0110] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0111] The server failure prediction device 500 based on multi-modality provided in the embodiment of the present application is used to perform the above Figure 1 The server failure prediction method based on multimodality shown can therefore achieve the same effect as the above-mentioned server failure prediction method based on multimodality.

[0112] An embodiment of the present application further provides a multimodal-based server fault prediction device, which can execute the multimodal-based server fault prediction method and related steps in the above method embodiment.

[0113] An embodiment of the present application further provides a computer-readable storage medium having instructions stored thereon, which, when executed, execute the multimodal-based server fault prediction method and related steps in the above-mentioned method embodiment.

[0114] An embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the multi-modal server failure prediction method and related steps in the above method embodiment.

[0115] In some embodiments, the methods described herein may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0116] The embodiment of the present application also provides a server fault prediction system 600 based on multi-modality, such as Figure 6 As shown, the multi-modality-based server fault prediction system 600 includes at least one processor 601 and at least one interface circuit 602 .

[0117] As an example, when the multi-modal server fault prediction system 600 includes a processor and an interface circuit, the processor may be Figure 6 The processor 601 shown in the solid line frame (or the processor 601 shown in the dotted line frame) may be Figure 6 The interface circuit 602 shown in the solid line frame (or the interface circuit 602 shown in the dotted line frame). When the multi-modal server fault prediction system 600 includes two processors and two interface circuits, the two processors include Figure 6 The processor 601 shown in the solid line frame and the processor 601 shown in the dotted line frame, the two interface circuits include Figure 6 The interface circuit 602 shown in the solid line frame and the interface circuit 602 shown in the dotted line frame are not limited to this.

[0118] The processor 601 and the interface circuit 602 can be interconnected via a line. For example, the interface circuit 602 can be used to receive signals. For another example, the interface circuit 602 can be used to send signals to other devices (such as the processor 601). For example, the interface circuit 602 can read computer instructions stored in the memory and send the computer instructions to the processor 601. The processor 601 executes the instructions and, in conjunction with the input and output devices, implements the various steps in the above embodiments, such as implementing Figures 1-4 Of course, the multi-modal server fault prediction system may also include other discrete components, which are not specifically limited in the present embodiment.

[0119] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the contributing part or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A server failure prediction method based on multimodality, characterized in that: Applied to a server platform, the server platform includes at least one node server, the method includes: Collect multimodal data from the node server; the multimodal data includes at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed, and fan audio; Extracting modal features corresponding to each modality from the multimodal data; fusing modal features of all modalities in the multimodal data to predict fault information of the node server; The fault information is mapped to the thermal image to obtain a fault risk map of the node server; the fault risk map is used to overlay and display the fault information on the thermal image.

2. The method according to claim 1, characterized in that The fault information includes: all fault types and the fault probability corresponding to each fault type; the fusing of modal features of all modes in the multimodal data to predict the fault information of the node server includes: Perform time alignment and feature fusion on the modal features of all modes to construct feature vectors; The trained fault prediction model is used to perform fault classification and prediction on the feature vector to determine all fault types of the node server and the fault probability corresponding to each fault type.

3. The method according to claim 2, characterized in that The method further comprises: Obtain a fault knowledge graph pre-built on the server platform; the fault knowledge graph includes multiple nodes; the nodes include a modal feature node, a fault type node, a device type node, and a feedback information node; the modal feature node corresponds to the multimodal data; the fault type node corresponds to the fault type; the device type node is used to represent the device model of the device in the node server; the device includes a GPU and a fan; the feedback information node is used to represent the feedback of the operation and maintenance personnel on the fault information; The confidence level of the fault information is determined using the fault knowledge graph; the confidence level is used to verify the authenticity of the fault information.

4. The method according to claim 2, characterized in that The modal features of all modalities are time-aligned and feature-fused to construct a feature vector, including: For any target mode among all modes, determine the fluctuation intensity of the target mode within the current acquisition window; the fluctuation intensity is used to represent the degree of change of the modal feature corresponding to the target mode in the current acquisition window; Adjusting the attention weight of the target modality using the fluctuation intensity to obtain a fluctuation weight corresponding to the target modality; The characteristic vector is determined according to the modal features of all modes and the fluctuation weights corresponding to the modes.

5. The method according to claim 1, wherein Mapping the fault information to the thermal image to obtain a fault risk map of the node server includes: Determining, based on the thermal image, a thermal channel activation heat map corresponding to the thermal image; the thermal channel activation heat map is used to represent an image area in the thermal image that is related to the fault information; The fault information is mapped to a thermal imaging map through the thermal channel activation heat zone map to determine the fault risk map of the node server.

6. The method according to claim 1, characterized in that The method further comprises: Receiving feedback from operation and maintenance personnel regarding the fault information; Binding the feedback information with the multimodal data to determine the labeled samples of the node server; Determine the node weight corresponding to each node server by using the device types of multiple node servers under the server platform and the corresponding labeled samples; According to the node weight corresponding to each node server and the model parameters of the fault prediction model corresponding to each node server, the global parameters of the global fault prediction model of the server platform are adjusted; the global fault prediction model is used to indicate the adjustment of the fault prediction model of each node server.

7. The method according to claim 6, characterized in that After receiving feedback from the operation and maintenance personnel regarding the fault information, the method further includes: The fault knowledge graph of the server platform is updated using the fault information of the node server, the modal features corresponding to each modality in the multimodal data, the device type of the node server, and the feedback information of the operation and maintenance personnel on the fault information.

8. The method according to claim 1, characterized in that The method further comprises: When it is detected that any modal data in the multimodal data is missing, reasoning is performed using the association relationship in the pre-built fault knowledge graph to determine the missing modal data; The missing model data is correspondingly filled into the multimodal data.

9. A server failure prediction device based on multimodality, characterized in that: Applied to a server platform, the server platform includes at least one node server, and the device includes: A collection module for collecting multimodal data from the node server; the multimodal data includes at least two of thermal images, system logs, GPU utilization, power consumption, temperature, fan speed, and fan audio; An extraction module, configured to extract modal features corresponding to each modality from the multimodal data; A prediction module, configured to fuse the modal features of all modes in the multimodal data to predict fault information of the node server; A mapping module is used to map the fault information to the thermal image to obtain a fault risk map of the node server; the fault risk map is used to overlay the fault information on the thermal image.

10. A server failure prediction device based on multimodality, characterized in that: The multimodal-based server fault prediction device includes a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the multimodal-based server fault prediction method according to any one of claims 1 to 8.

Images