Fault detection method and device, electronic equipment and storage medium
By acquiring and reconstructing the detection data of the target detection indicators, calculating differences to determine the fault detection results, the timeliness and accuracy of machine fault detection is solved, data processing is simplified and detection efficiency is improved.
Patent Information
- Application Number
- CN202410171866.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-06
- Publication Date
- 2025-08-08
AI Technical Summary
In the prior art, machine fault detection has problems such as insufficient timeliness, high complexity and waste of manpower in multi-equipment joint work scenarios, especially in large-scale training tasks, which are long and difficult to detect.
By obtaining the detection data of the target detection indicators, reconstructing and calculating the differences, using the differences to determine the fault detection results, reducing data noise and processing volume, and improving detection accuracy and timeliness.
It realizes efficient and accurate fault detection, reduces waste of manpower and time, and improves the accuracy and timeliness of fault detection.
Smart Images
Figure CN120448228A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a fault detection method, device, electronic device, and storage medium. Background Art
[0002] When multiple devices are working together, problems can arise due to machine failures. For example, with distributed computing and deep learning algorithms, production environments often face the challenge of "faulty machine detection." Hardware or software failures are common in real-world production environments, forcing entire training tasks to halt until repairs and restarts. Consequently, fault detection methods are a pressing issue. Summary of the Invention
[0003] In view of this, the present disclosure provides a fault detection method, device, electronic device and storage medium to solve the problem of fault detection.
[0004] In a first aspect, the present disclosure provides a fault detection method, the method comprising:
[0005] Obtaining detection data of target detection indicators corresponding to each device to be detected, wherein the target detection indicators are obtained by screening after statistical analysis of detection data of preset detection indicators;
[0006] Reconstructing the detection data of the target detection indicator to obtain target detection data of the target detection indicator;
[0007] For each of the devices to be detected, determining the difference between the target detection data of the target detection indicator and the target detection data of the target detection indicators corresponding to the remaining devices to be detected, so as to obtain the difference of the target detection indicators corresponding to each of the devices to be detected;
[0008] Based on the differences in target detection indicators corresponding to the various devices to be detected, the fault detection results of the various devices to be detected are determined.
[0009] In a second aspect, the present disclosure provides a fault detection device, the device comprising:
[0010] A detection data acquisition module is used to obtain detection data of target detection indicators corresponding to each device to be detected, wherein the target detection indicators are obtained by screening after statistical analysis of the detection data of preset detection indicators;
[0011] A reconstruction processing module, used to reconstruct the detection data of the target detection index to obtain the target detection data of the target detection index;
[0012] a difference determination module, configured to determine, for each of the devices to be detected, a difference between the target detection data of the target detection indicator and the target detection data of the target detection indicators corresponding to the remaining devices to be detected, so as to obtain a difference in the target detection indicators corresponding to the devices to be detected;
[0013] The detection result determination module is used to determine the fault detection result of each of the devices to be detected based on the differences in target detection indicators corresponding to each of the devices to be detected.
[0014] In a third aspect, the present disclosure provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the fault detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the fault detection method of the first aspect or any corresponding embodiment thereof.
[0016] The fault detection method provided by the embodiment of the present disclosure performs fault detection based on the detection data of the target detection index that is screened out after statistical analysis of the detection data based on the preset detection index, which can reduce the data processing volume of the detection data; and also reconstructs the detection data of the target detection index to increase the distinction between the detection data of the fault state and the non-fault state, so as to more accurately perform fault detection. Specifically, when performing fault detection, the differences between the target detection data are utilized to simplify data processing while ensuring the detection effect. This is because the differences between the detection data of the presence of faults and the absence of faults are relatively large. Therefore, fault detection through differences can accurately achieve fault detection. Therefore, this method can improve the accuracy and timeliness of fault detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 is a flowchart of a fault detection method according to an embodiment of the present disclosure;
[0019] Figure 2 is a flowchart of another fault detection method according to an embodiment of the present disclosure;
[0020] Figure 3 is a flowchart of another fault detection method according to an embodiment of the present disclosure;
[0021] Figure 4 is a schematic structural diagram of a reconstruction model according to an embodiment of the present disclosure;
[0022] Figure 5 is a structural block diagram of a fault detection device according to an embodiment of the present disclosure;
[0023] Figure 6 Schematic diagram of the hardware structure of the electronic device according to the embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.
[0025] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0026] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.
[0027] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.
[0028] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0029] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.
[0030] In related technologies, fault detection methods rely on manual analysis of system logs and test data. Specifically, a detailed search of each machine's logs and test data is performed until the exact location of the fault is confirmed. However, this fault detection method has some shortcomings. First, the relevant personnel are not reminded to trigger the diagnosis until the task stops, which leads to insufficient timeliness of the operation. Second, due to the duplication and incompleteness of the recorded log content, the process of finding the faulty machine is very complicated and time-consuming. Finally, the judgment of the faulty machine requires multiple methods such as hardware and software, and requires multiple people to cooperate with each other, resulting in a large amount of unnecessary manpower and time. For larger training tasks, especially ultra-large-scale model training, the detection of faulty machines takes longer and is more difficult.
[0031] Therefore, an automatic fault detection method is very necessary, which has the characteristics of high efficiency and accuracy, so that relevant personnel can receive timely alarms about faulty machines, eliminating the need for manual operation. Based on this, the embodiments of the present disclosure provide a fault detection method with low latency and high accuracy.
[0032] According to an embodiment of the present disclosure, an embodiment of a fault detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0033] In this embodiment, a fault detection method is provided, which can be used in electronic devices such as computers, servers, mobile terminals, etc. Figure 1 is a flow chart of a fault detection method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:
[0034] Step S101: Acquire detection data of target detection indicators corresponding to each device to be detected.
[0035] Among them, the target detection index is obtained by screening after statistical analysis of the detection data based on the preset detection index.
[0036] The number of devices to be tested is determined by the application scenario. For example, if five devices are involved in model training, fault detection will be performed on these five devices. If ten devices are involved in data processing, fault detection will be performed on all ten devices. Therefore, there is no limit on the specific number of devices to be tested. Of course, the type of device to be tested also depends on the application scenario. For example, the device to be tested can be a server or other data processing equipment.
[0037] The preset detection indicators are all detection indicators related to the operating process of the device to be tested. For example, there are 10 preset detection indicators. The target detection indicators are selected from these 10 preset detection indicators, that is, the most representative and practical detection indicators are selected. The specific selection is based on statistical analysis of the detection data of the preset detection indicators. Because statistical analysis can represent the analysis of the detection data of each preset detection indicator, this method can select and process the most representative target detection indicators.
[0038] The test data for a target detection metric is the test result corresponding to the target detection metric, and the test result is represented in the form of data. For example, for the memory usage metric, real-time memory usage is collected and then calculated to obtain the memory usage rate. The calculated result serves as the test data for this target detection metric. For each target detection metric, its test data is collected in real time, and the collection time is recorded during the collection process to facilitate subsequent data analysis.
[0039] It should be noted that the number of target detection indicators used during fault detection is set based on actual needs or determined by the current application scenario. For example, before fault analysis, the preset detection indicators are screened based on the established application scenario to obtain the target detection indicators that are suitable for the current application scenario.
[0040] Based on this, fault detection is performed on M devices to be detected, and each device to be detected corresponds to N target detection indicators. Wherein, M and N are both integers greater than 1.
[0041] Step S102 : reconstructing the detection data of the target detection index to obtain target detection data of the target detection index.
[0042] Reconstruction processing is used to reconstruct the length of input data into an output representation of arbitrary length. For example, reconstruction processing reconstructs detection data in a normal state into similar output data and detection data in a fault state into more significant output data. In other words, reconstruction processing removes noise from the input detection data, compresses it, and reconstructs it into more significant output data.
[0043] The reconstruction process can be implemented based on a pre-trained network model, such as an encoder-decoder model, or a network model based on other structures. No limitation is imposed on the specific implementation method of the reconstruction process.
[0044] For target detection indicators, the detection data is reconstructed to obtain target detection data, that is, after the reconstruction, target detection data of each target detection indicator of each device to be detected is obtained.
[0045] Step S103 : for each device to be detected, determining the difference between the target detection data of the target detection index and the target detection data of the target detection indexes corresponding to the other devices to be detected, so as to obtain the difference of the target detection indexes corresponding to the devices to be detected.
[0046] As shown above, fault detection targets M devices to be tested, each of which has N target detection metrics. For the jth target detection metric of the i-th device to be tested, the difference between the target detection data and the jth target detection metrics of the remaining M-1 devices to be tested is calculated. Therefore, for the jth target detection metric of the i-th device to be tested, M-1 difference values are obtained. Based on this, these M-1 difference values are fused to obtain the difference of the jth target detection metric of the i-th device to be tested.
[0047] The above process is performed on each of the M devices to be detected to obtain the differences of the target detection indicators of each device to be detected. That is, each of the N target detection indicators of each device to be detected has its own corresponding difference.
[0048] Methods for calculating the difference include, but are not limited to, the distance between two target detection data. Since the target detection data is represented in vector form, the difference is represented by the distance between the two vectors. Of course, the difference can also be calculated using other methods, which are not limited here.
[0049] Step S104 : determining the fault detection results of the respective devices to be detected based on the differences in the target detection indicators corresponding to the respective devices to be detected.
[0050] Each target detection indicator can be assigned an indicator threshold. The difference in the target detection indicator is compared with the indicator threshold. If the difference exceeds the indicator threshold, it indicates that the corresponding device under test has failed. Of course, after obtaining the difference in the target detection indicator corresponding to each device under test, other methods can be used to determine the fault detection result, such as using a network model. The specific method for determining the fault detection result is not limited herein.
[0051] The fault detection method provided in this embodiment performs fault detection based on the detection data of the target detection index, which is screened out after statistical analysis of the detection data based on the preset detection index. This method can reduce the data noise and data processing volume of the detection data. It also reconstructs the detection data of the target detection index to increase the distinction between the detection data of the fault state and the non-fault state, so as to more accurately perform fault detection. Specifically, when performing fault detection, the differences between the target detection data are utilized to simplify data processing while ensuring the detection effect. This is because the differences between the detection data of the presence and absence of faults are relatively large. Therefore, fault detection based on the differences can accurately achieve fault detection. Therefore, this method can improve the accuracy and timeliness of fault detection.
[0052] In this embodiment, a fault detection method is provided, which can be used in electronic devices such as computers, servers, mobile terminals, etc. Figure 2 is a flow chart of a fault detection method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:
[0053] Step S201: Acquire detection data of target detection indicators corresponding to each device to be detected.
[0054] Among them, the target detection index is obtained by screening after statistical analysis of the detection data based on the preset detection index.
[0055] For information on how to obtain target detection data, see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0056] In some optional implementations, the target detection index may be determined by:
[0057] Step a1: Acquire detection data of preset detection indicators corresponding to each device to be detected.
[0058] Step a2: for each preset detection indicator, based on the detection data of the preset detection indicator, determine a second statistical analysis result corresponding to the preset detection indicator.
[0059] Step a3: Obtain the fault label of the device to be detected at the sampling time point corresponding to the detection data.
[0060] Step a4: determining a target detection indicator from the preset detection indicators based on the second statistical analysis result corresponding to the preset detection indicators and the fault label.
[0061] For each device to be tested, multiple preset test indicators are included. These multiple preset test indicators are screened to obtain N target test indicators. Specifically, since the processing methods for each device to be tested are similar, the following description uses a single device to be tested as an example. Test data for the preset test indicators corresponding to the device to be tested is obtained. For example, if the device to be tested includes 15 preset test indicators, test data for each preset test indicator is obtained and the sampling time of the test data is recorded.
[0062] For example, consider M devices to be tested and 15 preset test indicators, from which N target test indicators are screened. Since the processing objective is to screen the target test indicators, the test data is analyzed based on the dimensions of the preset test indicators. Specifically, statistical analysis is performed on the test data for each preset test indicator of each device to be tested, yielding a second statistical analysis result for each preset test indicator. The second statistical analysis result includes, but is not limited to, the mean, variance, or standard deviation of the test data.
[0063] Because the same test data can be significantly different in a faulty state and a non-faulty state, screening target test indicators based on the fault labels at the sampling time points corresponding to each test data point can improve the accuracy of test indicator screening. The fault labels at the sampling time points can be manually annotated and include both faulty and non-faulty data.
[0064] As described above, for the same detection indicator, a clear distinction must be made between the fault state and the non-fault state. Therefore, by comparing the second statistical analysis results between the fault state and the non-fault state, the preset detection indicator with a clear distinction is determined as the target detection indicator. Of course, other methods can also be used to screen the preset detection indicators, and this is not intended to be limiting.
[0065] The screening of preset detection indicators is based on the second statistical analysis results and fault labels corresponding to the preset detection indicators. Since the statistical analysis results can characterize the changes in the detection data, they indicate that the target detection indicators obtained by screening are the most representative and practical.
[0066] In some optional implementations, the above step a4 includes:
[0067] Step a41 : For each preset detection indicator, based on the second statistical analysis results corresponding to the preset detection indicators of all devices to be detected, an analysis result of the preset detection indicator is obtained.
[0068] Step a42 , screening out the maximum analysis result corresponding to each preset detection indicator and the optional to-be-detected device corresponding to the maximum analysis result from all analysis results.
[0069] Step a43: extracting the fault label of the optional device to be detected at the sampling time point corresponding to the maximum analysis result.
[0070] Step a44 : Based on the maximum analysis results corresponding to the preset detection indicators and the fault labels, the maximum analysis results corresponding to all preset detection indicators are classified, and a target detection indicator is determined from the preset detection indicators.
[0071] Specifically, for the jth preset detection index of the i-th device to be detected, the z score z of the jth preset detection index of the i-th device to be detected is obtained through its corresponding second statistical analysis result: ij ,Right now:
[0072] z ij =(x ij -average j ) / std j
[0073] Among them, x ij is the test data of the jth preset test index of the i-th device to be tested, average j is the mean value of all devices to be tested on the jth preset detection index, std j is the standard deviation of all devices to be tested on j preset detection indicators.
[0074] Based on this, the jth preset detection index of the i-th device to be tested obtains the corresponding z score z ij , for the same preset detection indicator, the maximum z score is screened out to obtain the maximum analysis result corresponding to each preset detection indicator. Accordingly, the optional device to be detected corresponding to the maximum analysis result is obtained. When no fault occurs, the detection data of each device to be detected has similar values and fluctuations during training, so the overall z score is low. However, when a fault occurs, the affected device to be detected will exhibit abnormal behavior, resulting in a higher z score on some abnormal detection data. Therefore, for each preset detection indicator, the maximum z score of all devices to be detected in this indicator is used to indicate the degree of similarity between the devices to be detected.
[0075] Since the collection of detection data records the collection time point, and the fault label at the collection time point represents the fault state, the maximum analysis result corresponding to the preset detection indicator and the fault label are combined, and the maximum analysis results corresponding to all preset detection indicators are classified through random forest analysis, and the target detection indicator is determined among the preset detection indicators.
[0076] Specifically, a random forest approach is used to determine the importance of each pre-set detection metric in fault detection, based on the maximum z-score of each pre-set detection metric within each time window. At each sampling time point, the maximum z-score of each pre-set detection metric is combined as a sample. Based on whether there are faulty machines within this time window, each machine is labeled as normal or abnormal to obtain a fault label. This sample and label are then used to train a random forest using supervised learning to select the target detection metric.
[0077] There are two main reasons for using random forests for metric selection. First, the logical structure of random forests is similar to rule-based strategies commonly used in network detection systems. For example, simple threshold rules (when certain detection data drops below a certain value) are used to determine whether a device under detection is abnormal. Second, random forests, with their decision-making power and direct representation capabilities, provide high expressiveness and high fidelity for detection metrics.
[0078] Starting from the root node, the random forest analyzes the z-scores of the detection data hop by hop to classify the samples. Nodes closer to the root node indicate that the corresponding detection metric is more useful for fault detection. Based on importance, target detection metrics closer to the root node are prioritized. Examples include CPU utilization, GPU utilization, GPU power consumption, GPU graphics engine utilization, GPU tensor core utilization, PFC packet sending rate, and network interface card bandwidth.
[0079] For each preset detection indicator, the maximum analysis result of all devices to be detected in the indicator is used to represent the degree of similarity between the devices to be detected. The maximum analysis result is then classified in combination with the fault label to obtain the most representative target detection indicator.
[0080] Step S202: reconstruct the detection data of the target detection index to obtain the target detection data of the target detection index. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0081] Step S203 : for each device to be detected, determining the difference between the target detection data of the target detection index and the target detection data of the target detection indexes corresponding to the other devices to be detected, so as to obtain the difference of the target detection indexes corresponding to the devices to be detected.
[0082] Specifically, the above step S203 includes:
[0083] Step S2031 , for the current device to be detected, distance calculation is performed based on the target detection data of the target detection index and the target detection data of the target detection indexes corresponding to the other devices to be detected, to obtain a distance calculation result of the target detection data between the current device to be detected and the other devices to be detected.
[0084] Since the target detection indicators are specific to each device under test, the processing process of one device under test is used as an example for convenience. The device under test currently being processed is referred to as the current device under test, and all devices under test other than the current device under test are referred to as the remaining devices under test.
[0085] For the target detection index j of the current device to be detected, the distance between the target detection data and the target detection data of the remaining devices to be detected is calculated. This calculates the distance between the target detection data of the current device to be detected and each of the remaining devices to be detected. This distance calculation result corresponds to the target detection index j. Distance calculation methods include, but are not limited to, Euclidean distance, Manhattan distance, Mahalanobis distance, and others, and are set based on actual needs and are not limited to any of them here.
[0086] Step S2032 , performing normalization processing based on the distance calculation results of the target detection data between the current device to be detected and the remaining devices to be detected, to obtain the difference of the target detection index corresponding to the current device to be detected.
[0087] The normalization result is used to characterize the variability of the target detection metrics for the device under test. If the current application scenario has M devices under test, then the target detection metric j for the device under test has M-1 distance calculation results. These M-1 distance calculation results are normalized to determine the variability of the target detection metrics for the device under test. Normalization methods include, but are not limited to, mean and variance calculations.
[0088] In some optional implementations, the above step S2032 includes:
[0089] Step b1: performing statistical analysis based on the distances of target detection data between the current device to be detected and the remaining devices to be detected to obtain a first statistical analysis result, the first statistical analysis result including a distance mean and a distance standard deviation.
[0090] Step b2: based on the distance calculation result of the target detection data between the current device to be detected and the other devices to be detected and the first statistical analysis result, obtain the difference of the target detection index corresponding to the current device to be detected.
[0091] For the current device to be detected, the z-score of the sum of the distance calculation results of the target detection data between it and each of the remaining devices to be detected is used as the result of the normalization process. The calculation of the z-score is related to the distance mean and distance standard deviation. Based on this, it is necessary to first perform a statistical analysis on the distance calculation results of the target detection data to obtain a first statistical analysis result. Then, combining the first statistical analysis result with the distance calculation result, a z-score of the sum of the distance calculation results of the target detection data between the current device to be detected and each of the remaining devices to be detected is obtained. This z-score is used as the difference in the target detection indicator corresponding to the current device to be processed.
[0092] Based on the distance calculation results, the normalization results are obtained by combining statistical analysis. Since statistical analysis can integrate the data features of all target detection data, the accuracy of the difference of target detection indicators is further improved.
[0093] Step S204 : determining the fault detection results of the respective devices to be detected based on the differences in the target detection indicators corresponding to the respective devices to be detected.
[0094] Specifically, the above step S204 includes:
[0095] Step S2041 , comparing the differences of target detection indicators corresponding to the devices to be detected, and obtaining the maximum difference of the target detection indicators.
[0096] The target detection index j corresponding to each device to be detected obtains a corresponding difference calculation result. By comparing the M difference calculation results, the maximum difference of the target detection index j and the device to be detected corresponding to the maximum difference are obtained.
[0097] Step S2042: If the maximum difference is greater than the difference threshold corresponding to the target detection index, the number of times the target detection device corresponding to the maximum difference has a fault is recorded.
[0098] The device to be detected corresponding to the maximum difference may be a faulty device. Therefore, the maximum difference is compared with the difference threshold corresponding to the target detection index j. If it exceeds the difference threshold, the target detection device corresponding to the maximum difference is considered a candidate faulty device, and the number of times the target detection device has failed is recorded.
[0099] Step S2043: If the target device to be detected has continuous faults and the number of faults exceeds the fault threshold, the target device to be detected is determined to be a faulty device.
[0100] A candidate fault device detected by a single sliding time window may be a false alarm due to some transient counting error. Therefore, the continuity characteristic of the fault duration is used to further detect the candidate fault devices. That is, a sliding time window with a certain step size is used to sequentially detect the fault candidate devices within each window. If the number of consecutive detections of the same fault candidate exceeds the continuity threshold, it is considered to be a true fault device. This is because fault performance typically persists for a period of time before terminating. For example, an appropriate continuity threshold can be set to 4 or 5 minutes, and the fault threshold is accordingly derived by combining the duration of a single fault detection. This is sufficient to filter out transient noise and does not exceed the duration of the fault performance.
[0101] The fault detection method provided in this embodiment determines the distance between any two target detection data points for each target detection indicator to represent the similarity between the two target detection data points. Normalization is then performed based on the similarity to obtain a more accurate representation of the differences in the target detection indicators. Because a single fault can be caused by fluctuations, continuous fault detection is performed on the faulty target device to more accurately identify the faulty device.
[0102] In this embodiment, a fault detection method is provided, which can be used in electronic devices such as computers, servers, mobile terminals, etc. Figure 3 is a flow chart of a fault detection method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:
[0103] Step S301: Acquire detection data of target detection indicators corresponding to each device to be detected.
[0104] Among them, the target detection index is obtained by screening after statistical analysis of the detection data based on the preset detection index.
[0105] Specifically, the above step S301 includes:
[0106] Step S3011: Acquire original detection data of each device to be detected.
[0107] The original detection data of each device to be detected is obtained after performing data detection on each target detection index of the device to be detected. For example, according to each target detection index, detection data of the device to be detected is collected to obtain the original detection data.
[0108] Step S3012: align and fill the original detection data to obtain initial detection data.
[0109] Since different target detection indicators of the same device to be detected may have problems such as timestamp asynchrony when collecting data, the original detection data obtained by different collection methods need to be aligned and padded to ensure that the initial detection data obtained by different collection methods have the same length and timestamp.
[0110] Step S3013 , performing sampling time alignment and normalization processing on the initial detection data of each target detection indicator to obtain the detection data of the target detection indicator.
[0111] Initial detection data for the same target detection metric from different devices under test may also have issues with timestamp misalignment and inconsistent data magnitudes. To address this, the initial detection data is sampled and normalized to ensure that the multi-dimensional initial detection data is integrated into evenly distributed data, resulting in the target detection metric's detection data.
[0112] Step S302 : reconstructing the detection data of the target detection index to obtain target detection data of the target detection index.
[0113] Specifically, the above step S302 includes:
[0114] Step S3021: Determine the corresponding reconstruction model based on the type of target detection indicator.
[0115] The reconstruction model is used to reconstruct input data of the first dimension into output data of the second dimension.
[0116] The reconstruction model is a pre-trained network model whose input is the target detection indicator's detection data, and whose output is the reconstructed target detection data. The reconstruction model corresponds to the type of target detection indicator. Different types of target detection indicators use different reconstruction models, while the same type of target detection indicators uses the same reconstruction model.
[0117] In some optional embodiments, the target detection indicator includes: a first detection indicator and a second detection indicator set, wherein the second detection indicator set includes multiple detection indicators of the same type. The reconstruction model includes a first reconstruction model corresponding to the first detection indicator and a second reconstruction model corresponding to the second detection indicator set.
[0118] For example, the first detection indicator includes CPU occupancy, PFC packet transmission rate, and network card bandwidth. Accordingly, the first detection indicator corresponds to the reconstruction model one-to-one. The second detection indicator set is a set of multiple detection indicators, for example, GPU utilization, GPU power consumption, GPU graphics engine utilization, and GPU tensor core utilization. These four detection indicators correspond to one reconstruction model. For ease of distinction, the reconstruction model corresponding to the first detection indicator is referred to as the first reconstruction model, and the reconstruction model corresponding to the second detection indicator set is referred to as the second reconstruction model.
[0119] The same reconstruction model is used to reconstruct the same type of detection indicators, which reduces the number of reconstruction models and improves processing efficiency.
[0120] In some optional embodiments, the training method of the reconstruction model includes:
[0121] Step c1: acquiring sample detection data corresponding to each target detection indicator, wherein the sample detection data includes first sample detection data of a fault state and second sample detection data of a non-fault state.
[0122] In step c2, the sample detection data is input into the encoder of the initial reconstruction model corresponding to the target detection index for encoding to obtain an encoding vector, and then the encoding is connected and input into the decoder of the initial reconstruction model to obtain a reconstruction vector.
[0123] Step c3: Update the model parameters of the initial reconstruction model based on the reconstruction vector to obtain a reconstruction model corresponding to the target detection index.
[0124] When training the reconstruction model, the first reconstruction model is trained independently. Specifically, the normalized time series detection data within a time window is used as input. For example, when training the first reconstruction model corresponding to CPU usage, the normalized CPU usage detection data is segmented using a sliding time window of a certain length (n) and step size (m). Therefore, multiple 1*n vectors are used as input for training the model.
[0125] When training the second reconstruction model, the detection data for each target detection metric in the second detection metric set is used as input. Specifically, this data is used as input, with a size of 4*n, to train the second reconstruction model corresponding to the GPU. Therefore, four reconstruction models are obtained by training for three first detection metrics and one second detection metric set, each corresponding to the type of target detection metric.
[0126] in, Figure 4The structure of the reconstruction model is shown, including an encoder and a decoder. The encoder extracts the input sequence as an intermediate vector z. Subsequently, the decoder uses the intermediate vector z to restore the data to a new dimension and output it as a reconstruction of the distribution. It should be noted that various statistical methods or machine learning techniques can be used for the encoder and decoder. Given the temporal continuity of the sampling sequence, a long short-term memory network (LSTM) is used as an encoder and decoder to extract temporal features. In order to achieve fault detection, the detection data from the device to be detected is used as the input of the reconstruction model, and the output reconstructed data is used to perform subsequent detection of the faulty device.
[0127] The trained model reconstructs non-faulty detection data into similar reconstructed data, while input faulty detection data is reconstructed into more significant abnormal reconstructed data. Therefore, the reconstruction model can be used to enhance the accuracy and robustness of anomaly detection in the absence of labels.
[0128] When training the reconstruction model, data processing is performed based on the encoder and decoder, and the training process enhances the accuracy and robustness of anomaly detection in the absence of labels.
[0129] It should be noted that in Figure 4 The first dimension shown in the figure is n, and the second dimension is also n, which are the same. However, the first dimension and the second dimension can also be different. Figure 4 Just an example.
[0130] Step S3022: input the detection data of the target detection index into the corresponding reconstruction model to obtain the target detection data of the target detection index.
[0131] In determining the reconstruction model, the detection data of the target detection index is input into the reconstruction model, and after being processed by the reconstruction model, the target detection data of the target detection index is obtained.
[0132] Step S303: for each device to be detected, determine the difference between the target detection data of the target detection index and the target detection data of the target detection index corresponding to the other devices to be detected, so as to obtain the difference of the target detection index corresponding to each device to be detected. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.
[0133] Step S304: Determine the fault detection results of each device to be detected based on the differences in the target detection indicators corresponding to each device to be detected. Figure 2 Step S204 of the illustrated embodiment will not be described in detail here.
[0134] The fault detection method provided in this embodiment performs alignment and padding after acquiring raw detection data values to eliminate data synchronization issues. For each target detection metric, sampling time alignment is performed, followed by normalization techniques to ensure that multidimensional detection data is integrated into evenly distributed data, enhancing the performance of deep learning algorithms. Data reconstruction using a reconstruction model simplifies the data reconstruction process and improves processing efficiency. Reconstruction is used to reconstruct input data into an output representation of arbitrary length without losing the original features.
[0135] As a specific application embodiment of the embodiment of the present disclosure, in the training process of a large-scale network model, 10 servers are involved, and the detection data of the target detection index is detected for each server. Among them, the detection can be timed triggering, or real-time detection, etc. After obtaining the detection data of the target detection index in each server, it is reconstructed to obtain the target detection data. Then, a similarity and continuity fault detection is performed on it to determine the server that currently has a fault. Since the detection data of the target detection index is used for analysis when performing fault detection, then after determining the server with a fault, the fault can also be located by the target detection index with a fault, thereby improving the efficiency of fault location.
[0136] This embodiment also provides a fault detection device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.
[0137] This embodiment provides a fault detection device, such as Figure 5 Shown, including:
[0138] The detection data acquisition module 501 is used to obtain detection data of target detection indicators corresponding to each device to be detected. The target detection indicators are obtained by screening after statistical analysis of detection data based on preset detection indicators.
[0139] The reconstruction processing module 502 is used to reconstruct the detection data of the target detection index to obtain the target detection data of the target detection index.
[0140] The difference determination module 503 is used to determine the difference between the target detection data of the target detection index of each device to be detected and the target detection data of the target detection indexes corresponding to the other devices to be detected, so as to obtain the difference of the target detection indexes corresponding to the devices to be detected.
[0141] The detection result determination module 504 is configured to determine the fault detection result of each device to be detected based on the differences in target detection indicators corresponding to each device to be detected.
[0142] In some optional implementations, the difference determination module 503 includes:
[0143] The distance calculation unit is used to perform distance calculation on the target detection data of the current device to be detected based on the target detection index and the target detection data of the target detection indicators corresponding to the other devices to be detected, and obtain the distance calculation result of the target detection data between the current device to be detected and the other devices to be detected.
[0144] The normalization processing unit is used to perform normalization processing based on the distance calculation result of the target detection data between the current device to be detected and the other devices to be detected, so as to obtain the difference of the target detection index corresponding to the current device to be detected.
[0145] In some optional implementations, the normalization processing unit includes:
[0146] The statistical analysis subunit is used to perform statistical analysis based on the distance of the target detection data between the current device to be detected and the remaining devices to be detected to obtain a first statistical analysis result, which includes a distance mean and a distance standard deviation.
[0147] The difference determination subunit is used to obtain the difference of the target detection index corresponding to the current device to be detected based on the distance calculation result of the target detection data between the current device to be detected and the other devices to be detected and the first statistical analysis result.
[0148] In some optional implementations, the detection result determination module 504 includes:
[0149] The comparison unit is used to compare the differences of the target detection indicators corresponding to the devices to be detected, and obtain the maximum difference of the target detection indicators.
[0150] The recording unit is configured to record the number of times the target detection device corresponding to the maximum difference has failed if the maximum difference is greater than a difference threshold corresponding to the target detection index.
[0151] The faulty device determining unit is configured to determine that the target device to be detected is a faulty device if the target device to be detected continuously has faults and the number of faults exceeds a fault threshold.
[0152] In some optional implementations, the target detection index determination module includes:
[0153] The detection data acquisition unit is used to obtain detection data of preset detection indicators corresponding to each device to be detected.
[0154] The statistical analysis result determining unit is configured to determine, for each preset detection indicator, a second statistical analysis result corresponding to the preset detection indicator based on the detection data of the preset detection indicator.
[0155] The fault label acquisition unit is used to obtain the fault label of the device to be detected at the sampling time point corresponding to the detection data.
[0156] The target detection index determining unit is configured to determine the target detection index from the preset detection indicators based on the second statistical analysis result corresponding to the preset detection indicators and the fault label.
[0157] In some optional implementations, the target detection index determination unit includes:
[0158] The analysis result determination subunit is configured to obtain, for each preset detection indicator, an analysis result of the preset detection indicator based on the second statistical analysis results corresponding to the preset detection indicators of all devices to be detected.
[0159] The maximum analysis result determination subunit is used to filter out the maximum analysis result corresponding to each preset detection indicator and the optional to-be-detected device corresponding to the maximum analysis result from all analysis results.
[0160] The fault label extraction subunit is used to extract the fault label of the optional device to be detected at the sampling time point corresponding to the maximum analysis result.
[0161] The classification subunit is used to classify the maximum analysis results corresponding to all preset detection indicators based on the maximum analysis results corresponding to the preset detection indicators and the fault labels, and determine the target detection indicator among the preset detection indicators.
[0162] In some optional implementations, the reconstruction processing module 502 includes:
[0163] The reconstruction model determination unit is used to determine the corresponding reconstruction model based on the type of target detection index, and the reconstruction model is used to reconstruct the input data of the first dimension into the output data of the second dimension.
[0164] The reconstruction processing unit is used to input the detection data of the target detection index into the corresponding reconstruction model to obtain the target detection data of the target detection index.
[0165] In some optional embodiments, the training device for reconstructing the model includes:
[0166] The sample detection data acquisition module is used to acquire sample detection data corresponding to each target detection indicator, and the sample detection data includes first sample detection data in a fault state and second sample detection data in a non-fault state.
[0167] The encoding module is used to input the sample detection data into the encoder of the initial reconstruction model corresponding to the target detection index to encode it to obtain an encoding vector, and then input the encoding into the decoder of the initial reconstruction model to obtain a reconstruction vector.
[0168] The parameter updating module is used to update the model parameters of the initial reconstruction model based on the reconstruction vector to obtain the reconstruction model corresponding to the target detection index.
[0169] In some optional embodiments, the target detection indicator includes: a first detection indicator and a second detection indicator set, the second detection indicator set including multiple detection indicators of the same type; the reconstruction model includes a first reconstruction model corresponding one-to-one to the first detection indicator and a second reconstruction model corresponding to the second detection indicator set.
[0170] In some optional implementations, the detection data acquisition module 501 includes:
[0171] The detection data acquisition unit is used to obtain the original detection data of each device to be detected.
[0172] The alignment processing unit is used to perform alignment and padding processing on the original detection data to obtain initial detection data.
[0173] The normalization processing unit is used to perform sampling time alignment and normalization processing on the initial detection data of each target detection indicator to obtain the detection data of the target detection indicator.
[0174] The fault detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0175] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0176] The present disclosure also provides an electronic device having the above Figure 5 The fault detection device shown.
[0177] See also Figure 6 , Figure 6 is a structural diagram of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 6As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.
[0178] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.
[0179] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0180] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0181] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.
[0182] The electronic device further includes a communication interface 30 for the electronic device to communicate with other devices or a communication network.
[0183] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.
[0184] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A fault detection method, characterized in that: The method comprises: Acquire detection data of target detection indicators corresponding to each device to be detected, wherein the target detection indicators are obtained by screening after statistical analysis of detection data of preset detection indicators; Reconstructing the detection data of the target detection indicator to obtain target detection data of the target detection indicator; For each of the devices to be detected, determining the difference between the target detection data of the target detection indicator and the target detection data of the target detection indicators corresponding to the remaining devices to be detected, so as to obtain the difference of the target detection indicators corresponding to each of the devices to be detected; Based on the differences in target detection indicators corresponding to the various devices to be detected, the fault detection results of the various devices to be detected are determined.
2. The method according to claim 1, characterized in that The step of determining, for each of the devices to be detected, a difference between the target detection data of the target detection indicator and the target detection data of the target detection indicators corresponding to the remaining devices to be detected, so as to obtain the difference of the target detection indicators corresponding to the devices to be detected, includes: For the current device to be detected, performing distance calculation based on the target detection data of the target detection indicator and the target detection data of the target detection indicators corresponding to the remaining devices to be detected, to obtain a distance calculation result of the target detection data between the current device to be detected and the remaining devices to be detected; Normalization processing is performed based on the distance calculation result of the target detection data between the current device to be detected and the remaining devices to be detected to obtain the difference of the target detection index corresponding to the current device to be detected.
3. The method according to claim 2, characterized in that The normalizing process based on the distance calculation result of the target detection data between the current device to be detected and the remaining devices to be detected to obtain the difference of the target detection index corresponding to the current device to be detected includes: Performing statistical analysis based on the distances of the target detection data between the current device to be detected and the remaining devices to be detected to obtain a first statistical analysis result, the first statistical analysis result including a distance mean and a distance standard deviation; Based on the distance calculation result of the target detection data between the current device to be detected and the remaining devices to be detected and the first statistical analysis result, the difference of the target detection index corresponding to the current device to be detected is obtained.
4. The method according to claim 1, wherein The determining of the fault detection results of each of the devices to be detected based on the differences in target detection indicators corresponding to each of the devices to be detected includes: Comparing the differences of the target detection indicators corresponding to the devices to be detected to obtain the maximum difference of the target detection indicators; If the maximum difference is greater than the difference threshold corresponding to the target detection index, then the number of times the target detection device corresponding to the maximum difference has a fault is recorded; If the target device to be detected has faults continuously and the number of faults exceeds a fault threshold, the target device to be detected is determined to be a faulty device.
5. The method according to claim 1, wherein Methods for determining the target detection index include: Obtaining the test data of the preset test indicators corresponding to each device to be tested; For each of the preset detection indicators, determining a second statistical analysis result corresponding to the preset detection indicator based on the detection data of the preset detection indicator; Obtaining a fault label of the device to be detected at a sampling time point corresponding to the detection data; Based on the second statistical analysis result corresponding to the preset detection indicator and the fault label, the target detection indicator is determined from the preset detection indicators.
6. The method according to claim 5, characterized in that The determining the target detection indicator from the preset detection indicators based on the second statistical analysis result corresponding to the preset detection indicator and the fault label includes: For each of the preset detection indicators, based on the second statistical analysis results corresponding to the preset detection indicators of all the devices to be detected, an analysis result of the preset detection indicator is obtained; Filtering out the maximum analysis result corresponding to each of the preset detection indicators and the optional to-be-detected device corresponding to the maximum analysis result from all the analysis results; Extracting the fault label of the optional device to be detected at the sampling time point corresponding to the maximum analysis result; Based on the maximum analysis result corresponding to the preset detection index and the fault label, the maximum analysis results corresponding to all the preset detection indicators are classified, and the target detection index is determined among the preset detection indicators.
7. The method according to claim 1, characterized in that The reconstructing the detection data of the target detection indicator to obtain the target detection data of the target detection indicator includes: Determining a corresponding reconstruction model based on the type of the target detection indicator, wherein the reconstruction model is used to reconstruct input data of the first dimension into output data of the second dimension; The detection data of the target detection index is input into the corresponding reconstruction model to obtain the target detection data of the target detection index.
8. The method according to claim 7, characterized in that The training method of the reconstruction model includes: Acquire sample detection data corresponding to each of the target detection indicators, the sample detection data including first sample detection data in a fault state and second sample detection data in a non-fault state; Input the sample detection data into the encoder of the initial reconstruction model corresponding to the target detection index to obtain an encoding vector, and then input the encoding vector into the decoder of the initial reconstruction model to obtain a reconstruction vector; The model parameters of the initial reconstruction model are updated based on the reconstruction vector to obtain a reconstruction model corresponding to the target detection index.
9. The method according to claim 7, characterized in that The target detection indicators include: a first detection indicator and a second detection indicator set, the second detection indicator set including multiple detection indicators of the same type; the reconstruction model includes a first reconstruction model corresponding one-to-one to the first detection indicator and a second reconstruction model corresponding to the second detection indicator set.
10. The method according to any one of claims 1 to 9, characterized in that The step of obtaining the detection data of the target detection indicators corresponding to each device to be detected includes: Obtaining original detection data of each of the devices to be detected; Performing alignment and padding processing on the original detection data to obtain initial detection data; The initial detection data of each target detection indicator is sampled and time-aligned and normalized to obtain the detection data of the target detection indicator.
11. A fault detection device, characterized in that: The device comprises: A detection data acquisition module is used to obtain detection data of target detection indicators corresponding to each device to be detected, wherein the target detection indicators are obtained by screening after statistical analysis of the detection data of preset detection indicators; A reconstruction processing module, used to reconstruct the detection data of the target detection index to obtain the target detection data of the target detection index; a difference determination module, configured to determine, for each of the devices to be detected, a difference between the target detection data of the target detection indicator and the target detection data of the target detection indicators corresponding to the remaining devices to be detected, so as to obtain a difference in the target detection indicators corresponding to the devices to be detected; The detection result determination module is used to determine the fault detection result of each of the devices to be detected based on the differences in target detection indicators corresponding to each of the devices to be detected.
12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the fault detection method according to any one of claims 1 to 10 by executing the computer instructions.
13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the fault detection method according to any one of claims 1 to 10.