A memory device failure prediction method, apparatus, medium, and product

Abstract

The application discloses a memory device fault prediction method and device, medium and product, relates to the technical field of storage, and obtains log files collected by multiple management controllers of a device, that is, obtains multiple-source log files, so as to cover more information sources related to memory device faults; event sequences of the multiple-source log files are aligned to obtain an aligned event set, storage unit features of a target storage device and first associated features of the storage unit features are extracted from the aligned event set, and multiple fault prediction models different in input data are called to predict faults of the target storage device based on the storage unit features and the first associated features, so as to solve the problem that in related technologies, only storage unit errors are focused on, the detection rate of potential errors is low, the detection probability of abnormal information of the storage device is improved, the fault prediction result of the target storage device is determined according to the output result of the fault prediction model, and therefore the accuracy of the fault prediction result of the storage device is improved.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a method, device, medium, and product for predicting storage device failures. Background Technology

[0002] Failures in storage devices such as RAM and solid-state storage can lead to serious consequences, including data loss. Therefore, it is necessary to predict potential failure scenarios for these storage devices. Related technologies predict failures by monitoring the number of memory cell errors (such as correctable errors, CEs). However, this method has extremely low coverage of potential storage device failures, meaning it cannot accurately predict many scenarios where storage devices are about to fail.

[0003] Improving the accuracy of fault prediction for storage devices is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a method, device, medium, and product for predicting storage device failures, in order to at least solve the problem of low accuracy in predicting storage device failures in related technologies.

[0005] This invention provides a method for predicting storage device failures, comprising:

[0006] Obtain log files collected by multiple management controllers of the device;

[0007] Extract the event sequences corresponding to the log files, and align multiple event sequences to obtain an aligned event set;

[0008] Extract the storage cell features of the target storage device and the first associated feature of the storage cell features from the alignment event set;

[0009] Multiple fault prediction models are invoked to predict the fault of the target storage device based on the characteristics of the storage cell and the first associated feature, and the fault prediction result of the target storage device is determined according to the output of the fault prediction model.

[0010] The input data for different fault prediction models are different.

[0011] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described memory device fault prediction methods when executing the computer program.

[0012] The present invention also provides a non-volatile storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described storage device fault prediction methods.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described storage device fault prediction methods.

[0014] This invention obtains log files collected by multiple management controllers of the device, i.e., multi-source log files, covering more information sources related to storage device faults. The event sequences of the multi-source log files are aligned to obtain an aligned event set. Storage cell features and first associated features of the storage cell features of the target storage device are extracted from the aligned event set. Multiple fault prediction models with different input data are then invoked to predict the faults of the target storage device based on the storage cell features and the first associated features. This solves the problem in related technologies where focusing only on storage cell errors leads to a low detection rate of potential errors, improving the detection probability of abnormal information in storage devices. The fault prediction result of the target storage device is determined based on the output of the fault prediction model, thereby improving the accuracy of the fault prediction result of the storage device. Attached Figure Description

[0015] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a storage device fault prediction method provided in an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a log collection layer provided in an embodiment of the present invention;

[0018] Figure 3 A flowchart for event sequence alignment of multi-source logs is provided as an embodiment of the present invention;

[0019] Figure 4 A flowchart illustrating the workflow of an event sequence alignment engine provided in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram illustrating the propagation path of a memory fault according to an embodiment of the present invention;

[0021] Figure 6 An architecture diagram of a fault prediction model provided in an embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of a correctable error count decay provided in an embodiment of the present invention;

[0023] Figure 8 A flowchart for determining a fault activation event provided in an embodiment of the present invention;

[0024] Figure 9 A flowchart for fault prediction based on the duration of a fault activation event is provided for an embodiment of the present invention.

[0025] Figure 10 A schematic diagram illustrating the microscopic mechanism of the increment of the duration of a fault activation event, provided for an embodiment of the present invention;

[0026] Figure 11 A flowchart of a fault prediction process based on the incremental duration of a fault activation event, provided for an embodiment of the present invention;

[0027] Figure 12 A flowchart of a fault prediction process based on the incremental duration of a fault activation event, provided for an embodiment of the present invention;

[0028] Figure 13 A path diagram of a temperature-induced cascading failure provided in an embodiment of the present invention;

[0029] Figure 14 An associated fault propagation path diagram provided in an embodiment of the present invention;

[0030] Figure 15 This is a temperature monitoring architecture diagram provided for an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0032] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0033] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Failures of storage devices in equipment are frequent and destructive, leading to significant losses. Therefore, it is necessary to detect storage device failures in advance and replace them before they occur.

[0035] In related technologies, most methods predict storage device failures based on logs used to monitor storage devices. For example, for memory, memory failures are typically predicted by recording the number of correctable and uncorrectable errors in the Machine Check Exception (MCE) log (mcelog). However, in practical applications, it has been found that predicting memory failures solely based on the mcelog and set memory error thresholds can only cover a small number of potential failures, failing to detect most potential memory failures.

[0036] To address this issue, the storage device fault prediction method provided in this embodiment of the invention first obtains log files collected by multiple management controllers of the device from the perspective of information sources, and aligns the event sequences of the multiple source log files to obtain an aligned event set, thus solving the problem of low detection rate of potential storage device faults when analyzing only a single log in related technologies. Next, based on the aligned time set, this embodiment of the invention calls multiple fault prediction models to predict the faults of the target storage device based on storage cell characteristics and a first correlation feature, and these models have different input data, that is, analyzing the potential faults of the storage device from multiple perspectives, solving the problem of low detection rate of potential errors when focusing only on storage cell errors in related technologies, improving the detection probability of abnormal information of storage devices, and determining the fault prediction result of the target storage device based on the output results of multiple fault prediction models, thereby improving the accuracy of the fault prediction result of the storage device.

[0037] The embodiments of the present invention provide a method for predicting storage device failures. The method is described in detail below in conjunction with the execution flow of the storage device failure prediction method.

[0038] Figure 1 This is a flowchart of a storage device fault prediction method provided in an embodiment of the present invention.

[0039] like Figure 1 As shown, the storage device fault prediction method provided in this embodiment of the invention may include: S101: acquiring log files collected by multiple management controllers of the device.

[0040] S102: Extract the event sequence corresponding to the log file, and align multiple event sequences to obtain an aligned event set.

[0041] S103: Extract the storage cell features of the target storage device and the first associated feature of the storage cell features from the self-aligned event set.

[0042] S104: Invoke multiple fault prediction models to perform fault prediction of the target storage device based on storage cell characteristics and first associated characteristics, and determine the fault prediction result of the target storage device according to the output of the fault prediction model.

[0043] The input data for different fault prediction models are different.

[0044] The storage device fault prediction method provided in this embodiment of the invention can be applied to one or more management controllers in a device containing the target storage device to be monitored, or it can be applied to another monitoring device outside of that device. If applied to a management controller in the device, the management controller can be a management controller used to collect log files related to the target storage device, or it can be another type of management controller.

[0045] In this embodiment of the invention, the target storage device can be at least one of memory, solid-state drive, and video memory, or it can be other storage devices in the device.

[0046] In related technologies, fault prediction of storage devices relies solely on logs from a single source, resulting in insufficient data sources. Therefore, in this embodiment of the invention, log files collected by multiple management controllers are used to increase the number of data sources.

[0047] Figure 2 This is a schematic diagram of a log collection layer provided in an embodiment of the present invention. Figure 2 As shown, multi-source logs can be collected based on the log collection layer.

[0048] In this embodiment of the invention, for S101, the log file may include the device's in-band log and the device's out-of-band log. If the target storage device is memory, the in-band log may include at least one of the following: display message log (dmesg), system operation message log (messages), and machine check exception log (mcelog); the out-of-band log may include at least one of the following: system event log (SEL), intelligent platform management interface debug log (IPMIDebug Log (IDL), analysis result report (analyResultReport.json), and black box log.

[0049] The Display Message Log is a command used to display or control kernel messages. These messages typically include hardware detection information during system startup, device driver loading information, and system runtime error messages. The System Runtime Message Log is a log file, usually located at ` / var / log / messages`. It records various messages during system runtime, including hardware events, software errors, and system warnings. These messages help system administrators monitor system status and troubleshoot problems. The Machine Check Exception Log is a tool used to record and analyze Machine Check Exception (MCE) logs. MCEs are exceptions triggered by hardware errors (such as memory errors, CPU cache errors, etc.). The mcelog log helps system administrators detect and analyze hardware failures.

[0050] The system event log records hardware and firmware events, such as hardware failures, temperature warnings, and power events. These logs are typically generated and stored by the system management controller (such as the BMC) and can be accessed through interfaces such as the Intelligent Platform Management Interface (IPMI).

[0051] An analysis results report is a JSON file that typically contains the results of analysis on logs or data. This type of file may be generated by system administration tools to display hardware health status, performance analysis results, or other important information. The JSON format facilitates the structured storage and transmission of data.

[0052] Black-box logs are typically collected by the Baseboard Management Controller (BMC) from logs of the CPU, memory, hard drive, power supply, fans, and other hardware components (such as Peripheral Component Interconnect Express (PCIe) devices, network cards, etc.).

[0053] If the target storage device is a solid-state drive (SSD), the in-band log can include the Self-Monitoring, Analysis, and Reporting Technology (SMART) attribute, which can utilize the reallocated sector count / program failure count from the SMART attribute. The out-of-band log can include the Flash Translation Layer Mapping Table (FTL). Fault prediction can be performed using change records from the FTL mapping table, controller chip temperature analysis, and by integrating NAND block erase / write imbalances and read interference effects.

[0054] If the target storage device is the video memory of an accelerator card, the in-band log can include the Compute Unified Device Architecture Error-Correcting Code Log; the out-of-band log can include thermal system data, which can be combined with dynamic thresholds, such as analysis of current temperature > 85℃ + daily CE > 10 times → circuit breaker warning, to predict the failure of selected storage; the log file can also include application layer rendering engine errors.

[0055] While some related technologies employ multi-source logs for storage device fault prediction, they only analyze storage device-related information from these logs. Since different management controllers may use different timestamps to generate logs, and the recording times may also differ, the analysis of multi-source logs is isolated, leading to the breakdown of crucial evidence chains. Therefore, in this embodiment of the invention, event sequences are extracted from multiple log files, and these sequences are aligned to obtain an aligned event set. This allows the fault paths recorded in each log file to complement and corroborate each other, forming a more complete evidence chain.

[0056] In this embodiment of the invention, S102 extracts the event sequence corresponding to the log file and aligns multiple event sequences to obtain an aligned event set. This may include: converting the timestamps of multiple log files into a unified timestamp to obtain multiple event sequences based on the unified timestamp; and determining the connection path with the smallest cumulative difference for any two event sequences in the event sequence to align the two event sequences.

[0057] The log collection layer collects logs at regular intervals (including both in-band and out-of-band logs). Because the clock frequencies used for in-band and out-of-band logs are inconsistent, the timestamps of the collected in-band and out-of-band logs cannot be truly synchronized. Therefore, a timestamp alignment algorithm is needed to align the timestamps in the collected logs.

[0058] Figure 3 This is a flowchart of event sequence alignment for multi-source logs provided in an embodiment of the present invention.

[0059] like Figure 3 As shown, in the timestamp alignment algorithm, the collected raw logs are input into the preprocessing layer in the form of a log flow. The preprocessing layer performs operations on the raw logs, including clock source identification, time zone standardization, and conversion of the time format to a unified UNIX timestamp. To achieve efficient timestamp alignment, an event sequence alignment engine based on event sequence matching is used, and an aligned event set is output.

[0060] Figure 4 This is a flowchart illustrating the workflow of an event sequence alignment engine provided in an embodiment of the present invention.

[0061] like Figure 4 As shown, taking the event alignment of logs A and B as an example, the specific operations of the event sequence alignment engine can include: constructing a cost matrix based on the event sequences of log A and log B; finding the minimum cost path based on the cost matrix; and outputting the alignment mapping. The minimum cost path refers to the connection path with the smallest cumulative difference among various alignment methods for the two event sequences. The cumulative difference can be measured by the similarity of events occurring at the same time after alignment, or the temporal distance of the same event in different event sequences.

[0062] The storage device fault prediction method provided in this invention obtains log files collected by multiple management controllers of the device, i.e., multi-source log files, covering more information sources related to storage device faults; aligns the event sequences of the multi-source log files to obtain an aligned event set; extracts the storage unit features and the first associated features of the storage unit features of the target storage device from the aligned event set; and calls multiple fault prediction models with different input data to predict the faults of the target storage device based on the storage unit features and the first associated features. This solves the problem in related technologies that only focus on storage unit errors, resulting in a low detection rate of potential errors, and improves the detection probability of abnormal information of storage devices. The fault prediction result of the target storage device is determined according to the output result of the fault prediction model, thereby improving the accuracy of the fault prediction result of the storage device.

[0063] Based on the above embodiments, the present invention further introduces the fault prediction model.

[0064] Figure 5 This is a schematic diagram illustrating the propagation path of a memory fault, as provided in an embodiment of the present invention.

[0065] like Figure 5 As shown, taking memory as the target storage device as an example, when a bit misalignment occurs in memory (such as Dynamic Random Access Memory (DRAM)), the memory controller detects a memory cell error event and determines whether the error type is a correctable or uncorrectable error. If the error type is a correctable error, the correctable error counter is incremented by 1. After processing by the memory decay mechanism, it is determined whether the value of the correctable error counter exceeds the threshold that triggers a memory controller anomaly. If the error type is an uncorrectable error, the machine will be triggered to check for anomalies, thereby causing an abnormal state of the memory controller.

[0066] An abnormal memory controller state may trigger CPU errors. Error types can include: configuration conflicts causing CPU configuration errors (CPU Configuration Errors) that ultimately lead to system instability; internal fatal errors causing CPU Internal Errors (CPU IERRs) that ultimately lead to CPU core suspension; and cache pollution causing warm resets that ultimately lead to unexpected reboots.

[0067] All of the above errors could cause the operating system to crash, which in turn could cause the server to go down.

[0068] Operating system crashes caused by memory bit errors are recorded in the out-of-band system event log and the intelligent platform management interface debug log; server crashes caused by memory bit errors are recorded in the in-band machine check anomaly log and display message log. This allows for reverse tracing of memory controller anomalies by analyzing the out-of-band system event log and intelligent platform management interface debug log, and for correcting error counter over-threshold issues by using the in-band machine check anomaly log and display message log. When a memory controller anomaly occurs, information related to the Model Specific Register (MSR) will be recorded in the analysis results report. Parsing the MSR register information allows for the determination of the current memory error type and the value of the correctable error counter, for example:

[0069] IA32_MCi_STATUS bit analysis:

[0070] bit

[61] =1 → Uncorrectable error (UCE);

[0071] bit[52:38] → CE count (CECNT).

[0072] Then refer to Figure 5 As shown in the fault propagation path, it can be seen that before and after the target storage device fails, its potential fault will propagate in different components of the device. Therefore, the storage cell characteristics of the target storage device and the first associated characteristics associated with the potential fault of the target storage device can be used to detect the potential fault of the target storage device.

[0073] In this embodiment of the invention, for the aforementioned storage cell features and the first associated feature, the fault prediction model may include a first fault prediction model and a second fault prediction model; the first fault prediction model is used to predict the fault of the target storage device based on the storage cell features; and the second fault prediction model is used to predict the fault of the target storage device based on the first associated feature.

[0074] Figure 6 This is an architecture diagram of a fault prediction model provided in an embodiment of the present invention.

[0075] like Figure 6 As shown, the first fault prediction model may include at least one of a third fault prediction model, a fourth fault prediction model, and a fifth fault prediction model. The third fault prediction model is used to determine the memory cell error tolerance threshold of the target storage device based on its capacity, and to determine the fault prediction result of the target storage device based on the memory cell error tolerance threshold and the actual number of memory cell errors in the target storage device. The fourth fault prediction model is used to determine the fault prediction result of the target storage device based on the duration of a fault activation event. The fifth fault prediction model is used to determine the fault prediction result of the target storage device based on the increment of the duration of the fault activation event. A fault activation event is an event in which the number of memory cell errors exceeds the memory cell error tolerance threshold.

[0076] The core judgment condition of the third fault prediction model is: by dynamically balancing the number of errors in the quantified storage unit with the error count decay mechanism, the complex storage unit decay process is transformed into a calculable fault threshold, such as 4 times / 24h.

[0077] Taking the target storage device as memory as an example, the storage unit can be a memory page. The relationship between the memory page count and the number of correctable errors is as follows:

[0078] Standard memory page size: 4KB (industry standard).

[0079] Page count calculation for a single 16GB memory module:

[0080] .

[0081] Error counting events are essentially: each correctable error corresponds to a bit flip error in a single memory page, counted by the memory controller (CECNT).

[0082] To avoid false alarms caused by transient interference, this embodiment of the invention introduces an error count decay mechanism. For example, for memory, the correctable error count can be decremented by 1 every minute, calculated as follows:

[0083] CE_count(t)=max(0,CE_count(t-1)-1);

[0084] Where CE_count(t) represents the correctable error count at time t.

[0085] By introducing an error count decay mechanism, if the memory is healthy, the count of occasional correctable errors will be quickly cleared to zero (e.g., an error disappears after 1 minute); if the memory fails, continuous high-frequency correctable errors will cause the count to accumulate (decay rate < error occurrence rate).

[0086] Figure 7 This is a schematic diagram of a correctable error count decay provided in an embodiment of the present invention.

[0087] like Figure 7 As shown, after a correctable error occurs in each storage unit of memory, a correctable error count is performed, and the count is decayed by decreasing by one correctable error per minute.

[0088] In this embodiment of the invention, the modeling steps of the third fault prediction model may include: determining a unit storage cell error tolerance threshold based on the upper limit of the number of storage cell errors corresponding to the unit storage capacity of the target storage device and the maximum decay of the storage cell error count within a first unit time; determining a storage cell error tolerance threshold for the target storage device based on the capacity of the target storage device and the unit storage cell error tolerance threshold; and configuring the storage cell error tolerance threshold of the third fault prediction model based on the storage cell error tolerance threshold of the target storage device.

[0089] For example, for memory, testing shows that the maximum number of correctable errors for 16GB of memory is 6500. Using a decay rate of 1 error per minute, with the first unit of time being one day, the maximum decay rate per day is 1440 errors. Obtaining a preset fault tolerance rate, for example, a conservative design of 0.05%, the fault tolerance threshold for a 16GB memory storage unit is: Maximum correctable error count per day × Fault tolerance rate = (Maximum correctable error count + Maximum daily decay rate) × Fault tolerance rate = (6500 + 1440) × 0.05% ≈ 3.97, rounded down to 4 errors.

[0090] Therefore, for 16GB of memory, if the number of correctable errors occurring in the memory on a single day is ≥4, it means that the error rate exceeds the natural decay capacity and the memory enters an irreversible decay state.

[0091] Therefore, we can determine the safety threshold per GB: Safety threshold per GB = 4 times / 16GB = 0.25 times / GB. The thresholds for different memory sizes are as follows:

[0092] Table 1

[0093]

[0094] For safety reasons, a threshold of 4 tests is applied to memory sizes smaller than 16GB.

[0095] The real-time detection logic of the third fault prediction model can be implemented using the following pseudocode:

[0096] def rule1_ce_frequency_monitor(dimm_id):

[0097] ce_events = [] # Stores the timestamps of correctable error events in this memory;

[0098] while True:

[0099] if detect_ce_event(dimm_id): # CE event detected;

[0100] ce_events.append(current_time) # Record timestamps;

[0101] # Clear events from 24 hours ago;

[0102] ce_events = [t for t in ce_events if current_time - t <24*3600]

[0103] if len(ce_events) >= 4: # Trigger threshold;

[0104] generate_alert(f"DIMM {dimm_id} has ≥4 CE events within 24 hours!");

[0105] sleep(check_interval) # Check interval (e.g., 1 minute).

[0106] The diagnostic process of the third fault prediction model may include:

[0107] The log analysis engine detected ≥ 4 correctable error events in the same memory within 24 hours (taking 16G memory as an example).

[0108] Excluded temperature exceeding limits (sensor displays 35℃ < 38℃ threshold);

[0109] Judgment result: The fault prediction alarm of the third fault prediction model is triggered.

[0110] In summary, the third fault prediction model transforms the complex memory decay process into a calculable fault threshold by dynamically balancing the memory page error rate and the memory decay mechanism.

[0111] In this embodiment of the invention, the modeling steps of the fourth fault prediction model may include: pre-testing to obtain a first correspondence between the duration of the fault activation event and the fault type of the target storage device; and configuring a monitoring threshold for the duration of the fault activation event of the fourth fault prediction model according to the first correspondence.

[0112] Taking memory as an example, the core judgment condition of the fourth fault prediction model can be: the duration of the count of correctable errors in memory from the fault-activated state (Asserted) to the fault-deasserted state (Deasserted). Exceeding a threshold (e.g., 7 hours), or in a series of events If the increment exceeds a preset value (such as 1 hour), the memory is determined to be in a high-risk state.

[0113] Figure 8 This is a flowchart for judging a fault activation event provided in an embodiment of the present invention.

[0114] Combining the decay mechanism and the duration of the fault activation event described above, the process of determining the transition from a fault-activated state to a fault-cleared state after a correctable error occurs can be described as follows: Figure 8 As shown.

[0115] The longer the time span between the fault-correctable error state and the fault-cleared state, the greater the number of correctable errors in the memory, or the continuous generation of correctable errors. This is because in the memory decay mechanism, the correctable error counter continuously decreases. If the faulty memory cannot automatically clear itself (change from fault-active to fault-cleared) within a short period, there are two possibilities: either a large number of correctable errors were generated in a short time, or the correctable error count is continuously increasing. Of course, factors affecting the time span of memory fault clearance, such as server power on / off cycles, and memory replacements, also need to be excluded.

[0116] Testing revealed that different fault activation events correspond to different memory states in terms of duration, allowing for the setting of corresponding fault levels, as shown in Table 2.

[0117] Table 2

[0118]

[0119] in The range is a number obtained through testing and verification, targeting different levels. The specific range of values ​​will be adjusted according to the actual situation.

[0120] For example, in real-world application scenarios, the changes in memory fault states recorded in the system event log, such as the CPU0_A0 event sequence for memory slot, are shown in Table 3:

[0121] Table 3

[0122]

[0123] The diagnostic process based on the example above is as follows:

[0124] Second incident: If 7.25h > 7h, then the fault level is high-risk.

[0125] The 3rd incident: Increment = 18.67 - 7.25 = 11.42h If the damage continues to increase, the fault level will be classified as an emergency.

[0126] Figure 9 A flowchart for fault prediction based on the duration of a fault activation event is provided for an embodiment of the present invention.

[0127] like Figure 9 As shown, in the collected logs, it is first necessary to eliminate interfering logs, such as those related to server restarts or memory replacements. Then, the memory information at the time of fault activation is recorded, and any unclosed fault activation events are searched in the logs. The corresponding fault duration is then updated, allowing the calculation of the time it takes for the memory state to change from fault activation to fault recovery. ,judge If the threshold is exceeded, it indicates a potential problem with the memory and the memory needs to be replaced. This is verified by checking both the initial and subsequent checks. Value calculation The increment, and analysis If the increment is increasing, it indicates that the memory is corrupted and accelerating, and an alarm message should be issued to focus on this memory.

[0128] In summary, the fourth fault prediction model transforms the memory fault state transitions (fault activation state → fault recovery state) triggered by the memory decay mechanism into observable... The indicator visualizes latent faults, using time to measure the cumulative count of invisible correctable errors; it predicts decay trends through... Incremental capture of memory corruption increments; precise operation and maintenance decisions: generating tiered early warnings by combining thresholds and trends.

[0129] In this embodiment of the invention, the modeling step of the fifth fault prediction model may include: pre-testing to obtain a second correspondence between the increment of the duration of the fault activation event and the fault type of the target storage device; and configuring a monitoring threshold for the increment of the duration of the fault activation event of the fifth fault prediction model according to the second correspondence.

[0130] Taking memory as an example, the core judgment condition of the fifth fault prediction model can be: for the same memory module, when the fault state of consecutive CE events is cleared (fault activation state → fault recovery state) the time taken ( The increment of ) If the time exceeds a preset threshold (e.g., 1 hour), it is determined that memory corruption is accelerating, and an immediate warning and replacement are required. It can be used to assess the acceleration caused by damage.

[0131] Figure 10 This is a schematic diagram illustrating the microscopic mechanism of the increment of the duration of a fault activation event, as provided in an embodiment of the present invention.

[0132] Causes damage acceleration The microscopic mechanisms of change can be as follows Figure 10 As shown, initial defects can include electromigration effects, hot carrier injection, gate oxide degradation, etc. Electromigration effects cause the electromigration effect conductors to become thinner, hot carrier injection causes the threshold voltage to drift, and gate oxide degradation causes the leakage current to increase. The factors such as conductor thinning, threshold voltage drift, and leakage current increase the resistance, which in turn leads to an increase in the bit error rate of the memory, and can correct the accumulation of error counts.

[0133] Through experiments, the growth formula for the bit error rate of memory can be obtained as follows:

[0134] ;

[0135] in, Represents the initial error rate. This represents the damage acceleration coefficient.

[0136] and The relationship is ,Right now That is, the damage acceleration coefficient is positively correlated with the increment of the duration of the fault activation event.

[0137] In this embodiment of the invention, configuring the monitoring threshold for the increment of the duration of the fault activation event in the fifth fault prediction model according to the second correspondence relationship may include: determining a baseline threshold in the fifth fault prediction model for the increment of the duration of the fault activation event when the fault activation event first occurs, according to the second correspondence relationship; when the fault activation event does not occur for the first time, updating the baseline threshold in the fifth fault prediction model using the second associated feature of the target storage device.

[0138] The second associated feature includes at least one of the following: load parameters, temperature value, predicted remaining lifetime value, and process parameters of the target storage device.

[0139] Many factors influence the number of correctable memory error events, such as: memory load rate (load percentage 0-100); ambient temperature (excessive external temperature, low fan speed leading to heat dissipation failure); memory usage time (aging of memory materials); and process node (advanced processes are more sensitive).

[0140] Figure 11 A flowchart of a fault prediction process based on the incremental duration of a fault activation event is provided for an embodiment of the present invention.

[0141] To adapt to the influence of the above multi-dimensional factors, real-time adaptive thresholds are required. The real-time calibration process can be as follows: Figure 11 As shown. Based on the original damage acceleration. Determine whether the fault activation event is the first occurrence. If it is the first occurrence, use the baseline threshold. If it is not the first occurrence, calculate the current influencing factors and comprehensively adjust the threshold through at least one of the following: load rate correction, temperature correction, lifespan correction, and process correction. Update the baseline threshold, obtain the corrected threshold, and apply it.

[0142] The lifetime correction can be obtained by predicting the remaining lifetime of the target storage device using a lifetime prediction algorithm. The pseudocode for the remaining lifetime prediction algorithm can be as follows:

[0143] def predict_remaining_life(history, current_beta):

[0144] """

[0145] history: [( [timestamp]

[0146] current_beta: Current damage acceleration

[0147] return: Remaining lifespan (hours)

[0148] """

[0149] # Phase 1: History of Linear Regression trend

[0150] X = np.array([i for i in range(len(history))])

[0151] y = np.array([calc_beta(history[:i+1]) for i in range(2, len(history))])

[0152] slope, _ = linregress(X[2:], y)

[0153] # Phase 2: Predicting the Future

[0154] future_beta = current_beta + slope * PREDICTION_WINDOW

[0155] # Phase 3: Calculate Remaining Lifetime

[0156] if future_beta > BETA_CRITICAL:

[0157] return 24 # State of Emergency

[0158] else:

[0159] return max(24, 72 / (future_beta / BETA_BASE)).

[0160] The fifth fault prediction model can output different early warning information based on the correspondence between the duration increment of the fault activation event and the threshold, as shown in Table 4:

[0161] Table 4

[0162]

[0163] Here, 70%-min (30%, 5% / increment) means that each new addition... When an event occurs, the penalty value increases by 5%, with a maximum penalty cap of 30% (to avoid invalid alerts due to confidence levels below 40%). For example, the first time a new alert is added... At the time of the event, the confidence level is 70% - min(30%, 5% × 1) = 65%.

[0164] Figure 12 A flowchart of a fault prediction process based on the incremental duration of a fault activation event is provided for an embodiment of the present invention.

[0165] like Figure 12 As shown, taking memory as an example, the prediction process of the fifth fault prediction model can include: collecting correctable memory error events and cleaning the collected correctable error events. During the data cleaning process, invalid data needs to be excluded: data from restarts, memory replacements, and sudden temperature changes needs to be excluded. Historical damage acceleration coefficients are then determined through linear regression. Trends, predicting the future Then calculate the remaining lifespan. Trigger tiered alarms based on the remaining memory lifespan and initiate a maintenance work order in the system to replace the faulty memory.

[0166] In summary, the fifth fault prediction model establishes the increment of the duration of the fault activation event as the damage acceleration coefficient. →A complete quantitative chain of remaining lifetime was established, transforming microscopic mechanisms such as electromigration / hot carriers into prediction parameters, and initially establishing bit error rate and... The mathematical relationships were established; dynamic adaptive adjustment of four-dimensional thresholds (load, temperature, lifespan, and process) was realized, and a three-level confidence warning system was established; anti-interference data cleaning pipeline and automated operation and maintenance integration were realized.

[0167] In this embodiment of the invention, the modeling step of the second fault prediction model may include: pre-determining the association monitoring threshold of the first association feature based on the feature parameters of the first association feature when the memory cell error of the target memory device occurs; configuring the second fault prediction model according to the association monitoring threshold of the target memory device, so that the second fault prediction model outputs the fault prediction result based on the association monitoring threshold and the memory cell error.

[0168] In practical implementation, taking memory as an example, the second fault prediction model can adopt a dual-path judgment condition. Path A is the temperature over-limit judgment. For example, when the external heat dissipation environment of the server is normal, if the memory temperature exceeds its threshold (such as 38℃), a fault warning will be triggered. Path B is the associated fault. For example, when a memory error event causes a central processing unit level error, it can be directly determined that the memory is faulty and needs to be replaced.

[0169] Actual testing revealed that memory temperature issues stem from factors including memory current leakage (leakage power consumption) and row activation charge refresh energy consumption. Temperature and failure rate showed a positive correlation; test results confirmed that the failure rate doubled for every 10°C increase in memory temperature.

[0170] Figure 13This is a path diagram of a temperature-induced cascading failure provided in an embodiment of the present invention. For example... Figure 13 As shown, when the memory temperature is greater than 38°C, memory leakage increases, causing a 30% increase in bit error rate, which can correct the error count exceeding the threshold and cause a central processing unit configuration error.

[0171] Figure 14 This is a fault propagation path diagram provided in an embodiment of the present invention.

[0172] like Figure 14 As shown, for associated faults, when a memory error event occurs, the event is reported to the central processing unit (CPU) via the memory controller (IMC). If the memory fault is severe, it can trigger a series of problems in the CPU, such as memory errors, configuration errors, and hot resets. Although the fault is reported by the CPU, the root cause is a memory failure.

[0173] In the second fault prediction model, the temperature over-limit judgment logic can be as follows: if the memory temperature exceeds 38℃ (threshold is configurable), it is necessary to determine whether the ambient temperature of the computer room is <25℃ (to ensure that external heat dissipation is normal), the fan speed is >80% (to ensure that forced heat dissipation is normal), and the temperature difference between adjacent memory modules is <5℃ (to exclude local hot spots). At this time, it can be determined that the memory is overheating - the memory temperature rise is caused by physical damage.

[0174] The logic for identifying memory faults by associating fault paths can be as follows: retrieve concurrent memory events (such as error events), locate the associated memory slots (based on the CPU topology), and verify the fault propagation chain. When the CPU reports errors of the following types: memory error, configuration error, or hot reset, and an error event occurs in the memory directly connected to the CPU, it can be determined that the memory is faulty (based on data verification and fault propagation paths, the probability of memory faults is greater than the probability of CPU faults).

[0175] Figure 15 This is a temperature monitoring architecture diagram provided for an embodiment of the present invention.

[0176] like Figure 15 As shown, the temperature monitoring architecture of the second fault prediction model can include: collecting temperature data through the temperature sensor of the baseboard management controller, out-of-band logs (such as system event logs and intelligent platform management interface debugging logs), and in-band tools, with a sampling frequency of 1 time / second (to avoid transient missed detections). To eliminate interference, short-term peak values ​​(<5 seconds) are considered noise; and a multi-sensor voting mechanism is introduced (at least two sensors exceeding limits are required to trigger subsequent processes). The temperature analysis engine analyzes whether the memory temperature exceeds the set threshold; if the memory temperature exceeds the threshold, it is necessary to check whether the associated heat dissipation is normal; if the associated heat dissipation is also normal, a memory alarm message is triggered, indicating that the memory may be about to fail.

[0177] In the associated fault diagnosis, the in-band and out-of-band logs are searched for fault information of the types of memory errors, configuration errors, and hot resets. If such fault information is found, the system searches for any memory error events occurring simultaneously under the CPU where the above fault information occurred. If a memory error event is found, the memory may be faulty, and a memory replacement request work order needs to be initiated to replace the faulty memory.

[0178] In summary, the second fault prediction model can cover several extreme scenarios of target storage device failures through multiple paths. For example, temperature exceeding the limit related to memory directly reflects hardware failure, and related faults at the system level reveal the error propagation chain. This can make up for the blind spot of correctable error counting in memory (such as physical damage with no correctable errors) and achieve minute-level high-risk fault response, supporting root cause tracing (precisely locating the faulty memory slot).

[0179] Based on the aforementioned fault prediction models, it is possible to determine whether a target storage device is faulty or is about to fail. Multiple fault prediction models can be analyzed in parallel. That is, as long as one fault prediction model outputs a fault prediction result indicating that the target storage device will fail, the corresponding fault prediction result triggers an early warning mechanism. Specifically, the fault prediction results output by multiple fault prediction models are combined using a union method. If a fault prediction model outputs a result, it directly connects to the operation and maintenance system and dispatches a repair work order to the operation and maintenance personnel to replace the faulty target storage device, ensuring the normal operation of the equipment.

[0180] In this embodiment of the invention, determining the fault prediction result of the target storage device based on the output result of the fault prediction model in S104 may include: calling the alarm configuration parameters corresponding to the fault prediction model, and outputting the corresponding alarm configuration parameters according to the monitoring threshold triggered by the output result of the fault prediction model; the alarm configuration parameters include at least a second-level alarm level; the alarm type corresponding to the alarm configuration parameters includes at least one of adding to the monitoring list, recording the storage device abnormality log, and outputting the storage device replacement period.

[0181] In practice, each fault prediction model can establish a multi-level alarm mechanism. Different levels of alarm mechanisms correspond to different alarm types and trigger different maintenance work orders.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0183] Embodiments of the present invention also provide a storage device fault prediction apparatus, which may include: an acquisition unit for acquiring log files collected by multiple management controllers of the device; a log alignment unit for extracting event sequences corresponding to the log files and aligning multiple event sequences to obtain an aligned event set; an extraction unit for extracting storage unit features of the target storage device and a first associated feature of the storage unit features from the aligned event set; and a prediction unit for calling multiple fault prediction models to predict the fault of the target storage device based on the storage unit features and the first associated feature, and determining the fault prediction result of the target storage device according to the output result of the fault prediction model; wherein, the input data of different fault prediction models are different.

[0184] In this embodiment of the invention, the extraction unit extracts the event sequence corresponding to the log file and aligns multiple event sequences to obtain an aligned event set. This may include: converting the timestamps of multiple log files into a unified timestamp to obtain multiple event sequences based on the unified timestamp; and determining the connection path with the smallest cumulative difference for any two event sequences in the event sequence to align the two event sequences.

[0185] In this embodiment of the invention, the fault prediction model includes a first fault prediction model and a second fault prediction model; the first fault prediction model is used to predict the fault of the target storage device based on the characteristics of the storage unit; the second fault prediction model is used to predict the fault of the target storage device based on the first associated characteristics.

[0186] In this embodiment of the invention, the first fault prediction model includes at least one of a third fault prediction model, a fourth fault prediction model, and a fifth fault prediction model; the third fault prediction model is used to determine the memory cell error tolerance threshold of the target memory device based on the capacity of the target memory device, and to determine the fault prediction result of the target memory device based on the memory cell error tolerance threshold and the actual number of memory cell errors of the target memory device; the fourth fault prediction model is used to determine the fault prediction result of the target memory device based on the duration of the fault activation event of the target memory device; the fifth fault prediction model is used to determine the fault prediction result of the target memory device based on the increment of the duration of the fault activation event; the fault activation event is an event in which the number of memory cell errors exceeds the memory cell error tolerance threshold.

[0187] In this embodiment of the invention, the modeling steps of the third fault prediction model may include: determining a unit storage cell error tolerance threshold based on the upper limit of the number of storage cell errors corresponding to the unit storage capacity of the target storage device and the maximum decay of the storage cell error count within a first unit time; determining a storage cell error tolerance threshold for the target storage device based on the capacity of the target storage device and the unit storage cell error tolerance threshold; and configuring the storage cell error tolerance threshold of the third fault prediction model based on the storage cell error tolerance threshold of the target storage device.

[0188] In this embodiment of the invention, the modeling steps of the fourth fault prediction model may include: pre-testing to obtain a first correspondence between the duration of the fault activation event and the fault type of the target storage device; and configuring a monitoring threshold for the duration of the fault activation event of the fourth fault prediction model according to the first correspondence.

[0189] In this embodiment of the invention, the modeling step of the fifth fault prediction model may include: pre-testing to obtain a second correspondence between the increment of the duration of the fault activation event and the fault type of the target storage device; and configuring a monitoring threshold for the increment of the duration of the fault activation event of the fifth fault prediction model according to the second correspondence.

[0190] In this embodiment of the invention, configuring the monitoring threshold for the increment of the duration of the fault activation event in the fifth fault prediction model according to the second correspondence relationship may include: determining a baseline threshold in the fifth fault prediction model for the increment of the duration of the fault activation event when the fault activation event first occurs, according to the second correspondence relationship; when the fault activation event does not occur for the first time, updating the baseline threshold in the fifth fault prediction model using the second associated feature of the target storage device.

[0191] In this embodiment of the invention, the second associated feature may include at least one of the following: load parameters of the target storage device, temperature value, predicted remaining lifetime value, and process parameters.

[0192] In this embodiment of the invention, the modeling step of the second fault prediction model may include: pre-determining the association monitoring threshold of the first association feature based on the feature parameters of the first association feature when the memory cell error of the target memory device occurs; configuring the second fault prediction model according to the association monitoring threshold of the target memory device, so that the second fault prediction model outputs the fault prediction result based on the association monitoring threshold and the memory cell error.

[0193] In this embodiment of the invention, the prediction unit determines the fault prediction result of the target storage device based on the output result of the fault prediction model, which may include: calling the alarm configuration parameters corresponding to the fault prediction model, and outputting the corresponding alarm configuration parameters according to the monitoring threshold triggered by the output result of the fault prediction model; the alarm configuration parameters include at least a second-level alarm level; the alarm type corresponding to the alarm configuration parameters includes at least one of adding to the monitoring list, recording the storage device abnormality log, and outputting the storage device replacement period.

[0194] In this embodiment of the invention, the log file may include the device's in-band log and the device's out-of-band log; the in-band log may include at least one of the following: display message log, system operation message log, and machine check anomaly log; the out-of-band log may include at least one of the following: system event log, intelligent platform management interface debugging log, analysis result report, and black box log.

[0195] For a description of the features in the embodiment corresponding to the storage device fault prediction device, please refer to the relevant description of the embodiment corresponding to the storage device fault prediction method, which will not be repeated here.

[0196] Embodiments of the present invention also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the memory device fault prediction method.

[0197] Embodiments of the present invention also provide a non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the storage device failure prediction method when running.

[0198] In one exemplary embodiment, the aforementioned non-volatile storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0199] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the storage device fault prediction method.

[0200] Embodiments of the present invention also provide another computer program product, including a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the storage device fault prediction method.

[0201] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0202] The foregoing has provided a detailed description of a storage device fault prediction method, device, medium, and product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to aid in understanding the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for predicting storage device failures, characterized in that, include: Obtain log files collected by multiple management controllers of the device; Extract the event sequences corresponding to the log files, and align multiple event sequences to obtain an aligned event set; Extract the storage cell features of the target storage device and the first associated feature of the storage cell features from the alignment event set; Multiple fault prediction models are invoked to predict the fault of the target storage device based on the characteristics of the storage cell and the first associated feature, and the fault prediction result of the target storage device is determined according to the output of the fault prediction model. The input data for different fault prediction models are different; The fault prediction model includes a first fault prediction model and a second fault prediction model. The first fault prediction model is used to predict the faults of the target storage device based on the characteristics of the storage cell; The second fault prediction model is used to predict the fault of the target storage device based on the first correlation feature; The first fault prediction model includes a third fault prediction model, a fourth fault prediction model, and a fifth fault prediction model; The third fault prediction model is used to determine the memory cell error tolerance threshold of the target storage device based on the capacity of the target storage device, and to determine the fault prediction result of the target storage device based on the memory cell error tolerance threshold and the actual number of memory cell errors of the target storage device. The fourth fault prediction model is used to determine the fault prediction result of the target storage device based on the duration of the fault activation event of the target storage device. The fifth fault prediction model is used to determine the fault prediction result of the target storage device based on the increment of the duration of the fault activation event; The fault activation event is an event in which the number of errors in the storage unit exceeds the error tolerance threshold of the storage unit.

2. The storage device fault prediction method according to claim 1, characterized in that, Extract the event sequences corresponding to the log files, and align multiple event sequences to obtain an aligned event set, including: The timestamps of the multiple log files are converted into a unified timestamp to obtain multiple event sequences based on the unified timestamp; For any two event sequences in the event sequence, the connection path with the smallest cumulative difference is determined to align the two event sequences.

3. The storage device fault prediction method according to claim 1, characterized in that, The modeling steps of the third fault prediction model include: The unit storage cell error tolerance threshold is determined based on the upper limit of the number of storage cell errors corresponding to the unit storage capacity of the target storage device and the maximum decay of the storage cell error count within the first unit time. The memory cell error tolerance threshold of the target storage device is determined based on the capacity of the target storage device and the unit memory cell error tolerance threshold. Configure the memory cell error tolerance threshold of the third fault prediction model according to the memory cell error tolerance threshold of the target storage device.

4. The storage device fault prediction method according to claim 1, characterized in that, The modeling steps of the fourth fault prediction model include: A first correspondence between the duration of the fault activation event and the fault type of the target storage device is obtained through pre-testing; The monitoring threshold for the duration of the fault activation event of the fourth fault prediction model is configured according to the first correspondence.

5. The storage device fault prediction method according to claim 1, characterized in that, The modeling steps of the fifth fault prediction model include: A second correspondence is obtained in advance between the increment of the duration of the fault activation event and the fault type of the target storage device; The monitoring threshold for the incremental duration of the fault activation event in the fifth fault prediction model is configured according to the second correspondence.

6. The storage device fault prediction method according to claim 5, characterized in that, The monitoring threshold for the incremental duration of the fault activation event in the fifth fault prediction model is configured according to the second correspondence, including: Based on the second correspondence, a baseline threshold is determined in the fifth fault prediction model for the increment of the duration of the fault activation event when the fault activation event first occurs; When the fault activation event occurs for the first time, the baseline threshold in the fifth fault prediction model is updated using the second associated feature of the target storage device.

7. The storage device fault prediction method according to claim 6, characterized in that, The second associated feature includes at least one of the target storage device's load parameters, temperature value, predicted remaining lifetime value, and process parameters.

8. The storage device fault prediction method according to claim 1, characterized in that, The modeling steps of the second fault prediction model include: The association monitoring threshold of the first associated feature is determined in advance based on the feature parameters of the first associated feature when the storage cell error of the target storage device occurs; Configure the second fault prediction model according to the associated monitoring threshold of the target storage device, so that the second fault prediction model outputs fault prediction results based on the associated monitoring threshold and the storage unit error.

9. The storage device fault prediction method according to claim 1, characterized in that, Determining the fault prediction result of the target storage device based on the output of the fault prediction model includes: Call the alarm configuration parameters corresponding to the fault prediction model, and output the corresponding alarm configuration parameters based on the monitoring threshold triggered by the output result of the fault prediction model; The alarm configuration parameters include at least a level 2 alarm level; The alarm types corresponding to the alarm configuration parameters include at least one of the following: adding to the monitoring list, recording storage device anomaly logs, and outputting storage device replacement deadlines.

10. The storage device fault prediction method according to claim 1, characterized in that, The log file includes the device's in-band log and the device's out-of-band log; The in-band log includes at least one of the following: display message log, system operation message log, and machine check anomaly log; The out-of-band logs include at least one of the following: system event logs, intelligent platform management interface debugging logs, analysis result reports, and black-box logs.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the storage device failure prediction method as described in any one of claims 1 to 10 when executing the computer program.

12. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the storage device failure prediction method as described in any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the storage device failure prediction method as described in any one of claims 1 to 10.

Images