Memory fault processing method and device and related equipment

Abstract

The invention provides a memory fault processing method and device and related equipment.The memory fault processing method comprises the steps that multiple pieces of alarm information of a memory of a first node are obtained, and the alarm information is used for representing correctable error CE information; calculating a target fault score according to the multiple pieces of alarm information; under the condition that the target fault score is greater than a first preset threshold value, migrating a first service of which the priority is higher than a preset numerical value in a plurality of services of the first node to a second node, the second node being a node without a fault; when the target fault score is larger than a second preset threshold value and smaller than the first preset threshold value, first prompt information is output, and the first prompt information is used for indicating that the first node has the risk of downtime fault. In this way, the flexibility of fault processing is enhanced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of cloud computing, and particularly relate to a memory fault processing method and device and related equipment. BACKGROUND

[0002] With the continuous development of cloud computing technology, cloud computing occupies an increasingly important position in people's lives. When the service life of the current cloud computing server is relatively long, various faults of the server are prone to occur, and thus the server needs to be detected for faults. However, when the detected faults are processed, the server usually needs to be suspended for use. It can be seen that the flexibility in processing the faults is poor. SUMMARY

[0003] Embodiments of the present application provide a memory fault processing method and device and related equipment to solve the problem of poor flexibility in processing faults.

[0004] To solve the above problem, the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a memory fault processing method, comprising:

[0006] obtaining a plurality of alarm information of the memory of the first node, the alarm information being used to represent CE information;

[0007] calculating a target fault score according to the plurality of alarm information;

[0008] migrating a first service with a priority higher than a preset value from a plurality of services of the first node to a second node in a case where the target fault score is greater than a first preset threshold, the second node being a node without a fault;

[0009] outputting a first prompt information in a case where the target fault score is greater than a second preset threshold and less than the first preset threshold, the first prompt information being used to represent a risk of a down fault of the first node.

[0010] In a second aspect, the embodiments of the present application provide a memory fault processing device, comprising:

[0011] an obtaining module configured to obtain a plurality of alarm information of the memory of the first node, the alarm information being used to represent CE information;

[0012] a first calculating module configured to calculate a target fault score according to the plurality of alarm information;

[0013] a migration module, configured to migrate, when the target failure score is greater than a first preset threshold, a first service with a priority higher than a preset value among a plurality of services of the first node to a second node, the second node being a node without failure;

[0014] an output module, configured to output, when the target failure score is greater than a second preset threshold and less than the first preset threshold, a first prompt information, the first prompt information being used to indicate that the first node has a risk of having a down failure.

[0015] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a program stored in the memory and capable of running on the processor; the processor is configured to read the program in the memory to implement the steps in the method in the first aspect.

[0016] In a fourth aspect, a readable storage medium is provided, which is used to store a program; the program is executed by a processor to implement the steps in the method in the first aspect.

[0017] In a fifth aspect, a computer program product is provided, which includes computer instructions; the computer instructions are executed by a processor to implement the steps in the method in the first aspect.

[0018] In the embodiments of the present application, the target failure score is calculated according to a plurality of alarm information; when the target failure score is greater than a first preset threshold, a first service with a priority higher than a preset value among a plurality of services of the first node is migrated to a second node; when the target failure score is greater than a second preset threshold and less than the first preset threshold, a first prompt information is output, the first prompt information being used to indicate that the first node has a risk of having a down failure; in this way, according to different target failure scores, the first service with a priority higher than a preset value among a plurality of services of the first node can be flexibly migrated to the second node or the first prompt information can be output, that is, the flexibility in processing failure is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is one of the flowcharts of the memory failure processing method provided by the embodiments of the present application;

[0021] Figure 2FIG. 2 is a flowchart of a memory fault processing method according to an embodiment of the present application;

[0022] Figure 3 FIG. 3 is a structural diagram of a memory fault processing apparatus according to an embodiment of the present application;

[0023] Figure 4 FIG. 4 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0025] The terms "first", "second", and the like in the embodiments of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, a method, a system, a product, or an apparatus that includes a series of steps or units does not necessarily limit to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, the method, the product, or the apparatus. In addition, "and / or" is used in the present application to represent at least one of the connected objects, for example, A and / or B and / or C represents 7 cases including A alone, B alone, C alone, A and B both exist, B and C both exist, A and C both exist, and A, B and C all exist.

[0026] Please refer to Figure 1 , Figure 1 FIG. 1 is a flowchart of a memory fault processing method according to an embodiment of the present application. Figure 1 The memory fault processing method shown in the figure can be executed by an electronic device, and optionally, the electronic device can be a server.

[0027] As shown in Figure 1 , the memory fault processing method can include the following steps:

[0028] Step 101, obtaining a plurality of alarm information of the memory of the first node, the alarm information being used to represent CE information.

[0029] Among them, the memory alarm information of the first node can mainly come from two sources: the first source can be memory log information, such as: the above memory log information includes at least one of the following: kernel log (dmesg), / proc file system log information, / sys file system log information; the second source can be server out-of-band log information.

[0030] It should be noted that, optionally, the first information source may include at least one of the following fields:

[0031] 1. Timestamp: Used to record the specific time the error occurred, in the format of `[hour.minute.second]` or `datetime`;

[0032] 2. Error Level: Usually marked as `error` or `crit`, used to indicate severity;

[0033] 3. Device identifier: such as `EDACMC0` (memory controller number) or `CPU2` (CPU core where the error occurred);

[0034] 4. Error type: Clearly indicate `CorrectedError` (correctable) or `UncorrectedError` (uncorrectable);

[0035] 5. Error location: including memory address (e.g., `offset0x1234`), channel number (e.g., `channel0`), or grain size (e.g., `grain0x20`);

[0036] 6. Error codes / syndromes: such as `syndrome0x5`, used for further diagnosis of the cause of hardware failure;

[0037] 7. Additional information: May include process ID (`pid`) or thread ID (`tid`), if the error is related to a user-mode process.

[0038] Optionally, the second information source may include at least one of the following:

[0039] 1. Types and severity of memory failures

[0040] CE (CorrectableError): A correctable single-bit error, which will be clearly marked in the log as **"ECCError"** or **"CorrectableECCError"**.

[0041] UE (UncorrectableError): Uncorrectable multi-bit error, usually accompanied by system crash or shutdown, with a log level of **"SEVERE"** or **"CRITICAL"**.

[0042] 2. Fault occurrence time and frequency

[0043] The log records the specific timestamp of the error and counts the number of errors.

[0044] 3. Location of affected memory modules

[0045] This includes physical location information such as memory modules (DIMMs), channels, and memory ranks.

[0046] 4. Error Triggering Scenarios

[0047] It can be associated with hardware status (such as overheating or abnormal voltage) or operational behavior (such as memory inspection or system load).

[0048] It should be noted that multiple alarm messages can be obtained from the first and second information sources mentioned above, and these multiple alarm messages may include at least one of the following:

[0049] 1. Error source: Identify the source of memory alarm collection, which can be the Linux operating system side or the out-of-band system side;

[0050] 2. Error Count: Counts the cumulative number of correctable errors, summing up the counts of errors in the same physical location, and filtering out the portion of uncorrectable errors;

[0051] 3. Error location information: Records location-related information within the log data, including: - Memory slot.

[0052] Physical address information such as -Channel -Rank -Bank -Row -Column;

[0053] 4. Error bit location: The coordinates of the specific bit where the error occurred;

[0054] 5. Timing pattern: The temporal characteristics of the error (e.g., transient, intermittent, or permanent);

[0055] 6. Error Duration: The length of time the error state lasts;

[0056] 7. Memory Failure Modes: Combining error codes / syndromes, in-band additional information, or out-of-band system hardware status information, the underlying failure types are classified into several major categories (such as software errors and hardware defects).

[0057] 8. Contextual information: This includes related data such as CPU error correction codes (ECC) and memory controller logs. Keywords are extracted and categorized by a word segmenter.

[0058] Step 102: Calculate the target fault score based on the multiple alarm information.

[0059] The calculation of the target fault score based on multiple alarm information can be understood as standardizing and quantifying multiple alarm information. The specific method for calculating the target fault score based on multiple alarm information is not limited here.

[0060] The target fault score can be understood as the CE risk coefficient value of the memory.

[0061] Step 103: If the target fault score is greater than the first preset threshold, migrate the first service with a priority higher than the preset value among the multiple services of the first node to the second node, where the second node is a node that has not experienced a fault.

[0062] Specifically, when the target fault score is greater than the first preset threshold, it is considered that the memory of the first node may damage the first service with a priority higher than the preset value. Therefore, the eviction mechanism can be triggered to restart the first service on the second node to ensure that the first service can be processed normally.

[0063] It should be noted that the aforementioned first preset threshold can be referred to as the high threshold A or the sensitive threshold.

[0064] Step 104: If the target fault score is greater than the second preset threshold and less than the first preset threshold, output the first prompt information. The first prompt information is used to indicate that the first node is at risk of downtime.

[0065] Specifically, when the target fault score is greater than the second preset threshold and less than the first preset threshold, the first prompt message is output, so as to promptly notify the user that the current first node is at risk of downtime.

[0066] The aforementioned second preset threshold can be referred to as the low threshold B or the downtime threshold.

[0067] As one optional implementation, the second preset threshold can be determined empirically. Alternatively, a large amount of data on uncorrectable memory errors can be collected to obtain the final downtime threshold score. This involves continuously refining the parameters of the memory score using a large model to ultimately achieve a threshold score accuracy exceeding the preset requirements. Since the collected data is unstructured, varied in format, and sparse with multiple dimensions, it is highly likely to fit this scenario. The random forest algorithm can effectively predict the probability of uncorrectable UCE errors, thereby controlling downtime; therefore, the random forest algorithm can be applied to the aforementioned large model.

[0068] The optimization process of the aforementioned large model can be described as follows:

[0069] During training, since the resource pool contains network elements of various shapes and sizes, and the positive samples far outnumber the negative samples, the accuracy of the random forest model is increased through two optimization points: dynamic calibration of feature weights and dynamic allocation of sample weights.

[0070] (1) Dynamic calibration of feature weights

[0071] Domain indicator mapping matrix: Constructing the feature-indicator correlation matrix:

[0072]

[0073] The α value is determined through the experience of domain experts, and a weighted Gini coefficient is introduced when calculating feature importance:

[0074]

[0075] Where p k p represents the proportion of samples in class k. When weighting, p can be adjusted based on the number of samples or the contribution of features. k The weight.

[0076] When constructing each tree, the size m of the feature subset is dynamically adjusted based on the current metric (such as call setup success rate). The calculation method is as follows:

[0077]

[0078] Where k is the number of basic features, β is the neighborhood sensitivity coefficient, d is the total number of features, and C metric This is the normalized value of the current metric (such as call setup success rate).

[0079] (2) Dynamic allocation of sample weights

[0080] Regarding the dynamic allocation of sample weights, a multi-objective weighted loss function is designed. Considering the performance requirements in the telecommunications sector and the prediction of service disruptions due to host outages, a balance between accuracy and sensitivity is needed. Therefore, the following weighted loss function is designed:

[0081] loss=γ1·L crossentropy +γ2·L dice +γ3·L focal ;

[0082] γ1+γ2+γ3=1;

[0083] Among them, L crossentropy L represents the cross-entropy loss function, which measures the difference between the model's predicted probability distribution and the true label probability distribution; diceThe DICE function primarily focuses on the overlap between the predicted and actual segmentation regions, thus effectively addressing the problem of data imbalance; L focal This represents the focus loss function, an improved loss function for the imbalanced sample problem. It is harmonicized using three different types of functions. The following are some key steps in determining the hyperparameters:

[0084] 1. First, determine the range of values ​​for hyperparameters γ1, γ2, and γ3 using a grid search.

[0085] 2. Configure according to the reliability focus of telecommunications services. When the upper layer service focuses on recall rate (downtime rate must be strictly controlled), the value of γ2 is increased to the highest, the value of γ1 is decreased to the lowest, and the remaining share of γ3 is taken proportionally.

[0086] 3. Configure according to the reliability focus of telecommunications services. When the upper layer service focuses on accuracy (to hit as many predicted faults as possible), the value of γ1 is increased to the highest, the value of γ2 is decreased to the lowest, and the remaining share of γ3 is taken proportionally.

[0087] 4. Configure according to the reliability focus of telecommunications services. When the upper layer services want to take both types of indicators into account, the value of γ3 should be increased to the highest, and the values ​​of γ1 and γ2 should be evenly distributed.

[0088] 5. After formal training, further optimize the parameters by combining feature engineering within the domain. When implementing this, attention should be paid to balancing the weight adjustment range with the model stability.

[0089] In this embodiment, steps 101 to 104 calculate a target fault score based on multiple alarm messages. If the target fault score is greater than a first preset threshold, the first service with a priority higher than a preset value among the multiple services of the first node is migrated to the second node. If the target fault score is greater than a second preset threshold but less than the first preset threshold, a first prompt message is output, indicating that the first node is at risk of downtime. This allows for flexible migration of the first service with a priority higher than a preset value among the multiple services of the first node to the second node, or outputting the first prompt message, depending on the target fault score, thus enhancing the flexibility of fault handling. Furthermore, this embodiment uses a tiered processing method based on the target fault score, i.e., by classifying and structuring the alarm messages, it reduces the failure rate of important services while also considering resource pool utilization.

[0090] As an optional implementation, the step of calculating the target fault score based on the multiple alarm information includes:

[0091] Calculate a first fault score corresponding to the multiple alarm messages in a fully correctable fault mode, and calculate a second fault score corresponding to the multiple alarm messages in a partially correctable fault mode.

[0092] The target fault score is calculated based on the first fault score and the second fault score.

[0093] The first fault score can be referred to as the fully correctable CE score θ1, and the second fault score can be referred to as the partially correctable CE score θ2. The calculation process for the first and second fault scores can be found in the following description:

[0094] The calculation process for the first fault score is as follows:

[0095] (1) Address Record: For out-of-band / in-band alarm information that contains both physical address information and correctable error information, record the chip, rank, bank, and row ID number of the corresponding unit, line by line.

[0096] (2) Cumulative count rule: If the error message contains the cumulative number of errors that can be corrected by counting a single bit, then take that number directly as the cumulative count; if the error message does not contain it, then set it to 1.

[0097] (3) Fault mode rules: Based on the specific content of out-of-band / in-band alarm information, faults are classified into the following categories:

[0098] 1. Random Bit Flip: The state of a memory cell changes by one or more bits due to hardware failure, radiation (such as cosmic rays), or electrical noise. The number of occurrences is merged row by row, with the total number of occurrences per row set to N, and a weight set to [value missing]. This type of fault usually reflects an abnormal hardware state, and multiple alarms may be generated for bits that have already experienced a fault. Therefore, this situation needs to be considered, and spatial weights need to be constructed and adjusted based on entropy and spatial distance.

[0099] The calculation method is as follows:

[0100] 1) Normalize the attribute values ​​at each point:

[0101]

[0102] 2) Calculate information entropy:

[0103]

[0104] Where E∈[0,1], E=0 means that all N are concentrated in one row (lowest diversity), and E=1 means that N are uniformly distributed (highest diversity).

[0105] 3) Entropy weight adjustment, the weight correction factor is:

[0106]

[0107] 4) The final random bit flip is:

[0108]

[0109] 2. Transient faults: These include electrical interference (such as bus noise, signal interference), high-energy particle impacts (such as alpha particles), etc., which may lead to single-bit or multi-bit errors. The number of occurrences is set to η², and the weight is set to ω². This type of fault is usually caused by external factors leading to instability. It is necessary to consider merging the occurrences to eliminate the impact of batch faults. When more than 3 memory locations simultaneously experience K faults within a very short time window... n If there are more than 10 alarms, the count will be set to K, and the calculation method is as follows:

[0110] 1) Alarm count within the sliding window, for any time point t∈[t start ,t end -Δt], sum of all alarm counts within the statistics window:

[0111]

[0112] 2) External influence triggers if there are at least 3 distinct positions p1, p2, p3 ∈ P within the window, satisfying:

[0113] C(pi,t)≥N(i=1,2,3);

[0114] This triggers an external influence flag, setting the total number of occurrences within this time period to K. The formula defines an indicator function I(·), which takes the value 1 when the condition is met and 0 otherwise:

[0115]

[0116] The final score for transient failure is:

[0117] score2=(K·I(·)+Total normal (t)·(1-I(·)))·ω2;

[0118] 3. Chip-level soft errors: These are caused by alpha particles released from the decay of radioactive elements (such as uranium or thorium) within the memory chip package, leading to abnormal cell charge states. Let the total number of occurrences be N, and the number of errors per line be n. i The weight is set to ω3. This type of fault is a major factor in correctable memory errors. This type of fault needs to reflect the randomness of decay and the characteristics of soft error propagation. The calculation method is as follows:

[0119]

[0120] Where α, β, and γ are adjustment coefficients adjusted according to the memory scenario, τ is the time window length, δ is the distribution penalty factor, and Var(n) i ) represents the variance of the number of errors per row, to reflect the uniformity of the error distribution.

[0121] 4. System-level soft errors: Caused by electromagnetic interference (EMI), power supply fluctuations, or data bus noise, potentially leading to misreading or writing of memory data. These faults actually involve a wider range of factors and can be scored from the perspectives of cumulative occurrence over time and distribution variance. Let the total number of occurrences be N, and the number of single-line errors be n. i The weight is set to ω4. The calculation method is as follows:

[0122]

[0123] Where, λ t It is the time weighting coefficient, λ var It is the memory fault distribution coefficient, τ is the statistical time window length, Var(n) i ) represents the variance of the number of errors per row.

[0124] 5. Timing Issues: Incorrect memory configuration parameters (such as CAS latency) can cause abnormal read / write timing, potentially leading to system instability. The total number of occurrences is N, the number of occurrences for each row is n, and the weight is set to ω5. The calculation method is as follows:

[0125]

[0126] Where, λ t These are the time-series weighting coefficients; w(p) i ) represents spatial weight, with higher weight assigned to memory space covered by the CPU passthrough; β is the error count decay factor, which can reduce the weight for older faults that occurred long ago; α is the timing deviation amplification factor, adjusting the impact of deviation on the score; γ is the global timing deviation factor; Δt i This represents the absolute deviation between the actual and nominal values ​​of timing parameters (such as CAS latency) in the i-th row. The overall score for the entire timing category is characterized by a comprehensive consideration of timing-sensitive decay in memory "row" units and global timing.

[0127] 6. Other. The total number of occurrences is set as η6, and the weight is set as ω6. The calculation method is as follows:

[0128] score6 = η6·ω6;

[0129] In summary, the fully correctable CE score (i.e., the first fault score) based on memory is the sum of the scores of the above six categories.

[0130] The calculation process for the second fault score is as follows:

[0131] First, it is necessary to correctly identify some corrective alarms. The judgment criteria are as follows:

[0132] A. Error Type Identifier: The log may explicitly record the error type as related to partial correction. For example, the appearance of identifiers such as "Partial ECC Correction" or "ECC Partially Successful" clearly indicates that the ECC is in a partial correction state. The server log will specifically state "Memory error detected: ECC attempted correction, but only partial success. Error pattern indicates complexity beyond full correction capabilities," clearly presenting the nature of the error and the ECC processing result.

[0133] B. Error Bit Information Recording: To assist in analyzing the cause of errors and assessing memory status, the log may record detailed information such as the location and number of error bits. For example, "Error bits located in multiple data pins and beats, total error bits: [X]", where "multiple data pins and beats" reflects the complexity of the error distribution, classifying this fault as a partially corrected error.

[0134] C. Frequency and Timestamp Recording: To track memory error trends, the log records the frequency of partially corrected errors and the timestamp of each occurrence. If the log contains "errors occurred [X] times within [timeperiod]", it is considered a partially corrected error.

[0135] (1) Address Record: For out-of-band / in-band alarm information that contains both physical address information and correctable error information, record the chip, rank, bank, and row ID number of the corresponding unit, line by line.

[0136] (2) Cumulative count rule: If the error message contains the cumulative number of errors that can be corrected by counting a single bit, then take that number directly as the cumulative count; if the error message does not contain it, then set it to 1.

[0137] (3) Fault mode rules: Based on the specific content of out-of-band / in-band alarm information, faults are classified into the following categories:

[0138] 1. Random Bit Flip: Memory cell states change in one or more bits due to hardware failure, radiation (such as cosmic rays), or electrical noise. The number of occurrences is merged row by row, with the total number of occurrences per row set as N, and the weight set as ω1. This type of fault usually reflects abnormal hardware status, and multiple alarms may be generated for bits that have already failed. Therefore, this situation needs to be considered, and spatial weights need to be constructed and adjusted based on entropy and spatial distance. For partially corrected errors (CEs), the following factors need to be considered: first, the number and proportion of uncorrected bits; second, the number of times a single error triggers a repair.

[0139] The calculation method is as follows:

[0140] 1) Normalize the attribute values ​​at each point:

[0141]

[0142] 2) Calculate information entropy:

[0143]

[0144] Where E∈[0,1], E=0 means that all N are concentrated in one row (lowest diversity), and E=1 means that N are uniformly distributed (highest diversity).

[0145] 3) Entropy weight adjustment, the weight correction factor is:

[0146]

[0147] 4) The final score for random bit flips needs to consider additional dimensions: the number of uncorrected bits u, the proportion p, and the number of times a single error triggers a repair m.

[0148]

[0149] Among them, the sum of the weighting coefficients ρ1 and ρ2 is 1, and k1, k2, and k3 are control coefficients to keep the total score within a certain range.

[0150] 2. Transient Faults: These include electrical interference (such as bus noise, signal interference), high-energy particle impacts (such as alpha particles), etc., which may lead to single-bit or multi-bit errors. The number of occurrences is set as η², and the weight is set as ω². This type of fault is usually caused by external factors leading to state instability, and the number of occurrences needs to be merged to eliminate the impact of batch faults. For partially corrected errors, the system will attempt to repair multiple times (if all repairs are successful, it belongs to fully correctable CE). In this case, the factor of multiple partial correction repairs needs to be added. The first point is the recovery interval: the time difference between two repairs. A short interval indicates that the system continuously triggers the repair mechanism, which may mask the hardware degradation trend. A long interval indicates a delay in the repair mechanism response, which may allow errors to accumulate. The second point is the reproducibility rate: the number of recurrences per unit time. A high reproducibility rate indicates that the repair mechanism has failed or the hardware is deteriorating rapidly. A low reproducibility rate indicates occasional interference or temporary faults.

[0151] If more than three memory locations simultaneously contain K within a very short time window... n If there are more than 10 alarms, the count will be set to K, and the calculation method is as follows:

[0152] 1) Alarm count within the sliding window, for any time point t∈[t start ,t end -Δt], sum of all alarm counts within the statistics window:

[0153]

[0154] 2) External influence triggers if there are at least 3 distinct positions p1, p2, p3 ∈ P within the window, satisfying:

[0155] C(pi,t)≥N(i=1,2,3);

[0156] This triggers an external influence flag, setting the total number of occurrences within this time period to K. The formula defines an indicator function I(·), which takes the value 1 when the condition is met and 0 otherwise:

[0157]

[0158] 3) Recovery time interval: Calculate the average time interval D between multiple memory CE failures occurring at the same location, and sum the risk coefficients for all locations on the memory module. k is the control coefficient. t For harmonic parameters:

[0159]

[0160] 4) Recurrence rate tracking: Count the highest number of recurrences Q of the same position within the statistical period, and sum them up to obtain the total number of recurrences ∑Q;

[0161] The final score for transient failure is:

[0162] score2=(K·I(·)+Total normal (t)·(1-I(·))+t during +∑Q)·ω2;

[0163] 3. Chip-level soft errors: These are caused by alpha particles released from the decay of radioactive elements (such as uranium or thorium) within the memory chip package, leading to abnormal cell charge states. Let the total number of occurrences be N, and the number of errors per line be n. i The weight is set to ω3. This type of fault is a major factor in correctable memory errors. This type of fault needs to reflect the randomness of decay and the characteristics of soft error propagation. The calculation method is as follows:

[0164]

[0165] Where α, β, and γ are adjustment coefficients adjusted according to the memory scenario, τ is the time window length, δ is the distribution penalty factor, and Var(n) i ) represents the variance of the number of errors per row, to reflect the uniformity of the error distribution.

[0166] 4. System-level soft errors: Caused by electromagnetic interference (EMI), power supply fluctuations, or data bus noise, potentially leading to misreading or writing of memory data. These faults actually involve a wider range of factors and can be scored from the perspectives of cumulative occurrence over time and distribution variance. Let the total number of occurrences be N, and the number of single-line errors be n. i The weight is set to ω4. The calculation method is as follows:

[0167]

[0168] Where, λ t It is the time weighting coefficient, λ var It is the memory fault distribution coefficient, τ is the statistical time window length, Var(n) i ) represents the variance of the number of errors per row.

[0169] 5. Timing Issues: Incorrect memory configuration parameters (such as CAS latency) can cause abnormal read / write timing, potentially leading to system instability. The total number of occurrences is N, the number of occurrences for each row is n, and the weight is set to ω5. The calculation method is as follows:

[0170]

[0171] Where, λ t These are the time-series weighting coefficients; w(p) i) represents spatial weight, with higher weight assigned to memory space covered by the CPU passthrough; β is the error count decay factor, which can reduce the weight for older faults that occurred long ago; α is the timing deviation amplification factor, adjusting the impact of deviation on the score; γ is the global timing deviation factor; Δt i This represents the absolute deviation between the actual and nominal values ​​of timing parameters (such as CAS latency) in the i-th row. The overall score for the entire timing category is characterized by a comprehensive consideration of timing-sensitive decay in memory "row" units and global timing.

[0172] 6. Other. The total number of occurrences is set as η6, and the weight is set as ω6. The calculation method is as follows:

[0173] score6=η6·ω6

[0174] In summary, the partial correction CE score (i.e., the second fault score) based on memory is the sum of the scores of the above six categories.

[0175] In this embodiment of the application, the target fault score can be calculated based on the first fault score and the second fault score, thereby improving the accuracy of the target fault score.

[0176] It should be noted that the specific method for calculating the target fault score based on the first fault score and the second fault score is not limited here. Optionally, the sum of the first fault score and the second fault score can be directly determined as the target fault score.

[0177] As an optional implementation, calculating the target fault score based on the first fault score and the second fault score includes:

[0178] A first weighting coefficient and a second weighting coefficient are obtained, wherein the first weighting coefficient and the second weighting coefficient are determined based on the importance of the fully correctable fault mode and the partially correctable fault mode, respectively.

[0179] Calculate the first product of the first fault score and the first weight coefficient, and calculate the second product of the second fault score and the second weight coefficient;

[0180] The sum of the first product and the second product is determined as the target fault score.

[0181] Optionally, the first weighting coefficient and the second weighting coefficient can be determined based on the importance of the fully correctable fault mode and the partially correctable fault mode, respectively. For example, the first weighting coefficient and the second weighting coefficient can be determined in a ratio of 1:2. Alternatively, the first weighting coefficient and the second weighting coefficient can be empirical coefficients, i.e., values ​​that can be obtained through summarization and calculation.

[0182] In this embodiment, a first product of a first fault score and a first weighting coefficient is calculated, and a second product of a second fault score and a second weighting coefficient is calculated; the sum of the first product and the second product is determined as the target fault score. This makes the calculation method of the target fault score more flexible and the accuracy of the calculated target fault score higher.

[0183] For example, the first weighting coefficient can be λ1, the second weighting coefficient can be λ2, and the target fault score can be represented by δ. Therefore, the calculation formula for the target fault score can be found below:

[0184] δ=λ1θ1+λ2θ2.

[0185] As an optional implementation, after outputting the first prompt message when the target fault score is greater than a second preset threshold and less than the first preset threshold, the method further includes:

[0186] The distribution density is obtained by analyzing the multiple alarm messages using a kernel density estimation algorithm.

[0187] Adjust the multiple services or the memory according to the distribution density.

[0188] Among them, the kernel density estimation (KDE) method can abstract the distribution of CE errors (i.e. multiple alarm messages) into a one-dimensional space, measure the distribution of CE errors, and then adjust multiple services or memory.

[0189] In this embodiment, multiple services or memory are adjusted according to the distribution density, which makes the adjustment results of multiple services or memory more accurate.

[0190] As an optional implementation, adjusting the plurality of services or the memory according to the distribution density includes:

[0191] When the distribution density is greater than a third preset threshold, the first memory page corresponding to the multiple alarm messages is closed, and the data stored on the first memory page is stored on the second memory page. The second memory page is an idle memory page, and the memory includes the first memory page and the second memory page.

[0192] When the distribution density is less than or equal to the third preset threshold, the probability of the first node experiencing a downtime failure within the target time period is predicted by an artificial intelligence model, and the multiple services are adjusted based on the information output by the artificial intelligence model.

[0193] When the distribution density is greater than the third preset threshold, it is considered that the probability of triggering a CE storm is high. The pages associated with these memory "rows" (i.e., the first memory pages) must be closed, and the data stored in the first memory pages will be replaced on the remaining free pages (i.e., the second memory pages).

[0194] Specifically, when the distribution density is less than or equal to a third preset threshold, the inference capabilities of the artificial intelligence model are used to predict the risk of system downtime in the future based on load conditions and memory error distribution. This artificial intelligence model can also be referred to as an Artificial Intelligence (AI) model.

[0195] In this application embodiment, the diversity and flexibility of adjusting multiple services or memory can be further increased.

[0196] As an optional implementation, when the distribution density is less than or equal to the third preset threshold, the probability of the first node experiencing a downtime failure within a target time period is predicted using an artificial intelligence model, and the multiple services are adjusted based on the information output by the artificial intelligence model, including:

[0197] When the artificial intelligence model outputs a second prompt message, the multiple services are sequentially adjusted to the second node. The second prompt message indicates that the probability of the first node experiencing a downtime failure is greater than a preset probability within the target time period.

[0198] When the artificial intelligence model outputs a third prompt, the current node corresponding to the multiple services is marked as a high-risk node, and the second service with a priority lower than a preset value among the multiple services is migrated to the second node. The third prompt indicates that the probability of the first node experiencing a downtime failure within the target time period is less than or equal to the preset probability.

[0199] If the AI ​​model predicts that the system will crash within the target time period, services will be gradually moved to a safe node, i.e., multiple services will be moved to the second node in sequence. Otherwise, the system will continue to be used, and the node will be marked as a high-risk node. Only low-priority services can be scheduled. After a time period, the system will be tested again, i.e., the current nodes corresponding to multiple services will be marked as high-risk nodes, and the second service with a priority lower than a preset value among the multiple services will be migrated to the second node.

[0200] In this application embodiment, the diversity and flexibility of adjusting multiple services or memory can be further increased.

[0201] It should be noted that, optionally, the identification process for the above-mentioned multiple services can be described as follows:

[0202] The network element system of a business is highly complex, and each network element is not entirely consistent. The same type of network element often serves services in different geographical regions across different resource pools. Therefore, the number of users and traffic carried by a network element often varies. Businesses adjust specifications and other parameters as needed; some adjust the number of virtual machines / containers, while others deploy multiple replicas. Therefore, a universal importance identification and scoring scheme is proposed. Assume a network element has virtual machines θ1, θ2…θ… n To differentiate the priorities of virtual machines, important and critical components are assigned a larger percentage to indicate their importance, and so on, assigning a percentage φ1, φ2...φ to each virtual machine distribution. n ,in:

[0203]

[0204] Each network element is also assigned an overall weight score τ. Therefore, the final score of a single virtual machine is τφ. A threshold ν is set. When the score τφ of a single virtual machine / container is higher than ν, the component is considered a critical component with high importance. When the hardware is in a sub-healthy state, it is preferentially evacuated to other healthy nodes.

[0205] To more fully illustrate the above embodiments, a specific embodiment is given below as an example. For detailed steps, please refer to [link to specific examples]. Figure 2 The specific steps shown will not be repeated here.

[0206] See Figure 3 , Figure 3 This is a structural diagram of the memory fault handling device provided in the embodiments of this application, as shown below. Figure 3 As shown, the memory fault handling device 300 includes:

[0207] The acquisition module 301 is used to acquire multiple alarm information of the memory of the first node, wherein the alarm information is used to represent CE information;

[0208] The first calculation module 302 is used to calculate the target fault score based on the multiple alarm information;

[0209] The migration module 303 is used to migrate the first service with a priority higher than a preset value among the multiple services of the first node to the second node when the target fault score is greater than a first preset threshold. The second node is a node that has not experienced a fault.

[0210] The output module 304 is used to output a first prompt message when the target fault score is greater than a second preset threshold and less than the first preset threshold. The first prompt message is used to indicate that the first node is at risk of downtime.

[0211] As an optional implementation, the first computing module 302 includes:

[0212] The first calculation submodule is used to calculate a first fault score corresponding to the multiple alarm information in a fully correctable fault mode, and to calculate a second fault score corresponding to the multiple alarm information in a partially correctable fault mode.

[0213] The second calculation submodule is used to calculate the target fault score based on the first fault score and the second fault score.

[0214] As an optional implementation, the second computing submodule includes:

[0215] An acquisition unit is used to acquire a first weighting coefficient and a second weighting coefficient, wherein the first weighting coefficient and the second weighting coefficient are determined based on the importance of the fully correctable fault mode and the partially correctable fault mode, respectively.

[0216] The calculation unit is configured to calculate a first product of the first fault score and the first weight coefficient, and to calculate a second product of the second fault score and the second weight coefficient.

[0217] A determining unit is configured to determine the sum of the first product and the second product as the target fault score.

[0218] As an optional implementation, the memory fault handling device 300 further includes:

[0219] The analysis module is used to analyze the multiple alarm messages using a kernel density estimation algorithm to obtain the distribution density;

[0220] An adjustment module is used to adjust the multiple services or the memory according to the distribution density.

[0221] As an optional implementation, the adjustment module includes:

[0222] The shutdown submodule is used to shut down the first memory page corresponding to the multiple alarm messages when the distribution density is greater than a third preset threshold, and to store the data stored on the first memory page on the second memory page. The second memory page is an idle memory page. The memory includes the first memory page and the second memory page.

[0223] The adjustment submodule is used to predict the probability of the first node experiencing downtime within a target time period using an artificial intelligence model when the distribution density is less than or equal to the third preset threshold, and to adjust the multiple services based on the information output by the artificial intelligence model.

[0224] As an optional implementation, the adjustment submodule includes:

[0225] The adjustment unit is used to sequentially adjust the multiple services to the second node when the artificial intelligence model outputs the second prompt information. The second prompt information indicates that the probability of the first node experiencing a downtime failure is greater than a preset probability within the target time period.

[0226] The migration unit is used to mark the current node corresponding to the multiple services as a high-risk node when the artificial intelligence model outputs a third prompt message, and to migrate the second service with a priority lower than a preset value among the multiple services to the second node. The third prompt message indicates that the probability of the first node experiencing a downtime failure within the target time period is less than or equal to the preset probability.

[0227] The memory fault handling device 300 can realize the functions described in the embodiments of this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.

[0228] This application also provides an electronic device. Please refer to [link to relevant documentation]. Figure 4 The electronic device may include a processor 401, a memory 402, and a program 4021 stored in the memory 402 and executable on the processor 401. When the electronic device is a server, the program 4021, when executed by the processor 401, can achieve... Figure 1 Any steps in the corresponding method embodiments and the achievement of the same beneficial effects will not be repeated here.

[0229] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a readable medium. This application also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the above-described methods. Figure 1 Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0230] The storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0231] This application also provides a computer program product, including computer instructions, which, when executed by a processor, can perform the above-described functions. Figure 1Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0232] The above description represents the preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A memory fault handling method, characterized in that, include: Obtain multiple alarm messages from the memory of the first node, wherein the alarm messages are used to indicate correctable error (CE) messages; Calculate the target fault score based on the aforementioned multiple alarm information; If the target fault score is greater than a first preset threshold, the first service with a priority higher than a preset value among the multiple services of the first node will be migrated to the second node, which is a node that has not experienced a fault. If the target fault score is greater than the second preset threshold and less than the first preset threshold, a first prompt message is output, which indicates that the first node is at risk of downtime.

2. The method according to claim 1, characterized in that, The calculation of the target fault score based on the multiple alarm information includes: Calculate a first fault score corresponding to the multiple alarm messages in a fully correctable fault mode, and calculate a second fault score corresponding to the multiple alarm messages in a partially correctable fault mode. The target fault score is calculated based on the first fault score and the second fault score.

3. The method according to claim 2, characterized in that, The step of calculating the target fault score based on the first fault score and the second fault score includes: A first weighting coefficient and a second weighting coefficient are obtained, wherein the first weighting coefficient and the second weighting coefficient are determined based on the importance of the fully correctable fault mode and the partially correctable fault mode, respectively. Calculate the first product of the first fault score and the first weight coefficient, and calculate the second product of the second fault score and the second weight coefficient; The sum of the first product and the second product is determined as the target fault score.

4. The method according to any one of claims 1 to 3, characterized in that, After outputting a first prompt message when the target fault score is greater than a second preset threshold and less than the first preset threshold, the method further includes: The distribution density is obtained by analyzing the multiple alarm messages using a kernel density estimation algorithm. Adjust the multiple services or the memory according to the distribution density.

5. The method according to claim 4, characterized in that, The adjustment of the multiple services or the memory according to the distribution density includes: When the distribution density is greater than a third preset threshold, the first memory page corresponding to the multiple alarm messages is closed, and the data stored on the first memory page is stored on the second memory page. The second memory page is an idle memory page, and the memory includes the first memory page and the second memory page. When the distribution density is less than or equal to the third preset threshold, the probability of the first node experiencing a downtime failure within the target time period is predicted by an artificial intelligence model, and the multiple services are adjusted based on the information output by the artificial intelligence model.

6. The method according to claim 5, characterized in that, When the distribution density is less than or equal to the third preset threshold, the probability of the first node experiencing a downtime failure within a target time period is predicted using an artificial intelligence model, and the multiple services are adjusted based on the information output by the artificial intelligence model, including: When the artificial intelligence model outputs a second prompt message, the multiple services are sequentially adjusted to the second node. The second prompt message indicates that the probability of the first node experiencing a downtime failure is greater than a preset probability within the target time period. When the artificial intelligence model outputs a third prompt, the current node corresponding to the multiple services is marked as a high-risk node, and the second service with a priority lower than a preset value among the multiple services is migrated to the second node. The third prompt indicates that the probability of the first node experiencing a downtime failure within the target time period is less than or equal to the preset probability.

7. A memory fault handling device, characterized in that, include: The acquisition module is used to acquire multiple alarm information of the memory of the first node, and the alarm information is used to represent CE information; The first calculation module is used to calculate the target fault score based on the multiple alarm information; The migration module is used to migrate the first service with a priority higher than a preset value among the multiple services of the first node to the second node when the target fault score is greater than a first preset threshold. The second node is a node that has not experienced a fault. The output module is used to output a first prompt message when the target fault score is greater than a second preset threshold and less than the first preset threshold. The first prompt message is used to indicate that the first node is at risk of downtime.

8. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps of the memory fault handling method as described in any one of claims 1 to 6.

9. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the memory fault handling method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the memory fault handling method as described in any one of claims 1 to 6.

Images