Fault core processing method, system, device, storage medium and program product

Abstract

The application discloses a fault core processing method, system, device, storage medium and program product, relates to the processor technical field, and an ARM server comprises a plurality of processor cores, an operating system, a system control processor and a trusted firmware. Each processor core has an independent external power supply. The external power supply of each processor core is controlled by a plurality of power supply policy units corresponding to the external power supply. When the operating system receives a fault event of an uncorrectable non-fatal error, the operating system sends a core offline command to the trusted firmware. The trusted firmware sends the core offline command to all processor cores. The processor cores that can normally respond send the core offline command to the system control processor. The system control processor sends an offline signal to the power supply policy unit corresponding to the fault core based on the core offline command, so as to realize power-off control or hot reset of the fault core. Thus, the technical problem that the fault core processing mode can cause system stability risk is solved.

Description

Technical Field

[0001] This application relates to the field of processor technology, and in particular to a fault core processing method, system, device, storage medium, and program product. Background Technology

[0002] Current ARM (Advanced RISC Machine, a processor architecture based on a reduced instruction set architecture) servers typically rely on the faulty core itself to perform the shutdown operation when handling uncorrectable non-fatal errors. Specifically, the operating system or firmware sends instructions to the faulty core, which then actively writes to its internal power management register to complete context saving, interrupt disabling, and low-power state switching, thereby achieving logical shutdown.

[0003] However, the above method requires the faulty core to have instruction execution capability and be able to respond to and complete register write operations. If the faulty core loses its responsiveness due to a serious anomaly (such as pipeline lock-up) and is unable to execute the shutdown instruction, the shutdown process cannot be completed, causing the faulty core to remain in an uncertain state and posing a risk to system stability. Summary of the Invention

[0004] The main objective of this application is to provide a fault kernel handling method, system, device, storage medium, and program product, which aims to solve the technical problem that fault kernel handling methods may cause system stability risks.

[0005] To achieve the above objectives, this application proposes a fault core handling method applied to an ARM server. The ARM server includes multiple processor cores, an operating system, a system control processor, and trusted firmware. Each processor core has an independent external power supply, which is controlled by multiple corresponding power policy units. The fault core handling method includes: When the operating system receives a fault event that is an uncorrectable non-fatal error, the operating system sends a core offline command to the trusted firmware; The trusted firmware sends the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor; The system control processor sends a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, so as to realize the power-off control or thermal reset of the faulty core.

[0006] In one embodiment, after the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the method further includes: The operating system marks the faulty core as offline; The system control processor updates the number of times the faulty core goes offline and determines whether the number of times the faulty core goes offline after the update exceeds a preset threshold. If it does not exceed the threshold, the faulty core is powered on or restarted after a preset cooling period. The trusted firmware notifies the operating system that the faulty core has been recovered via a software delegated exception interface. The operating system will reintegrate the recovered processor core into the scheduling resource pool.

[0007] In one embodiment, before the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the method further includes: The operating system is prompted to save the cached data of the faulty core to prevent the cached data from being cleared when the power policy unit controls the power outage of the faulty core; After the operating system marks the faulty core as offline, the method further includes: The system control processor records the fault information and number of times the fault event occurred, so that administrators can perform fault analysis.

[0008] In one embodiment, before the step of the operating system marking the faulty core as offline, the method further includes: The operating system responds to the count threshold configuration operation to obtain the preset count threshold; The preset number of times threshold is loaded into the internal register of the system control processor.

[0009] In one embodiment, before the step of the operating system sending a core offline command to the trusted firmware when the operating system receives a fault event of an uncorrectable non-fatal error, the method further includes: The operating system acquires the temporal electrical and thermal characteristics of the faulty core within a preset time window, and simultaneously collects the associated electrical and thermal characteristics of at least one physically adjacent processor core. The operating system analyzes the time-series electrical and thermal characteristics and related electrical and thermal characteristics through a preset fault judgment model to obtain fault analysis results. The fault analysis results include physical faults and external faults. The preset fault judgment model is obtained by iterative training based on historical fault data. After the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the system further includes: When the fault analysis result indicates an external fault and the external cause is detected to be eliminated, the system control processor is powered on again or the faulty core is restarted.

[0010] In one embodiment, the step of the operating system analyzing the time-series electrical-thermal characteristics and associated electrical-thermal characteristics using a preset fault judgment model to obtain fault analysis results includes: Based on the time-series electrical and thermal characteristics and the associated electrical and thermal characteristics, the operating system separates the global interference component reflecting systemic environmental fluctuations and the local fault component reflecting processor core abnormalities through the preset fault judgment model. If the total proportion of the global interference component in the time-series electrical-thermal features and the associated electrical-thermal features exceeds a preset threshold, and there is a synchronous fluctuation feature between the faulty core and the neighboring processor core, then the fault analysis result is determined to be an external fault.

[0011] In addition, to achieve the above objectives, this application also proposes an ARM server, which includes multiple processor cores, an operating system, a system control processor, and trusted firmware. Each processor core has an independent external power supply, and the external power supply of each processor core is controlled by multiple corresponding power strategy units. The operating system is used to send a core offline command to the trusted firmware when it receives a fault event of an uncorrectable non-fatal error. The trusted firmware is used to send the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor. The system control processor is used to send a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, so as to realize the power-off control or hot reset of the faulty core.

[0012] In addition, to achieve the above objectives, this application also proposes a fault kernel processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the fault kernel processing method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the fault kernel processing method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the fault kernel processing method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: The ARM server includes multiple processor cores, an operating system, a system control processor, and trusted firmware. Each processor core has an independent external power supply, which is controlled by multiple corresponding power policy units. When the operating system receives an uncorrectable non-fatal error, it sends a core offline command to the trusted firmware. The trusted firmware then sends the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor.

[0016] It is understandable that trusted firmware sends commands to all processor cores, not just the faulty core; even if the faulty core may not be able to respond to or forward the command, there are still processor cores in the ARM server that can respond normally. These processor cores that can respond normally act as relay nodes to forward the core offline command to the system control processor.

[0017] Furthermore, since each core has an independent external power supply and is controlled by an independent power strategy unit, the power-off operation for a faulty core does not require passing through the internal circuitry or registers of the faulty core. The system control processor directly sends a signal to the power strategy unit corresponding to the faulty core. Based on the core offline command, the system control processor sends an offline signal to the power strategy unit corresponding to the faulty core to achieve power-off control or hot reset of the faulty core. After receiving the signal, the power strategy unit directly cuts off the external power supply to the faulty core or performs a hot reset. This action is entirely completed by external hardware, without requiring the faulty core to execute any instructions or write to any internal registers. Even if the faulty core loses its instruction execution and responsiveness capabilities, its power supply can still be directly cut off or a hot reset can be performed through the external power strategy unit, avoiding system stability risks. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the fault handling method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the fault kernel handling method of this application; Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the fault kernel processing method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or ARM server capable of performing the above functions. The following description uses an ARM server as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide a fault kernel processing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the fault kernel handling method of this application.

[0026] In this embodiment, the fault kernel processing method includes steps S10 to S30: Step S10: When the operating system receives a fault event of an uncorrectable non-fatal error, the operating system sends a core offline command to the trusted firmware. Step S20: The trusted firmware sends the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor. In step S30, the system control processor sends a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, so as to realize the power-off control or thermal reset of the faulty core.

[0027] It should be noted that the fault core handling method is applied to an ARM server, which includes multiple processor cores (CORE), an operating system (OS), a system controller processor (SCP), a power policy unit (PPU), and trusted firmware (ARM Trusted Firmware, ATF). Each processor core has an independent external power supply, and the external power supply of each processor core is controlled by the corresponding multiple power policy units.

[0028] Trusted firmware refers to secure firmware running at the highest privilege level (EL3) of the ARM processor, which is responsible for handling secure calls, power management, and interaction with the underlying hardware.

[0029] When an ARM server is running, if a hardware error occurs in a processor core, the underlying hardware will capture the error and the trusted firmware will perform initial processing. The trusted firmware classifies errors according to their severity, including Corrected Error (the operating system simply prints the error information without performing any other processing), Uncorrected Fatal (the operating system enters a kernel panic and crashes and restarts), and Uncorrected Non-Fatal (uncorrectable non-fatal error).

[0030] If the trusted firmware determines that the hardware error is an uncorrectable non-fatal error, it will report the hardware error as a fault event to the operating system. After receiving this fault event indicating an uncorrectable non-fatal error, the operating system determines that the corresponding faulty core needs to be processed, generates a core shutdown command, and sends the core shutdown command to the trusted firmware.

[0031] The core offline command is an instruction used to instruct a specific processor core to be taken offline.

[0032] Specifically, after receiving a core offline command, the trusted firmware broadcasts the command to all processor cores. All processor cores capable of responding to interrupts and instructions act as relay nodes. When these responsive processor cores receive the core offline command from the trusted firmware, they further forward it to the control processor.

[0033] As can be understood, the trusted firmware sends the core offline command to all processor cores, and any responsive core forwards it to the system control processor, thus completely eliminating the reliance on the faulty core's own responsiveness. As long as at least one other healthy processor core in the ARM server can respond to the trusted firmware's broadcast, the core offline command can be successfully delivered to the system control processor. This improves the stability of the fault handling process, ensuring reliable isolation of the faulty core even if it completely loses its ability to function.

[0034] Furthermore, a processor core that can respond normally forwards the core offline command to the system control processor. As the unified manager of the power supply for all processor cores, the system control processor, upon receiving this core offline command, will parse the command and identify the faulty core that needs to be offline.

[0035] The system control processor directly sends a shutdown signal to the power policy unit corresponding to the faulty core. The shutdown signal instructs the power policy unit to perform hardware-level power operations, such as completely cutting off the external power supply to the faulty core (power-off control), causing it to stop working completely, or performing a rapid power-off and power-on (warm reset), allowing it to undergo a complete power-on cycle, thereby clearing all its internal volatile states; thus completing the physical isolation or state clearing of the faulty core. This fundamentally solves the limitation of traditional solutions that rely on the faulty core itself to execute software shutdown instructions.

[0036] In one feasible implementation, before the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the method further includes: The operating system is prompted to save the cached data of the faulty core to prevent the cached data from being cleared when the power policy unit controls the power outage of the faulty core.

[0037] Cache data refers to instructions and data stored in the processor core's internal cache (such as L1 / L2 Cache).

[0038] Specifically, the system control processor or trusted firmware will proactively prompt the operating system, requesting it to save as much cached data in the faulty core cache as possible that has not yet been written to main memory. This allows the operating system time to write critical dirty data back to memory before the power policy unit performs power-down control or hot reset operations, preventing this cached data from being directly cleared and lost due to forced hardware power-off.

[0039] This embodiment adds a data saving prompt before a forced power failure, which can mitigate the risk of data loss that may result from a forced power failure. Although an uncorrectable non-fatal error indicates that the cached data may be abnormal, making an effort to save the parts that may not be corrupted still helps maintain file system consistency and reduce business logic errors.

[0040] In one feasible implementation, after the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the method further includes: The operating system marks the faulty core as offline; the system controls the processor to update the number of times the faulty core goes offline and determines whether the number of times the faulty core goes offline after the update exceeds a preset threshold. If it does not exceed the threshold, the faulty core is powered on or restarted after a preset cooling period; the trusted firmware notifies the operating system that the faulty core has recovered through a software delegated exception interface; the operating system re-includes the recovered processor core into the scheduling resource pool.

[0041] It's important to note that processor cores in an offline state will not be assigned any new tasks or processes by the operating system's scheduler. After the system control processor performs a power-off or hot reset on the faulty core via the power policy unit, the operating system marks the faulty core as offline, ensuring that subsequent task scheduling will not be assigned to this offline processor core.

[0042] Meanwhile, the system control processor maintains a counter for each processor core, recording the cumulative number of times that core has been taken offline due to uncorrectable non-fatal errors. After the operating system marks the faulty core as offline, the system control processor updates the offline count record for that core, includes it in the current offline event, and compares the updated offline count with a preset threshold. If the updated offline count does not exceed the preset threshold, it indicates that the core's failure frequency is within an acceptable range. After a preset cooling period, the system control processor automatically controls the corresponding power strategy unit to power on the faulty core or perform a restart operation.

[0043] After the faulty core successfully powers on and completes initialization, the trusted firmware will proactively send a notification to the operating system via the software delegated exception interface, informing it that the recovered processor core has been restored and is ready to function normally. Upon receiving this notification, the operating system will add the recovered processor core back to the scheduling resource pool, enabling it to be used again to execute computing tasks.

[0044] In this embodiment, for recoverable failures caused by transient environmental factors (such as instantaneous overtemperature or voltage spikes), the cause of the processor core failure may have disappeared after it has been offline and undergone a preset cooling period. At this point, by judging the number of offline attempts against a threshold, as long as the preset threshold is not exceeded, the system will automatically attempt to restore the core and put it back into operation. This avoids permanently removing cores offline due to transient failures from the system, thereby maximizing the availability of the system's total computing resources without affecting system stability.

[0045] If the preset threshold number of failures is exceeded, the faulty core will no longer be reused, preventing a potentially defective core from being repeatedly brought online and offline without restriction, thus avoiding system turbulence and instability caused by this.

[0046] In one feasible implementation, after the operating system marks the faulty core as offline, it further includes: The system control processor records the fault information and number of times the fault event occurred, so that administrators can perform fault analysis.

[0047] Specifically, the system control processor, acting as the manager of the underlying hardware status, records key information related to this fault event, including the specific fault details and the latest number of times the core has been offline, in its internal storage space or a designated shared memory area. This recorded data is not easily lost with a system restart and can be queried and retrieved by administrators through specific interfaces (such as out-of-band management tools and system logs) for subsequent fault analysis, system health assessment, and operational decisions.

[0048] In one feasible implementation, before the operating system marks the faulty core as offline, the method further includes: The operating system responds to the count threshold configuration operation, obtains the preset count threshold, and loads the preset count threshold into the internal register of the system control processor.

[0049] Specifically, the operating system will respond to the number of times threshold configuration operation initiated by the administrator or the upper-level operation and maintenance platform, and obtain the user's expected preset number of times threshold by parsing the parameters carried in the number of times threshold configuration operation.

[0050] Furthermore, the operating system is responsible for transmitting this preset threshold number of attempts through a standard communication interface (such as the SCMI protocol) and loading it into the internal register of the system control processor, and persistently storing it in the underlying hardware management unit so that the system control processor can directly read the preset threshold number of attempts in the register during subsequent operation and use it as the basis for determining whether the faulty core is allowed to come back online.

[0051] It is understandable that different application scenarios have different requirements for reliability and resource utilization: in high-performance computing scenarios, users may want to set a high threshold to maximize resource utilization and provide multiple recovery opportunities even if the core fails occasionally; while in critical business scenarios, users may set a low threshold or even zero to prioritize ensuring the absolute stability of the system and permanently isolate the core once a failure occurs.

[0052] In this embodiment, the faulty core is reliably taken offline by external power control, completely eliminating the dependence on the faulty core's own response capability; at the same time, a closed-loop mechanism of offline, recording, threshold judgment, and automatic recovery is constructed to realize intelligent processing of core resource recovery and maximize system availability.

[0053] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S10, the fault kernel processing method further includes steps S01~S02: Step S01: The operating system acquires the temporal electrical and thermal characteristics of the faulty core within a preset time window, and simultaneously collects the associated electrical and thermal characteristics of at least one physically adjacent neighboring processor core. Step S02: The operating system analyzes the time-series electrical and thermal characteristics and the associated electrical and thermal characteristics through a preset fault judgment model to obtain fault analysis results. The fault analysis results include physical faults and external faults. The preset fault judgment model is obtained by iterative training based on historical fault data.

[0054] It should be noted that the preset time window is a pre-defined time period ending at the time of the fault occurrence (e.g., 100 milliseconds before the fault occurs). The time-series electrical-thermal characteristics are data points related to electrical characteristics (such as voltage and current) and thermal characteristics (such as temperature) continuously collected from the faulty core within the preset time window before the fault occurs, forming a time series. Adjacent processor cores are one or more processor cores that are physically adjacent to the faulty core on the chip. Associated electrical-thermal characteristics are electrical and thermal characteristic data synchronously collected from adjacent processor cores within the same preset time window.

[0055] Specifically, when a processor core is about to experience or has just experienced an uncorrectable non-fatal error, the operating system acquires the timing-based electrical and thermal characteristics of that faulty core within a preset time window prior to the error, such as voltage fluctuation curves and temperature change curves. Simultaneously, the operating system also collects the associated electrical and thermal characteristics of at least one neighboring processor core physically adjacent to the faulty core.

[0056] Furthermore, the collected temporal electrical and thermal characteristics of the fault core and the associated electrical and thermal characteristics of neighboring cores are used as input data and fed into a preset fault judgment model. The preset fault judgment model is obtained in advance through iterative training of a large amount of historical fault data using machine learning algorithms and has the ability to identify the cause of the fault from complex data.

[0057] The preset fault diagnosis model performs in-depth analysis and comparison of the input time-series electrical and thermal characteristics and related electrical and thermal characteristics, and outputs fault analysis results. These results can clearly indicate whether the fault is a physical fault of hardware damage type (e.g., transistor breakdown, metal wire breakage) or an external fault induced by environmental factors (e.g., instantaneous voltage drop, instantaneous high temperature caused by poor local heat dissipation).

[0058] The training process of the preset fault judgment model includes the following steps: Historical fault data collection: During the operation of a large number of ARM servers, time-series electrical and thermal characteristics of the faulty core were collected within a preset time window (e.g., 100 milliseconds before the fault occurred) each time an uncorrectable non-fatal error occurred. This included sensor data such as core voltage, current, and temperature, with a sampling frequency of 1 kHz. Simultaneously, the associated electrical and thermal characteristics of at least one neighboring processor core physically adjacent to the faulty core were collected within the same time window. For each fault record, its root cause category was labeled through expert analysis or post-fault diagnosis: if it was permanent damage to the core's own hardware, it was marked as a physical fault; if it was induced by external environmental factors and could be recovered, it was marked as an external fault. All historical fault data constituted the original sample set.

[0059] Data preprocessing is performed on the original sample set: samples with excessive noise or incomplete time windows are removed; time-series data such as voltage and temperature are normalized to eliminate the influence of dimensions. Multidimensional features are extracted from the time-series data (including time-series electrical and thermodynamic characteristics and associated electrical and thermodynamic characteristics) of each fault sample. These multidimensional features include: Statistical characteristics: mean, variance, maximum, minimum, and rate of change of fault core voltage; mean, variance, maximum, minimum, and rate of change of neighboring core voltages; similar statistics for temperature characteristics.

[0060] Frequency domain characteristics: Perform a fast Fourier transform on the time series data to extract the main frequency components and their energy proportions.

[0061] Correlation characteristics: Calculate the Pearson correlation coefficient between the voltage sequences of the faulty core and the neighboring cores, the similarity of the temperature sequences, and the temporal synchronicity indicators of the two (such as the peak value of the cross-correlation function).

[0062] Fluctuation characteristics: The time-series signal of the fault core is separated into global interference components and local fault components by signal decomposition techniques (such as empirical mode decomposition or wavelet transform), and the energy ratio of global interference components and the abrupt change intensity of local fault components are calculated.

[0063] The aforementioned multidimensional features are combined into a feature vector, which, along with the corresponding fault category labels, forms training samples to construct and train a classification model. This embodiment uses the random forest algorithm as the basic architecture of the pre-defined fault judgment model. The random forest algorithm has good anti-overfitting ability and feature importance interpretability. The sample set is randomly divided into a training set and a test set in a 7:3 ratio. The random forest model is trained using the training set, and the model hyperparameters (such as the number of trees, maximum depth, and minimum number of leaf samples) are optimized through grid search and five-fold cross-validation. During training, the model learns the mapping relationship from features to fault categories.

[0064] After training, the model performance is evaluated using a test set, requiring that metrics such as classification accuracy and recall reach preset thresholds (e.g., accuracy ≥ 95%). If the performance does not meet the requirements, iterative optimization is performed by increasing the sample size, adjusting features, or changing the model (e.g., XGBoost, Support Vector Machine) until a preset fault judgment model that meets the requirements is obtained.

[0065] The specific implementation method for separating the time-series signal of the fault core into global interference components and local fault components using signal decomposition techniques (such as empirical mode decomposition or wavelet transform), and calculating the energy proportion of the global interference components and the abrupt change intensity of the local fault components, can be as follows: Taking the voltage timing data of a faulty core as an example. Assume that 100 voltage data points were collected at a sampling rate of 1kHz within a time window 100 milliseconds before the fault occurred, forming the original timing signal; this original timing signal contains core operating noise, system-level power fluctuations, and possible fault mutation information. Discrete wavelet transform is performed on the original timing signal by selecting wavelet basis functions (such as the db4 wavelet) and the number of decomposition levels (e.g., 3 levels).

[0066] Discrete wavelet transform decomposes the original time-series signal into components at different frequency scales: approximation coefficients (corresponding to the low-frequency components of the signal, reflecting the overall trend and slow changes of the signal), and detail coefficients (corresponding to the high-frequency components of the signal, reflecting local abrupt changes and subtle fluctuations; the frequency gradually decreases as the number of layers increases). Through wavelet reconstruction, the signal components corresponding to each layer can be obtained: low-frequency trend signals (reconstructed from the approximation coefficients, representing components that change slowly within a time window, such as slow voltage drift due to changes in the overall chip load, or slow rise in ambient temperature), and high-frequency detail signals (reconstructed from the detail coefficients of each layer, representing rapid fluctuations in different frequency bands. Among them, the highest frequency detail signal usually corresponds to instantaneous noise or abrupt changes).

[0067] The low-frequency trend signal and the low-frequency, long-duration detail signal are combined to form the global interference component. The global interference component reflects system-level environmental fluctuations with a wide impact, such as global voltage drops caused by power management unit voltage regulation, or overall temperature changes caused by the cooling system's operating cycle.

[0068] The portion of the highest frequency detail signal that exceeds a set threshold is considered a local fault component. Local fault components reflect sudden, localized anomalies that occur only on the faulty core, such as voltage spikes or drops caused by a momentary short circuit in a circuit unit within the core.

[0069] Calculate the sum of squares of each signal in the original time-series signal to obtain the total energy of the original time-series signal. Calculate the sum of squares of each signal in the global interference component to obtain the total energy of the global interference component. Calculate the ratio of the total energy of the global interference component to the total energy of the original time-series signal.

[0070] The ratio R of the total energy of the global interference component to the total energy of the original time-series signal reflects the degree to which environmental fluctuations dominate the overall signal variation. If the ratio R is large (e.g., exceeding 70%), it indicates that the signal variation is mainly caused by systematic environmental factors.

[0071] The rate of change L′(t) is obtained by differentiating the local fault component L(t). The maximum absolute value of the rate of change L′(t) within a preset time window is calculated as M = max(|L′(t)|), and this maximum value M is used as the mutation intensity index. This mutation intensity index reflects the severity of the core's own anomaly when the fault occurs. The higher the mutation intensity, the greater the impact on the processor core itself.

[0072] In one feasible implementation, the step of the operating system analyzing the time-series electrical-thermal characteristics and associated electrical-thermal characteristics using a preset fault judgment model to obtain fault analysis results includes: The operating system, based on the time-series electrical-thermal characteristics and the associated electrical-thermal characteristics, separates the global interference component reflecting systemic environmental fluctuations and the local fault component reflecting processor core anomalies through the preset fault judgment model. If the total proportion of the global interference component in the time-series electrical-thermal characteristics and the associated electrical-thermal characteristics exceeds a preset proportion threshold, and there is a synchronous fluctuation characteristic between the faulty core and the neighboring processor core, then the fault analysis result is determined to be an external fault.

[0073] Specifically, the operating system inputs the collected temporal electrical and thermal characteristics of the faulty core and the associated electrical and thermal characteristics of neighboring cores into a preset fault judgment model. The preset fault judgment model uses signal processing techniques (such as wavelet transform or empirical mode decomposition) to separate global interference components and local fault components from the temporal electrical and thermal characteristics and associated electrical and thermal characteristics.

[0074] After separation is completed, the preset fault judgment model determines the proportion of the global interference component in the total energy of all characteristic signals (including time-series electrical and thermal characteristics and associated electrical and thermal characteristics) and determines whether this proportion exceeds the preset proportion threshold; the judgment also analyzes whether there are synchronous fluctuation characteristics between the faulty core and the neighboring processor core.

[0075] If the proportion exceeds the preset threshold and there are synchronous fluctuation characteristics, that is, the global interference component dominates and the faulty core and its neighbors exhibit synchronous fluctuation patterns, the preset fault judgment model determines that the fault is not caused by damage to the core itself, but by common external environmental factors, thus identifying the fault analysis result as an external fault.

[0076] This embodiment utilizes the inherent characteristic that physically adjacent cores are affected by the same environmental factors. When a voltage drop or temperature shock occurs, it is unlikely that only a single core is affected, but rather all cores within a certain area are affected. Therefore, when the faulty core and its neighboring cores exhibit synchronous characteristics dominated by global disturbance components, the fault can be inferred to originate from an external source with a high degree of confidence. This improves the accuracy of fault diagnosis.

[0077] In one feasible implementation, after the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the method further includes: When the fault analysis result indicates an external fault and the external cause is detected to be eliminated, the system control processor is powered on again or the faulty core is restarted.

[0078] The system control processor continuously monitors the environmental status of the ARM server and makes a comprehensive judgment based on the fault analysis results. If the fault analysis results indicate an external fault, and the system control processor detects that the external factors causing the fault (such as voltage fluctuations, excessive temperature, etc.) have returned to normal, the system control processor will actively trigger the recovery process, control the corresponding power strategy unit to power on the faulty core again or perform a restart operation, so that it can return to the working state.

[0079] Based on time (such as a fixed cooldown period) or number of attempts (such as the number of offline attempts not exceeding a threshold), blind retries may occur under harsh conditions, or recovery opportunities may be missed when conditions have recovered. This embodiment avoids permanent waste of core resources due to external failures by adopting the above method, while preventing ineffective recovery attempts under continuously deteriorating environmental conditions, thus maximizing the optimal balance between system reliability and resource utilization.

[0080] In one feasible implementation, the system control processor continuously monitors the real-time electrical and thermal data of the faulty core and the adjacent processor core, and extracts the real-time global interference component from the real-time electrical and thermal data; compares the energy of the real-time global interference component with a preset reference energy threshold; when the energy of the real-time global interference component is lower than the reference energy threshold for N consecutive sampling points, and the real-time fluctuation synchronization index of the faulty core and the adjacent processor core is lower than a preset synchronization threshold, it is determined that the external cause has been eliminated.

[0081] Specifically, the system collects real-time electrical and thermal data of the offline faulty core, and simultaneously collects real-time electrical and thermal data of its neighboring processor cores. For each set of real-time data, the system control processor calls the same signal processing algorithm as wavelet transform or empirical mode decomposition to separate the real-time global disturbance component reflecting the current systemic environmental fluctuations. The system control processor calculates the current energy value of this real-time global disturbance component and compares it with a preset reference energy threshold. This reference energy threshold can be a percentage of the global disturbance component energy at the time of the fault (e.g., 10%), or a statistical upper limit of the background noise energy during fault-free steady-state operation.

[0082] To ensure the reliability of the judgment, the system control processor requires that the energy of the real-time global interference component remains consistently below a baseline energy threshold for N consecutive sampling points (e.g., 10 consecutive sampling points, corresponding to 10 milliseconds). Simultaneously, the system control processor calculates the real-time fluctuation synchronization index between the faulty core and its neighboring cores at the current moment and determines whether this index is below a preset synchronization threshold. Only when both the energy and synchronization are consistently below the threshold does the system control processor ultimately determine that the external factor causing the fault has been eliminated.

[0083] In the high-density, highly dynamic computing environment of ARM servers, voltage and temperature inherently experience continuous and normal fluctuations. Relying solely on single-point energy thresholds can easily lead to misjudgments due to transient noise or normal load switching, prematurely restoring a core still in an unstable environment and triggering secondary failures. This embodiment filters out transient spikes by using N consecutive sampling points, requiring environmental fluctuations to remain consistently stable within a safe range before being considered truly subsided. Furthermore, once environmental interference truly disappears, the subtle fluctuations of each core will be dominated by its respective local load, naturally resulting in a significant decrease in synchronization. This avoids both false and ineffective restorations.

[0084] In one feasible implementation, the system control processor acquires the fault analysis results of the faulty core and identifies the fault type identifier corresponding to the fault analysis results; queries the historical offline records of the faulty core and obtains the number of historical offline events with the same fault type identifier as the base number of the same type; calculates the weighted offline count value of the current fault based on the base number of the same type and a preset time decay function. The weighted offline count value reflects the equivalent impact of the current fault after considering time decay. The time decay function makes the contribution of historical offline events further away from the current time smaller to the weighted offline count value; the weighted offline count value is accumulated into the offline count record of the same type of the faulty core as the updated offline count, and compared with the preset count threshold.

[0085] In the long-term operation of an ARM server, a core may fail multiple times at different points in time, but these failures differ in nature and timing. Simply accumulating counts ignores the failure type, leading to a core that previously experienced an occasional external failure being incorrectly flagged as exceeding a threshold and permanently offline much later due to a minor environmental fluctuation. This embodiment distinguishes between failure types, ensuring that only failures of the same type are accumulated and compared, avoiding interference from different types of failures. More importantly, the time decay function gradually reduces the impact of historical accumulation on current decisions, better aligning with the physical laws of hardware operation.

[0086] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the fault kernel processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0087] This application also provides an ARM server, which includes multiple processor cores, an operating system, a system control processor, and trusted firmware. Each processor core has an independent external power supply, and the external power supply of each processor core is controlled by multiple corresponding power policy units. The operating system is used to send a core offline command to the trusted firmware when it receives a fault event of an uncorrectable non-fatal error. The trusted firmware is used to send the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor. The system control processor is used to send a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, so as to realize the power-off control or hot reset of the faulty core.

[0088] The ARM server provided in this application, employing the fault kernel handling method in the above embodiments, can solve the technical problem that fault kernel handling methods may cause system stability risks. Compared with the prior art, the beneficial effects of the ARM server provided in this application are the same as those of the fault kernel handling method provided in the above embodiments, and other technical features of the ARM server are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0089] This application provides a fault kernel processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fault kernel processing method in Embodiment 1 above.

[0090] The following is for reference. Figure 3The diagram illustrates a structural schematic suitable for implementing the fault core processing device in the embodiments of this application. The fault core processing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 3 The faulty core processing device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0091] like Figure 3 As shown, the fault core processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the fault core processing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the fault core processing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a fault core processing device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0092] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0093] The fault kernel processing device provided in this application, employing the fault kernel processing method in the above embodiments, can solve the technical problem that fault kernel processing methods may cause system stability risks. Compared with the prior art, the beneficial effects of the fault kernel processing device provided in this application are the same as those of the fault kernel processing method provided in the above embodiments, and other technical features in this fault kernel processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0094] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0095] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0096] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the fault kernel processing method in the above embodiments.

[0097] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0098] The aforementioned computer-readable storage medium may be included in the fault core processing device; or it may exist independently and not assembled into the fault core processing device.

[0099] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the fault kernel processing device, cause the fault kernel processing device to perform the aforementioned fault kernel processing method.

[0100] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0103] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described fault kernel processing method, thereby solving the technical problem that fault kernel processing methods may cause system stability risks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the fault kernel processing method provided in the above embodiments, and will not be repeated here.

[0104] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the fault kernel processing method described above.

[0105] The computer program product provided in this application can solve the technical problem that fault kernel processing methods can cause system stability risks. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the fault kernel processing method provided in the above embodiments, and will not be repeated here.

[0106] The above descriptions are merely some embodiments of this application and do not limit the scope of protection of this application. Any equivalent structural transformations made based on the technical concept of this application and the content of this specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and policies of the country where the application is located and with authorization from the owner of the corresponding device.

Claims

1. A method for handling faulty kernels, characterized in that, This method is applied to ARM servers, which include multiple processor cores, an operating system, a system control processor, and trusted firmware. Each processor core has an independent external power supply, which is controlled by multiple corresponding power policy units. The fault core handling method includes: When the operating system receives a fault event that is an uncorrectable non-fatal error, the operating system sends a core offline command to the trusted firmware; The trusted firmware sends the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor; The system control processor sends a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, so as to realize the power-off control or thermal reset of the faulty core; Before the step of the operating system sending a core offline command to the trusted firmware when the operating system receives a fault event of an uncorrectable non-fatal error, the method further includes: The operating system acquires the temporal electrical and thermal characteristics of the faulty core within a preset time window, and simultaneously collects the associated electrical and thermal characteristics of at least one physically adjacent processor core. The operating system analyzes the time-series electrical and thermal characteristics and related electrical and thermal characteristics through a preset fault judgment model to obtain fault analysis results. The fault analysis results include physical faults and external faults. The preset fault judgment model is obtained by iterative training based on historical fault data. The training process of the preset fault judgment model also includes: Data preprocessing is performed on the original dataset, including extracting multidimensional features from the time-series electrothermal features and the associated electrothermal features. The original dataset is composed of the historical fault data. The multidimensional features include statistical features, frequency domain features, correlation features, and fluctuation features. The correlation features include the Pearson correlation coefficient between the voltage sequences of the fault core and the neighboring cores, the similarity of the temperature sequences, and the peak value of the cross-correlation function. The multidimensional features are combined into a feature vector, and then combined with the corresponding fault category label to form a training sample, thereby constructing and training the preset fault judgment model. After the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the system further includes: When the fault analysis result indicates an external fault and the external cause is detected to be eliminated, the system controls the processor to power on or restart the faulty core. The steps by which the operating system analyzes the time-series electrical-thermal characteristics and associated electrical-thermal characteristics using a preset fault diagnosis model to obtain fault analysis results include: The operating system, based on the time-series electrical and thermal characteristics and the associated electrical and thermal characteristics, separates the global interference component reflecting systemic environmental fluctuations and the local fault component reflecting processor core abnormalities through the preset fault judgment model. The separation is accomplished by signal decomposition techniques, including empirical mode decomposition or wavelet transform. If the total proportion of the global interference component in the time-series electrical-thermal features and the associated electrical-thermal features exceeds a preset threshold, and there is a synchronous fluctuation feature between the faulty core and the neighboring processor core, then the fault analysis result is determined to be an external fault.

2. The fault kernel handling method as described in claim 1, characterized in that, After the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the system further includes: The operating system marks the faulty core as offline; The system control processor updates the number of times the faulty core goes offline and determines whether the number of times the faulty core goes offline after the update exceeds a preset threshold. If it does not exceed the threshold, the faulty core is powered on or restarted after a preset cooling period. The trusted firmware notifies the operating system that the faulty core has been recovered via a software delegated exception interface. The operating system will reintegrate the recovered processor core into the scheduling resource pool.

3. The fault kernel handling method as described in claim 2, characterized in that, Before the step of the system control processor sending a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, the method further includes: The operating system is prompted to save the cached data of the faulty core to prevent the cached data from being cleared when the power policy unit controls the power outage of the faulty core; After the operating system marks the faulty core as offline, the method further includes: The system control processor records the fault information and number of times the fault event occurred, so that administrators can perform fault analysis.

4. The fault kernel handling method as described in claim 2, characterized in that, Before the step of the operating system marking the faulty core as offline, the method further includes: The operating system responds to the count threshold configuration operation to obtain the preset count threshold; The preset number of times threshold is loaded into the internal register of the system control processor.

5. An ARM server, characterized in that, The ARM server includes multiple processor cores, an operating system, a system control processor, and trusted firmware. Each processor core has an independent external power supply, and the external power supply of each processor core is controlled by multiple corresponding power policy units. The operating system is used to send a core offline command to the trusted firmware when it receives a fault event of an uncorrectable non-fatal error. The trusted firmware is used to send the core offline command to all processor cores, so that the processor cores that can respond normally can send the core offline command to the system control processor. The system control processor is used to send a shutdown signal to the power strategy unit corresponding to the faulty core based on the core shutdown command, so as to realize the power-off control or hot reset of the faulty core. The operating system is further configured to acquire the time-series electrical and thermal characteristics of the faulty core within a preset time window, and simultaneously collect the associated electrical and thermal characteristics of at least one physically adjacent neighboring processor cores. The operating system analyzes the time-series electrical and thermal characteristics and associated electrical and thermal characteristics using a preset fault judgment model to obtain fault analysis results. These results include physical faults and external faults. The preset fault judgment model is obtained through iterative training based on historical fault data. Data preprocessing is performed on the original dataset, including extracting multidimensional features from the time-series electrical and thermal characteristics and the associated electrical and thermal characteristics. The original dataset is composed of the historical fault data. The multidimensional features include statistical features, frequency domain features, correlation features, and fluctuation features. The correlation features include the relationship between the faulty core and neighboring nuclear power units. The Pearson correlation coefficient of the pressure sequence, the similarity of the temperature sequence, and the peak value of the cross-correlation function are used to construct and train the preset fault judgment model. Based on the time-series electrical-thermal features and the associated electrical-thermal features, the preset fault judgment model separates the global interference component reflecting systemic environmental fluctuations and the local fault component reflecting processor core anomalies. This separation is achieved through signal decomposition techniques, including empirical mode decomposition or wavelet transform. If the total proportion of the global interference component in the time-series electrical-thermal features and the associated electrical-thermal features exceeds a preset threshold, and there is synchronous fluctuation between the faulty core and the neighboring processor core, then the fault analysis result is determined to be an external fault. The system control processor is further configured to power on or restart the faulty core when the fault analysis result indicates an external fault and the external fault is detected to be eliminated.

6. A fault nuclear processing device, characterized in that, The fault kernel processing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the fault kernel processing method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the fault kernel processing method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the fault kernel processing method as described in any one of claims 1 to 4.

Images