Method for intelligent operating system reset of a supercomputing system node

Abstract

The application relates to the technical field of supercomputing system operation and maintenance, and discloses a method for intelligently resetting an operating system of a supercomputing system node, which comprises the following steps: step 101, collecting data of each computing node of the supercomputing system in real time, transmitting the data to a monitoring center through an RDMA high-speed network for storage, and forming historical operation data, wherein the historical operation data comprises historical index time sequence data and historical fault label data; step 102, generating a three-dimensional array in the historical operation data by using a sliding window method, wherein the three-dimensional array comprises sample information, time information and feature information, and the information in the three-dimensional array is sequentially spliced; and step 103, constructing a fault prediction model, inputting the three-dimensional array into the fault prediction model, and outputting a fault probability by the fault prediction model. By constructing the fault prediction model and collecting and processing the supercomputing system node data in real time, the fault occurrence probability can be predicted, and intelligent decision basis is provided for resetting the operating system of the supercomputing node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of supercomputing system operation and maintenance, and more particularly to a method for intelligent operating system reset of supercomputing system nodes. BACKGROUND

[0002] With the rapid development of frontiers such as artificial intelligence, climate simulation, and gene sequencing, supercomputers are advancing towards the trillion-parameter era of "million-node collaboration", and the scale of single-cluster nodes has broken through thousands or even tens of thousands of computing units.

[0003] However, as the core component of the supercomputing system, the operating system of the node is prone to kernel crashes, software dependency conflicts, and malicious program invasions during long-term high-load operation. According to industry practice, traditional node reset requires the node to be offline, obtain node information, use targeted recovery tools, and serial recovery, which cannot guarantee efficiency and speed. SUMMARY

[0004] The present application provides a method for intelligent operating system reset of supercomputing system nodes, which solves the problem that traditional node reset requires the node to be offline, obtain node information, use targeted recovery tools, and serial recovery, which cannot guarantee efficiency and speed.

[0005] To achieve the above purpose, the present application provides the following technical scheme: a method for intelligent operating system reset of supercomputing system nodes, comprising the following steps:

[0006] Step 101: Real-time collection of data of each computing node of the supercomputing system, the data being transmitted to the monitoring center through the RDMA high-speed network and stored as historical operation data, wherein the historical operation data includes historical index time series data and historical fault label data;

[0007] Step 102: Generation of a three-dimensional array in the historical operation data using the sliding window method, the three-dimensional array including sample information, time information, and feature information, and sequential splicing of the information in the three-dimensional array;

[0008] Step 103: Construction of a fault prediction model, input of the three-dimensional array into the fault prediction model, and output of the fault prediction model of the fault probability;

[0009] Step 104: Collection of real-time operation data of the supercomputing node, input of the real-time operation data of the supercomputing node into the fault prediction model after encoding, output of the predicted fault probability of the supercomputing node by the fault prediction model, and comparison of the predicted fault probability of the supercomputing node with the trigger threshold of the early warning and automatic reset process;

[0010] Step 105, when the prediction probability is greater than or equal to the early warning trigger threshold and less than the trigger threshold of automatically starting the reset process, early warning is performed, and when the prediction probability is greater than or equal to the trigger threshold of automatically starting the reset process, the operating system is reset.

[0011] Preferably, the historical index time series data includes CPU load of the hardware layer, memory ECC error, GPU video memory utilization, network throughput, PCIe bus bandwidth utilization, inode usage of the file system of the system layer, kernel error log, process abnormal exit times, disk I / O throughput, task progress of the job layer, checkpoint integrity, MPI communication delay, a total of 12 types of core indicators; the historical fault label data is the timestamp of the fault occurring in the historical running, the fault type and the indicator fluctuation record before and after the fault.

[0012] Preferably, the historical running data is preprocessed before the three-dimensional array is constructed, including the following steps:

[0013] Step one, for missing indicators caused by network transmission interruption and the like, short-term missing values within 10 seconds are completed by linear fitting based on valid data points before and after the missing values, and long-term missing segments exceeding 1 minute are marked as invalid sequences and removed;

[0014] Step two, based on criteria, extreme abnormal values in the indicators are identified and corrected;

[0015] Step three, the historical fault label data is aligned with the historical index time series data, the index sequence of the previous 120 minutes is intercepted as the fault association sequence with the fault occurrence time as the endpoint, and an equal length sequence is randomly intercepted from the fault-free period as a normal sample, to construct a training data set.

[0016] Preferably, when the three-dimensional array is constructed, the three-dimensional array generated by the preprocessed historical index data is , then:

[0017] ;

[0018] wherein:

[0019] the number of samples is the total number of valid index sequences intercepted by the sliding window, and the calculation formula is , wherein is the total sampling point number of the single-node historical index data, , is the total data duration, is the fixed sampling frequency; is the time step of a single sample; is the step length of the sliding window; For the floor function, ensure that each sample is a complete time series segment;

[0020] Time step : fixed as 600, corresponding to the 10-minute sliding window required by the fault prediction model, each time step corresponding to the collection value of 12 types of indicators within 1 second;

[0021] Feature number : fixed as 12, corresponding to 12 types of core indicators of hardware layer, system layer and job layer.

[0022] Preferably, the time series features in the three-dimensional array are fused, including the following steps:

[0023] Step one, calculate the time domain statistics of 12 types of indicators of each sample respectively:

[0024] Mean:

[0025] ;

[0026] Standard deviation:

[0027] ;

[0028] Peak value:

[0029] ;

[0030] Trend slope:

[0031] ;

[0032] Wherein, is the collection value of the nth sample, the ith time step and the fth type of indicator;

[0033] Step two, calculate the correlation degree of each indicator and the fault label by using mutual information entropy, and retain the top 50% of MI values;

[0034] Assign weights to different dimension indicators through attention mechanism, and the weight calculation formula is:

[0035] ;

[0036] Wherein: is proportional to is the attention weight of the nth sample, the ith time step and the fth type of indicator; is the mutual information entropy of the fth feature and the fault label, and the higher the value represents the stronger the correlation between the feature and the fault.

[0037] Preferably, the fault prediction model includes convolution layer, long short-term memory layer, attention layer and full connection layer;

[0038] The calculation formula of the convolution layer is:

[0039]

[0040] Wherein: is the output feature of the nth sample, the tth time step, and the kth convolution kernel; is the time window size of the convolution kernel; is the weight of the kth convolution kernel in the ith time offset and the fth index dimension; is the value of the nth sample, the t+i th time step, and the fth index; is the bias term of the kth convolution kernel;

[0041] The calculation formula of the long short-term memory layer is:

[0042]

[0043] Wherein: is the LSTM output time sequence feature of the nth sample, the tth time step, and the fth feature; is the hyperbolic tangent activation function, which maps the cell state value to [−1, 1], and the calculation formula is:

[0044]

[0045] The calculation formula of the attention layer is:

[0046]

[0047] Wherein: is the attention fusion feature vector of the nth sample;

[0048] The calculation formula of the fully connected layer is:

[0049]

[0050] Wherein: is the Sigmoid activation function; is the predicted failure probability of the nth sample; is the weight matrix of the fully connected layer; is the bias term of the fully connected layer.

[0051] Preferably, in step 104, a trigger threshold is set, when 0.85≤ failure probability <0.95, a warning is triggered, and when failure probability≥0.95, a reset process is automatically started.

[0052] Preferably, in step 105, the operating system reset includes the following steps:

[0053] ​​​​​Step one: verify node network connectivity, backup partition availability, power stability;

[0054] Step two: based on BaseImage, quickly restore the operating system, and perform incremental updates;

[0055] Step three: call the adaptation layer template, automatically install the driver, and configure the environment variable;

[0056] Step four: execute the basic test to verify the node state, and after passing, mark it as available and re-online, the basic test includes GPU computing power test, CPU load stress test, memory ECC error detection, network bandwidth test, file system read / write speed test, process startup or exit normality test.

[0057] The application also provides an intelligent operating system resetting system for supercomputing system nodes, comprising the following modules:

[0058] A data acquisition module (100) is used to acquire various core indicators of each computing node of the supercomputing system in real time as historical data;

[0059] A data processing module (200) is used to pre-process the collected historical data, generate a three-dimensional array of "sample-time-feature", and perform multi-dimensional fusion processing on the time sequence features in the three-dimensional array;

[0060] A fault prediction model construction and training module (300) is used to construct and train a fault prediction model, input the three-dimensional array subjected to multi-dimensional fusion processing into the fault prediction model, and map the fused features into a fault probability;

[0061] An execution module (400) is used to determine whether a threshold for triggering an early warning and automatically starting a resetting process is reached according to the prediction probability output by the fault prediction model, alarm if the early warning threshold is reached, and perform operating system resetting if the threshold for automatically starting the resetting process is reached.

[0062] The application also provides a storage medium storing non-transitory computer readable instructions, which can execute the steps of the above-mentioned intelligent operating system resetting method for supercomputing system nodes when executed by a computer.

[0063] The technical effects and advantages of the application are as follows:

[0064] The application can predict the fault occurrence probability by constructing a fault prediction model, collecting and processing supercomputing system node data in real time, and provide intelligent decision basis for supercomputing node operating system resetting.

[0065] The application can automate batch resetting, break through the bottleneck of large-scale operation and maintenance, realize efficiency leap, support parallel resetting of ten thousand nodes, and compress the time consumption of traditional manual operation from hours to minutes.

[0066] The application realizes zero manual intervention and avoids cluster-level fault diffusion caused by manual configuration errors.

[0067] When the operating system is reset, the application quickly restores the operating system based on BaseImage and performs incremental updating, calls the adaptation layer template to automatically install drivers and configure environment variables, and executes basic tests to verify node status, so that the system is more efficient and stable. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The application is a flowchart of the intelligent operating system resetting method of the supercomputing system node.

[0069] Figure 2 The application is a block diagram of the intelligent operating system resetting system of the supercomputing system node.

[0070] The reference signs are: 100, data acquisition module; 200, data processing module; 300, fault prediction model construction and training module; 400, execution module. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0072] Refer to the drawings in the specification Figure 1 An intelligent operating system resetting method of a supercomputing system node includes the following steps:

[0073] Step 101: Real-time data acquisition of each computing node of the supercomputing system, sampling frequency 1 second / time, data transmitted to the monitoring center through the RDMA high-speed network for storage, forming historical operation data.

[0074] It should be noted that the historical operation data is the monitoring and fault record in the long-term operation of the supercomputing node, which includes two types of data, historical index time series data and historical fault label data. The data of each computing node of the supercomputing system within 3 years or 5 years is collected.

[0075] The historical index time series data is continuously collected by a lightweight agent (<5 MB) deployed on each computing node, covering 12 core indicators of CPU load, memory ECC error, GPU memory utilization, network throughput, PCIe bus bandwidth utilization, file system inode usage, kernel error log, process abnormal exit times, disk I / O throughput, task progress, checkpoint integrity, and MPI communication delay at the system level, with a sampling frequency of 1 second / time, and transmitted to the monitoring center through the RDMA high-speed network for storage.

[0076] The historical fault label data is the timestamp of the kernel crash, software dependency conflict and other faults in the historical operation, the fault type and the indicator fluctuation record before and after the fault. The timestamp refers to the accurate time record of the supercomputing node failure; the fault type is the monitoring dimension corresponding to the historical index time series data, i.e., the 12 core indicators of the hardware layer, system layer, and job layer; the indicator fluctuation record before and after the fault refers to the dynamic change data of the 12 core indicators of the supercomputing node within a certain period of time before, at, and after the fault.

[0077] Step 102, processing historical operation data, including data preprocessing, constructing a three-dimensional array, and fusion processing of time series features in the three-dimensional array.

[0078] Step 1: Data preprocessing

[0079] For missing values caused by network transmission interruption, linear interpolation method is used to complete short-term missing values within 10 seconds based on valid data points before and after the missing values, avoiding the influence of a small amount of missing values on the continuity of time series data and the accuracy of subsequent feature extraction. Long-term missing segments exceeding 1 minute are marked as invalid sequences and removed, and linear interpolation method is not applicable.

[0080] Based on Extreme abnormal values in the indicators are identified and corrected according to the criteria, such as instantaneous CPU load surge caused by hardware failure, to avoid noise interference in model learning.

[0081] Align the historical fault label data with the historical index time series data, take the fault occurrence time as the endpoint, and extract the 120-minute indicator sequence before the fault as the fault correlation sequence. At the same time, randomly extract equal-length sequences from the fault-free period as normal samples to construct the training data set.

[0082] Step 2: Constructing a three-dimensional array

[0083] A three-dimensional array is generated in the historical operation data using the sliding window method. Let the three-dimensional array generated by the preprocessed historical operation data be Then:

[0084] ;

[0085] Wherein the definition of each parameter and calculation logic as follows: sample number : refers to the total number of effective index sequence intercepted by sliding window, the calculation formula is . In the formula, is the total sampling point number of single node historical index data, , is the total data duration, is the fixed sampling frequency; is the time step of a single sample; is the step length of sliding window (default 1 second, can be adjusted according to the accuracy requirement); is the floor function, which ensures that each sample is a complete time sequence segment.

[0086] Time step : fixed as 600, corresponding to the 10 minute sliding window required by the fault prediction model, derived from the sampling frequency , each time step corresponds to the collection value of 12 types of indicators within 1 second.

[0087] Feature number : fixed as 12, corresponding to the 12 types of core indicators collected by the agent program at the hardware layer, system layer and job layer.

[0088] For ease of understanding, the instantiation construction logic of three-dimensional array is illustrated by example: taking 100 minutes of historical index data of a supercomputing node as an example (1 second, second, ) with 1 second step length, ( ), intercepting 10 minute window ( ), the sample number , at this time the three-dimensional array is specifically represented as , wherein, represents the collection value of the 300th sample, the 150th time step (corresponding to second after the start of data), the 5th type of index (PCIe bus bandwidth utilization).

[0089] Step three: fusion processing of time series features in three-dimensional array, the purpose is to improve the model's ability to capture fault features, the core includes:

[0090] 1. Calculate the time domain statistics of 12 types of index sequences of each sample respectively, enhance the model's perception of index fluctuation rule, that is:

[0091] Mean:

[0092] ;

[0093] Standard deviation:

[0094] ;

[0095] Peak:

[0096] ;

[0097] Trend slope:

[0098] ;

[0099] wherein, is the acquisition value of the nth sample, the ith time step, and the fth index.

[0100] 2, Mutual information entropy (MI) is used to calculate the correlation degree of each index and the fault label, and the top 50% features of the MI value are retained; an attention mechanism (Attention) is used to assign weights to different dimensional indexes, such as assigning higher weights to strong fault correlation features such as kernel error logs and memory ECC errors, to realize adaptive fusion of multi-dimensional features, and the weight calculation formula is:

[0101] ;

[0102] wherein: is proportional to; is the attention weight of the nth sample, the ith time step, and the fth index; is the mutual information entropy of the fth index and the fault label (1 for fault, 0 for normal), and the higher the value, the stronger the correlation between the feature and the fault.

[0103] Step 103, constructing a fault prediction model, inputting a three-dimensional array into the fault prediction model, and the fault prediction model outputting a fault probability.

[0104] 1, Model architecture design

[0105] The fault prediction model includes a convolutional layer, a long short-term memory layer, an attention layer, and a fully connected layer.

[0106] Convolutional layer (CNN): extract spatial correlation features of 12 types of indexes in each time step through a 1D convolution kernel; through the sliding of the convolution kernel in the time dimension, the features of the indexes in each local time window are convolved, and the calculation formula is:

[0107] ;

[0108] wherein: is the output feature of the nth sample, the tth time step, and the kth convolution kernel; is the time window size of the convolution kernel; is the weight of the kth convolution kernel at the ith time offset and the fth index, and its value is obtained by adaptive adjustment during the model training process through the back propagation optimization of the cross-entropy loss function. The training process will continuously correct the value to improve the extraction accuracy of the fault feature and the prediction accuracy of the model. is the value of the fth index of the nth sample at the t+i time step. is the bias term of the kth convolution kernel.

[0109] Long Short-Term Memory (LSTM): captures the time sequence dependency of the feature sequence after convolution, and outputs the time sequence feature. The LSTM time sequence feature output formula is:

[0110] ;

[0111] wherein:

[0112] is the LSTM output time sequence feature of the fth index of the nth sample at the t time step, which contains time sequence dependency. is the hyperbolic tangent activation function, which maps the cell state value to [−1, 1], and the calculation formula is:

[0113] ;

[0114] Attention layer (Attention): focuses on the key time sequence segment before the fault occurs. The attention mechanism based on mutual information entropy (MI) is adopted to give higher weights to the features and time sequence segments with high correlation with the fault label. The fusion feature vector is output by weighting and summing the time sequence features output by the long short-term memory layer, and the calculation formula is:

[0115] ;

[0116] wherein:

[0117] is the attention fusion feature vector of the nth sample.

[0118] Fully connected layer: maps the fusion feature to the fault probability, and adopts the Sigmoid activation function. The output value range is [0, 1], which matches the physical meaning of the fault probability, and the activation formula is:

[0119] ;

[0120] wherein: is the Sigmoid activation function; is the predicted fault probability of the nth sample; is the weight matrix of the fully connected layer; is the bias term of the fully connected layer.

[0121] 2. Model training

[0122] A three-dimensional array is divided into a training set, a validation set, and a test set according to 7:2:1 with corresponding fault labels (fault samples are marked as 1, and normal samples are marked as 0).

[0123] The cross-entropy loss function is used to optimize the model parameters to achieve a fault prediction accuracy of > 92%. The cross-entropy loss function is:

[0124] ;

[0125] wherein, is the true label, is the predicted fault probability.

[0126] Training and early stopping: training on the AI chip, verifying the accuracy of the model on the validation set every round, and triggering early stopping when the accuracy does not improve for 5 consecutive rounds to avoid overfitting.

[0127] Step 104, collect supercomputing node real-time running data, encode the supercomputing node real-time running data, input the fault prediction model, and output the predicted supercomputing node fault probability , and compare the predicted supercomputing node fault probability with the trigger threshold value of the early warning and automatic start reset process to set the trigger threshold value. When the fault probability is > 0.85, trigger the early warning, and when it is > 0.95, automatically start the reset process.

[0128] Step 105, when 0.85 ≤ fault probability < 0.95, trigger the early warning, and when the fault probability ≥ 0.95, perform the operating system reset.

[0129] Operating system reset, including the following steps:

[0130] Step 1: Pre-check, verify node network connectivity, backup partition availability, and power stability. Node network connectivity verification: detect the communication link between the node and the monitoring center and the mirror server through the RDMA high-speed network to ensure that the data transmission channel is smooth; backup partition availability verification: check the storage space and read-write permissions of the local backup partition (used for temporarily storing snapshots or configuration files) of the node to avoid interruption of the reset due to storage abnormalities; power stability verification: collect voltage and current fluctuation data of the node power module to confirm stable power supply (fluctuation range ≤ ± 5%) to prevent unexpected power failure during the reset process.

[0131] Step two: system recovery, based on BaseImage (read-only image) to quickly restore the operating system, support incremental update (only synchronize difference file). Base image recovery: call the pre-stored BaseImage of the node corresponding role (login / computing / storage node), quickly cover the system partition through block device level copy technology, skip the redundant step of traditional file level copy; incremental update synchronization: only synchronize the difference file (such as customized configuration, security patch) between BaseImage and target system version, instead of full image transmission, further shorten the recovery time (single node base recovery <2 minutes).

[0132] Step three: environment configuration, call the adaptation layer template, automatically install the driver and configure the environment variable. Driver automatic installation: match the node hardware model from the "driver pool", install the corresponding kernel module and tool chain; Environment variable configuration: automatically write system environment variables (such as LD_LIBRARY_PATH pointing to the driver library path, PATH containing software execution directory), no need for manual configuration.

[0133] Step four: health check, execute basic test to verify node status, and mark as "available" and re-online after passing. Including GPU computing power test (running benchmark example), CPU load stress test, memory ECC error detection, network bandwidth test, file system read / write speed test, process start / exit normality test, etc.

[0134] Referring to the description attached Figure 2 The embodiment also provides an intelligent operating system reset system for supercomputing system nodes, comprising the following modules:

[0135] The data acquisition module 100 is used for collecting various core indicators of each computing node of the supercomputing system in real time as historical data.

[0136] The data processing module 200 is used for pre-processing the collected historical data, generating a three-dimensional array of "sample-time-feature", and performing multi-dimensional fusion processing on the time sequence features in the three-dimensional array.

[0137] The fault prediction model construction and training module 300 is used for constructing and training a fault prediction model, inputting the three-dimensional array subjected to multi-dimensional fusion processing into the fault prediction model, and mapping the fusion features into a fault probability.

[0138] The execution module 400 is used for judging whether a threshold value of triggering an early warning and automatically starting a reset process is triggered according to the prediction probability output by the fault prediction model, alarming if the early warning threshold value is triggered, and performing operating system reset if the threshold value of automatically starting the reset process is triggered.

[0139] The embodiment further provides a storage medium which stores non-transitory computer readable instructions capable of executing the steps in the method for resetting the intelligent operating system of a supercomputing system node as described above when the non-transitory computer readable instructions are executed by a computer.

[0140] Finally: the above only for the preferred embodiments of the present application, and not for limiting the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. A method for intelligently resetting the operating system of a supercomputing system node, characterized in that, The method comprises the following steps: Step 101, real-time collection of data of each computing node of the supercomputing system, the data being transmitted to a monitoring center through a high-speed RDMA network to form historical operation data, wherein the historical operation data comprises historical index time series data and historical fault label data; Step 102, generation of a three-dimensional array by using a sliding window method in the historical operation data, the three-dimensional array comprising sample information, time information and feature information, and sequential splicing of the information in the three-dimensional array; Step 103, construction of a fault prediction model, input of the three-dimensional array into the fault prediction model, and output of a fault probability by the fault prediction model; Step 104, collection of real-time operation data of the supercomputing node, input of the real-time operation data of the supercomputing node into the fault prediction model after coding, output of a predicted fault probability of the supercomputing node by the fault prediction model, and comparison of the predicted fault probability of the supercomputing node with a size of a triggering threshold of a pre-warning and an automatic starting reset process; Step 105, pre-warning when the predicted probability is greater than or equal to the triggering threshold of the pre-warning and less than the triggering threshold of the automatic starting reset process, and operating system reset when the predicted probability is greater than or equal to the triggering threshold of the automatic starting reset process; When constructing the three-dimensional array, let the three-dimensional array generated by the pretreated historical index data be Then: ； Wherein: the number of samples : refers to the total number of effective index sequences intercepted by the sliding window, and the calculation formula is , wherein, is the total sampling point number of single node historical index data, , is the total data duration, is the fixed sampling frequency; is the time step of a single sample; is the step length of the sliding window; is the floor function, which ensures that each sample is a complete time sequence segment; Time step : fixed as 600, corresponding to the 10-minute sliding window required by the failure prediction model, and each time step corresponds to the 12-class index collection value within 1 second; Feature number : fixed to 12, corresponding to 12 types of core indicators of hardware layer, system layer, and job layer Fusion processing of time series features in the three-dimensional array comprises the following steps: Step one, calculation of time domain statistics for 12 types of index sequences of each sample respectively: Mean: ； Standard deviation: ； Peak: ； Trend slope: ； wherein, is the collected value for the nth sample, the i-th time step, the f-th indicator. Step two, calculation of the correlation degree of each index and the fault label by using mutual information entropy, and reservation of features with the top 50% MI values; Weight calculation formula for different dimension indexes through an attention mechanism: ； wherein: is proportional to; is the attention weight of the nth sample, the ith time step, and the fth indicator; is the mutual information entropy of the fth feature and the fault label, and the higher the value, the stronger the correlation between the feature and the fault.

2. The method for intelligent operating system reset of a supercomputing system node according to claim 1, characterized in that: The historical index time series data comprises 12 types of core indexes, such as CPU load of a hardware layer, memory ECC error, GPU memory utilization, network throughput, PCIe bus bandwidth utilization, inode usage rate of a file system of a system layer, kernel error log, process abnormal exit number, disk I / O throughput, task progress of a job layer, checkpoint integrity, and MPI communication delay; The historical fault label data is the timestamp of a node fault in historical operation, fault type, and index fluctuation record before and after the fault.

3. The method for intelligent operating system reset of a supercomputing system node according to claim 2, wherein: Preprocessing of the historical operation data before construction of the three-dimensional array comprises the following steps: Step one, linear fitting for short-term missing values within 10 seconds based on valid data points before and after the missing values, and marking and removing long-term missing segments exceeding 1 minute as invalid sequences due to missing values in network transmission; Step two, based on Criteria identify and correct extreme outliers in the indicators; Step three, alignment of the historical fault label data and the historical index time series data, truncation of index sequences of 120 minutes before the fault as fault-associated sequences, and random truncation of equal-length sequences from a fault-free period as normal samples to construct a training data set.

4. The method for intelligent operating system reset of a supercomputing system node according to claim 3, wherein, The fault prediction model comprises a convolution layer, a long short-term memory layer, an attention layer and a fully connected layer; The calculation formula of the convolution layer is: ； wherein: is the output feature of the nth sample, the tth time step, the kth convolution kernel; is the time window size of the convolution kernel; is the weight of the kth convolution kernel in the i th time offset, the fth index dimension; is the value of the nth sample, the t+i th time step, the fth index; is the bias term of the kth convolution kernel; The calculation formula of the long short-term memory layer is: ； where: is the LSTM output time series feature for the nth sample, tth time step, fth feature; is the hyperbolic tangent activation function that maps cell state values to [−1, 1] with the formula: ； The calculation formula of the attention layer is: ； wherein: is the attention fusion feature vector for the nth sample; The calculation formula of the fully connected layer is: ； wherein: is a Sigmoid activation function; is the predicted failure probability for the nth sample; is a fully connected layer weight matrix; is a fully connected layer bias term.

5. The method for intelligent operating system reset of a supercomputing system node according to claim 4, wherein: In step 104, a trigger threshold is set, when 0.85 ≤ failure probability < 0.95, a pre-warning is triggered, and when failure probability ≥ 0.95, an automatic reset process is started.

6. The method for intelligent operating system reset of a supercomputing system node according to claim 5, wherein: In step 105, the operating system reset includes the following steps: Step one: verify node network connectivity, backup partition availability, and power stability; Step two: restore the operating system based on BaseImage, and perform incremental updates; Step three: call the adaptation layer template to automatically install drivers and configure environment variables; Step four: perform basic tests to verify node status, and mark it as available and re-online after passing, the basic tests include GPU computing power test, CPU load stress test, memory ECC error detection, network bandwidth test, file system read-write speed test, and process startup or exit normality test.

7. An intelligent operating system reset system of a supercomputing system node, configured to perform the method of intelligent operating system reset of a supercomputing system node according to any one of claims 1-6. The following modules are included: A data collection module (100) for real-time collection of various core indicators of each computing node of the supercomputing system as historical data; A data processing module (200) for preprocessing the collected historical data, generating a three-dimensional array of "sample-time-feature", and performing multi-dimensional fusion processing on the time sequence features in the three-dimensional array; A fault prediction model construction and training module (300) for constructing and training a fault prediction model, inputting the multi-dimensionally fused three-dimensional array into the fault prediction model, and mapping the fused features into a failure probability; An execution module (400) for determining whether to trigger a pre-warning and an automatic reset process threshold according to the prediction probability output by the fault prediction model, alarming if the pre-warning threshold is triggered, and performing an operating system reset if the automatic reset process threshold is triggered.

8. A storage medium storing non-transitory computer-readable instructions that, when executed by a computer, can perform the steps of a method for intelligent operating system reset of a supercomputing system node as claimed in claim 6.

Images