Server memory failure ai prediction and dynamic repair protection system

Abstract

The application relates to the technical field of server fault protection, in particular to an AI prediction and dynamic repair protection system for server memory faults, which comprises a data acquisition module, a feature extraction module, a state modeling module, a risk prediction module and a repair decision module. The data acquisition module continuously acquires memory running original data streams from a server cluster; the feature extraction module extracts multi-dimensional abnormal fluctuation features from the original data streams and integrates the multi-dimensional abnormal fluctuation features into a multi-dimensional abnormal feature vector; the state modeling module constructs a time-varying memory health state field with a timestamp, a server node number and a feature component value as axes by using the multi-dimensional abnormal feature vector; the risk prediction module applies a space-time convolution network to identify a fault propagation mode and outputs a risk field; and the repair decision module generates an active repair decision based on the risk field. The system can realize accurate prediction and active protection of memory faults and guarantee stable operation of the server cluster.

Description

Technical Field

[0001] This invention relates to the field of server fault protection technology, and in particular to a server memory fault AI prediction and dynamic repair protection system. Background Technology

[0002] As the core carrier of data processing and storage, the memory stability of server clusters directly determines the normal operation of the entire system. Memory failures can easily lead to data loss and service interruptions. Therefore, the prediction and protection of server memory failures has become a key research direction in related technical fields. In existing technologies, server memory failure detection mostly relies on fixed threshold triggering mechanisms. By collecting single or a few indicators such as memory usage and read / write speed, combined with simple time-series analysis methods, it is determined whether memory failure has occurred. Some technologies use traditional machine learning models to predict the memory status of single nodes, achieving preliminary early warning of failures.

[0003] Existing technical solutions have shortcomings: their representation of memory operating status is relatively simple, mostly using discrete time-series features or single-node independent state descriptions, which cannot reflect the correlation between memory states of different server nodes, nor can they capture the dynamic pattern of memory state changes over time, resulting in a lag in the identification of memory anomalies; existing prediction methods can only determine whether a single node has a fault risk or predict the probability of a fault occurring, but cannot accurately identify the fault propagation mode, and it is difficult to locate potential fault source nodes and fault propagation paths. They can only perform passive repairs after a fault occurs, and cannot achieve proactive protection.

[0004] How to construct a representation method that can comprehensively reflect the spatiotemporal correlation of the memory status of server clusters, achieve accurate identification of memory fault propagation patterns, and thus locate potential fault sources and propagation paths, breaking through the limitations of passive repair, has become a key issue that needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an AI-based server memory fault prediction and dynamic repair protection system.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a server memory fault AI prediction and dynamic repair protection system, comprising:

[0007] The data acquisition module continuously collects raw data streams corresponding to the memory running status from the deployed server cluster;

[0008] The feature extraction module performs multi-dimensional abnormal fluctuation feature extraction on the original data stream, detects the abnormal fluctuation features of each time-series data point in the original data stream based on an adaptive sliding window, and integrates the extracted multiple abnormal fluctuation features into a multi-dimensional abnormal feature vector.

[0009] The state modeling module uses the multidimensional anomaly feature vector to construct a time-varying memory health state field. The memory health state field uses the timestamp as the time axis, the server node number as the spatial axis, and the component values ​​of the multidimensional anomaly feature vector as the field strength dimension.

[0010] The risk prediction module uses a spatiotemporal convolutional network to identify fault propagation patterns in the memory health state field. The spatiotemporal convolutional network captures the correlation of the memory health state field in the spatial dimension and the evolution of the time sequence dimension, and outputs a risk field that represents potential fault source nodes and fault propagation paths.

[0011] The repair decision module generates proactive repair decisions based on the risk field. These proactive repair decisions include decisions on isolating faulty nodes, migrating loads, and switching backup memory regions.

[0012] As a further aspect of the present invention, multi-dimensional abnormal fluctuation feature extraction is performed on the original data stream, including:

[0013] The raw data stream includes the memory module's bit error rate timing, operating temperature timing, power supply voltage timing, and access latency timing;

[0014] The abnormal fluctuation characteristics include sudden increase, decay gradient, and oscillation frequency;

[0015] The statistical characteristics of the data within the window are calculated by applying an adaptive length sliding window to the bit error rate timing, operating temperature timing, power supply voltage timing, and access delay timing, respectively. The statistical characteristics include mean offset and standard deviation rate of change.

[0016] The difference sequence between adjacent data points is calculated within a sliding window, and segments that rise or fall continuously are identified from the difference sequence. The cumulative amplitude of the segments is extracted as the burst value.

[0017] Perform a second difference operation on the difference sequence and calculate the average absolute value of the second difference sequence as the decay gradient characterizing the intensity of fluctuation;

[0018] Perform a Fast Fourier Transform on the original data stream to extract the main frequency components in the spectrum that exceed the background noise threshold, and calculate the average frequency of the main frequency components as the oscillation frequency.

[0019] As a further aspect of the present invention, a time-varying memory health state field is constructed using the multidimensional anomaly feature vector, including:

[0020] Establish a three-dimensional spatial structure, where the first dimension is the timestamp sequence, the second dimension is the physical number sequence of all memory modules in the server cluster, and the third dimension is the components of the multidimensional anomaly feature vector.

[0021] Assign the multidimensional anomaly feature vector corresponding to each timestamp and each physical number to the corresponding coordinate point of the three-dimensional spatial structure;

[0022] For the missing coordinate points between adjacent timestamps and adjacent physical numbers in the three-dimensional spatial structure, the spatiotemporal kriging interpolation algorithm is used to interpolate and fill the gaps based on the multidimensional abnormal feature vector values ​​of the surrounding known points, forming a continuous and smooth memory health state field.

[0023] As a further aspect of the present invention, a spatiotemporal convolutional network is applied to identify fault propagation patterns in the memory health state field, including:

[0024] The memory health state field is used as input and fed into a pre-trained spatiotemporal convolutional network, which is composed of alternating stacks of causal convolutional layers in the time dimension and graph convolutional layers in the spatial dimension.

[0025] The temporal causal convolutional layer of the spatiotemporal convolutional network is used to extract the evolution features of the memory health state field along the time axis;

[0026] The spatial dimension graph convolutional layer of the spatiotemporal convolutional network constructs an adjacency matrix based on the physical connection topology of the server cluster, which is used to extract the correlation features between spatial nodes in the memory health state field;

[0027] The final output layer of the spatiotemporal convolutional network fuses temporal evolution features with spatial correlation features and maps them into a risk field with the same dimension as the input memory health state field, where the value of each coordinate point represents the fault risk intensity of the time and physical number node.

[0028] As a further aspect of the present invention, generating proactive repair decisions based on the risk field includes:

[0029] In the risk field, coordinate points whose fault risk intensity exceeds a preset risk threshold are identified as potential fault source nodes;

[0030] Taking each potential fault source node as the center, find the adjacent nodes in the risk field that are connected to it and whose risk intensity gradient is decreasing in the spatial dimension, and connect the adjacent nodes to form a fault propagation path.

[0031] For each identified potential fault source node, generate an isolation decision including its physical number;

[0032] For all nodes along the fault propagation path, generate migration decisions to move the memory load running on them to healthy nodes;

[0033] For isolated nodes or nodes whose load is moved out, a switching decision is generated to switch their memory address space to a preset physical spare memory region;

[0034] The isolation, migration, and switching decisions for all nodes are integrated into proactive repair decisions.

[0035] As a further aspect of the present invention, it also includes:

[0036] The repair control module encodes proactive repair decisions into a repair control instruction sequence and injects the repair control instruction sequence into the baseboard management controller of the server cluster, triggering a dynamic memory reconfiguration operation.

[0037] The online evaluation module collects real-time performance degradation curves while the dynamic reconfiguration operation is being executed. These performance degradation curves reflect changes in memory access latency and throughput during the repair process. The performance degradation curves are then input into an online reinforcement learning evaluator, which evaluates the effectiveness of the executed repair control instruction sequence and outputs an evaluation reward signal. Based on the evaluation reward signal, the generation strategy for proactive repair decisions is dynamically adjusted to optimize subsequent fault handling repair decisions.

[0038] The closed-loop iteration module, after completing the dynamic reconfiguration of memory, obtains the updated memory health indicators from the baseboard management controller and feeds the updated memory health indicators back to the calculation process of the memory health status field, forming a closed-loop fault prediction and repair iteration.

[0039] As a further aspect of the present invention, the step of injecting the repair control command sequence into the baseboard management controller of the server cluster specifically includes:

[0040] The repair control command sequence is encoded into a series of executable remote command messages according to the command interface specification of the baseboard management controller;

[0041] Remote command messages are sent sequentially to the baseboard management controller of the target server via the out-of-band management network;

[0042] The baseboard management controller receives and parses remote command messages, and executes low-level operations such as memory page detachment, memory address remapping, and memory channel disabling according to the instructions in the messages.

[0043] As a further aspect of the present invention, the acquisition of the real-time feedback performance degradation curve specifically includes:

[0044] During the dynamic reconfiguration operation performed by the baseboard management controller, the performance metrics of the memory subsystem are sampled at a high frequency by the operating system kernel module.

[0045] The collected performance metrics include real-time values ​​of memory access latency and real-time values ​​of memory throughput.

[0046] With sampling time on the horizontal axis and the real-time values ​​of memory access latency and memory throughput on the vertical axes respectively, two time series curves are plotted to form a performance degradation curve.

[0047] The performance degradation curve is input into an online reinforcement learning evaluator, specifically including:

[0048] Key evaluation features are extracted from the performance degradation curve. These key evaluation features include the time from the start of the repair operation to the peak memory access latency, the magnitude of the peak memory access latency, and the time required for memory throughput to recover to a steady-state level.

[0049] Key evaluation features are input into the policy evaluation network of the reinforcement learning evaluator, and the policy evaluation network outputs a scalar value as an immediate reward.

[0050] Based on the preset long-term stable operation reward, a comprehensive evaluation reward signal for the execution of this repair control command sequence is calculated.

[0051] As a further aspect of the present invention, the generation strategy for proactive repair decisions is dynamically adjusted based on the evaluation reward signal, including:

[0052] The policy gradient of the current proactive repair decision generation strategy is calculated using the evaluation reward signal;

[0053] The model parameters of the decision model used to generate proactive repair decisions are updated using the policy gradient ascent algorithm.

[0054] After the model parameters of the decision-making model are updated, when generating proactive repair decisions for similar risk fields, the model will tend to produce repair decisions that can obtain higher evaluation reward signals.

[0055] As a further aspect of the present invention, the step of obtaining updated memory health indicators from the baseboard management controller after completing the dynamic reconfiguration of memory includes:

[0056] After the baseboard management controller completes all repair control instructions, a health status query request is sent to the baseboard management controller.

[0057] The system receives real-time readings of the current memory module's bit error rate, operating temperature, power supply voltage, and access latency from the baseboard management controller, which are then used as updated memory health indicators.

[0058] The updated memory health metrics are bound to the corresponding timestamps and server physical numbers and appended to the original data stream for use in the next round of memory health status field construction and risk prediction.

[0059] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0060] Using timestamps as the time axis, server node numbers as the spatial axis, and the component values ​​of the multidimensional anomaly feature vector as the field strength dimension, the extracted multidimensional anomaly fluctuation features are integrated into a multidimensional anomaly feature vector. This constructs a spatiotemporally integrated time-varying memory health state field, breaking the limitations of conventional discrete time-series features and single-node independent state representation. It can comprehensively and dynamically present the spatiotemporal distribution characteristics of the memory operation status of the entire server cluster, clearly reflect the correlation and time-series evolution trend of the memory status of each server node, avoid the one-sidedness of single-node analysis, and make the representation of memory status more complete and dynamic. It can more accurately capture subtle anomalies in memory operation and provide more comprehensive state support for fault prediction.

[0061] This invention applies spatiotemporal convolutional networks to identify fault propagation patterns in a time-varying memory health state field. By capturing the correlation between server nodes in the spatial dimension and the state evolution patterns in the temporal dimension of the memory health state field, it outputs a risk field containing potential fault source nodes and fault propagation paths. Unlike conventional methods that only predict the probability or level of single-node faults, this invention can accurately identify the propagation patterns of faults in the server cluster, accurately locate potential fault source nodes and fault propagation paths, and provide a clear target direction for proactive repair. It breaks through the limitations of conventional technologies that passively respond to faults, and can take targeted measures before the fault spreads, reducing the impact of faults on the overall operation of the server cluster and reducing the risk of service interruption and data loss. Attached Figure Description

[0062] Figure 1 This is a timing diagram of the server memory fault AI prediction and dynamic repair protection system described in this invention;

[0063] Figure 2 A flowchart for extracting multi-dimensional abnormal fluctuation features;

[0064] Figure 3 A flowchart for identifying fault propagation patterns in spatiotemporal convolutional networks;

[0065] Figure 4 This is a performance degradation curve for the memory repair process;

[0066] Figure 5 This is a multi-dimensional timing monitoring curve for server memory. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0069] See Figure 1 The data acquisition module continuously collects raw data streams corresponding to the memory operating status from the deployed server cluster. The feature extraction module, connected to the data acquisition module, performs multi-dimensional abnormal fluctuation feature extraction on the acquired raw data stream. This extraction process is based on an adaptive sliding window to detect abnormal fluctuation features at each time-series data point in the raw data stream, and then integrates the extracted multiple abnormal fluctuation features into a multi-dimensional abnormal feature vector. The state modeling module receives the multi-dimensional abnormal feature vector output by the feature extraction module and uses these vectors to construct a time-varying memory health state field. This memory health state field uses timestamps as the time axis, server node numbers as the spatial axis, and the component values ​​of the multi-dimensional abnormal feature vectors as the field strength dimension. The risk prediction module, connected to the state modeling module, applies a spatiotemporal convolutional network to the constructed memory health state field to identify fault propagation patterns. This spatiotemporal convolutional network captures the spatial correlation and temporal evolution of the memory health state field, ultimately outputting a risk field characterizing potential fault source nodes and fault propagation paths. The repair decision module generates proactive repair decisions based on the risk field generated by the risk prediction module. These proactive repair decisions include decisions on isolating faulty nodes, migrating loads, and switching to backup memory areas.

[0070] See Figure 2In one embodiment of the present invention, the original data stream includes the bit error rate timing, operating temperature timing, power supply voltage timing, and access delay timing of the memory module. The abnormal fluctuation features to be extracted include sudden increase, decay gradient, and oscillation frequency. The feature extraction process applies adaptive-length sliding windows to the bit error rate timing, operating temperature timing, power supply voltage timing, and access delay timing, respectively, and calculates the statistical characteristics of the data within the window. The calculated statistical characteristics include the mean offset and the rate of change of standard deviation. Within the sliding window, the difference sequence between adjacent data points is calculated, and continuously rising or falling segments are identified from the difference sequence. The cumulative amplitude of these segments is extracted as the sudden increase. A second difference operation is performed on the difference sequence, and the average absolute value of the second difference sequence is calculated, which is used as the decay gradient characterizing the severity of the fluctuation. A fast Fourier transform is performed on the original data stream to extract the main frequency components exceeding the background noise threshold in the spectrum, and the average frequency of these main frequency components is calculated as the oscillation frequency. After feature extraction, a time-varying memory health state field is constructed using the obtained multidimensional abnormal feature vector. The construction process first establishes a three-dimensional spatial structure. The first dimension of this structure is a timestamp sequence, the second dimension is a sequence of physical IDs for all memory modules in the server cluster, and the third dimension is the components of a multidimensional anomaly feature vector. The multidimensional anomaly feature vector corresponding to each timestamp and each physical ID is assigned to the corresponding coordinate point in this three-dimensional spatial structure. For the missing coordinate points between adjacent timestamps and adjacent physical IDs in the three-dimensional spatial structure, a spatiotemporal kriging interpolation algorithm is used to fill the gaps based on the multidimensional anomaly feature vector values ​​of surrounding known points, thereby forming a continuous and smooth memory health state field.

[0071] In practical implementation, based on the raw data stream collected from the server cluster, the raw data stream includes the bit error rate timing, operating temperature timing, power supply voltage timing, and access latency timing of the memory module. In some embodiments, the bit error rate timing records the number of memory check errors per unit time, the operating temperature timing reflects the temperature sensor readings near the memory module's heatsink, the power supply voltage timing records the voltage fluctuations on the memory module's power supply pins, and the access latency timing characterizes the average latency of the memory controller accessing a specified address. The feature extraction module needs to extract multi-dimensional abnormal fluctuation features from these four types of raw timing data. The abnormal fluctuation features to be extracted include sudden increases, decay gradients, and oscillation frequencies.

[0072] In some embodiments, an adaptive-length sliding window is applied to the bit error rate (BER) time series data. The length of the sliding window is dynamically adjusted based on the local stability of the BER time series data. Within the sliding window, statistical characteristics of the data are calculated, including mean offset and standard deviation rate of change. The mean offset is the difference between the arithmetic mean of the data within the current window and the mean of a historical baseline. The standard deviation rate of change is the ratio of the standard deviation of the data within the current window to the standard deviation of the historical baseline. A difference sequence between adjacent data points is calculated within the sliding window, consisting of the value of the subsequent data point minus the value of the preceding data point. Continuous upward segments with continuously positive values ​​or continuous downward segments with continuously negative values ​​are identified from the difference sequence, and the cumulative amplitude of these continuous segments is extracted as the burst value. In a specific calculation example, the burst value is calculated by summing the absolute values ​​of all difference sequence elements within the selected segment, as expressed by:

[0073]

[0074] in: Indicates a sudden increase in value. In the difference sequence, the first... The value of each element, and These represent the start and end indices of a continuous rising or falling segment, respectively. A second difference operation is performed on the difference sequence of the bit error rate time series. The second difference sequence is the difference between adjacent elements of the difference sequence. The average absolute value of the second difference sequence is calculated, and the result is used as the decay gradient characterizing the severity of bit error rate fluctuations. A Fast Fourier Transform is performed on the original bit error rate time series data stream. From the transformed spectrum, the main frequency components with power exceeding the background noise threshold are extracted. The background noise threshold is determined by analyzing the spectrum of historical stable operating phases. The average frequency of the extracted main frequency components is calculated, and this average frequency is used as the oscillation frequency of the bit error rate time series.

[0075] It is understandable that the same processing flow as the bit error rate timing procedure is used to calculate the surge values, decay gradients, and oscillation frequencies of the operating temperature timing, supply voltage timing, and access delay timing. In specific implementations, for the operating temperature timing, the sliding window length is set larger to account for the relatively slow temperature changes. Its mean offset, standard deviation rate of change, differential sequence, and quadratic differential sequence are calculated, and the surge values, decay gradients, and oscillation frequencies of the operating temperature are finally output. For the supply voltage timing, its baseline fluctuations may be more frequent, so the sliding window length is shortened accordingly to capture finer voltage dips or spikes, and the surge values, decay gradients, and oscillation frequencies of the supply voltage are calculated. For the access delay timing, its mean offset, standard deviation rate of change, differential sequence, and quadratic differential sequence are calculated, and the surge values, decay gradients, and oscillation frequencies of the access delay are also calculated. Finally, for a memory module at a given point in time, the feature extraction module outputs a multidimensional anomaly feature vector containing 12 components. This vector is composed of 12 scalar values, namely, the burst increment, decay gradient, and oscillation frequency, corresponding to the four types of time-series data: bit error rate, operating temperature, power supply voltage, and access latency, respectively, connected in sequence.

[0076] In practical implementation, the state modeling module utilizes the aforementioned multidimensional anomaly feature vectors to construct a time-varying memory health state field. A three-dimensional spatial structure is established. The first dimension of the three-dimensional spatial structure is a continuous sequence of timestamps, with the interval of the timestamp sequence consistent with the data acquisition frequency. The second dimension of the three-dimensional spatial structure is a sequence of physical numbers of all memory modules in the server cluster, with the physical numbers globally uniquely encoded according to the hierarchy of server rack, server node, memory channel, and memory slot. The third dimension of the three-dimensional spatial structure consists of the components of the multidimensional anomaly feature vector, namely 12 feature scalars. The 12-dimensional anomaly feature vector corresponding to each acquisition timestamp and each memory module physical number is assigned to the corresponding coordinate point in the three-dimensional spatial structure. Each coordinate point is a tuple containing a timestamp, a physical number, and 12 feature values. Since data acquisition is not synchronized across all nodes at all times, there will be a large number of missing coordinate points in the three-dimensional spatial structure. The missing coordinate points between adjacent timestamps and adjacent physical numbers in the three-dimensional spatial structure are filled using a spatiotemporal kriging interpolation algorithm. The spatiotemporal kriging interpolation algorithm calculates an optimal linear unbiased estimate based on the multidimensional anomaly feature vector values ​​of known points surrounding the missing coordinate point, and considering the correlation distance between the known points and the missing defect in the time and spatial number dimensions. This estimate is then assigned to the missing coordinate point. By interpolating all missing defects using this algorithm, a continuous and smooth memory health state field is ultimately formed in the time and physical number dimensions. In this memory health state field, any timestamp and any physical number corresponds to a complete 12-dimensional feature vector.

[0077] See Figure 3In one embodiment of the present invention, a spatiotemporal convolutional network is applied to identify fault propagation patterns in a constructed memory health state field. The memory health state field is input to a pre-trained spatiotemporal convolutional network, which is composed of alternating stacks of causal convolutional layers in the temporal dimension and graph convolutional layers in the spatial dimension. The temporal causal convolutional layers of the spatiotemporal convolutional network are used to extract the evolution features of the memory health state field along the time axis. The spatial graph convolutional layers of the spatiotemporal convolutional network construct an adjacency matrix based on the physical connection topology of the server cluster, used to extract the correlation features between spatial nodes in the memory health state field. The final output layer of the spatiotemporal convolutional network fuses the temporal evolution features and spatial correlation features, mapping them to a risk field with the same dimensions as the input memory health state field, where the value of each coordinate point represents the fault risk intensity of that time and that physically numbered node. Based on the generated risk field, the generation of proactive repair decisions is initiated. In the risk field, coordinate points with fault risk intensity exceeding a preset risk threshold are identified as potential fault source nodes. Centered on each potential fault source node, neighboring nodes connected to it in the risk field along the direction of decreasing risk intensity gradient are identified in the spatial dimension. These neighboring nodes are connected to form a fault propagation path. For each identified potential fault source node, an isolation decision including its physical ID is generated. For all nodes on the fault propagation path, a migration decision is generated to migrate the memory load running on them to healthy nodes. For isolated nodes or nodes whose loads have been migrated, a switching decision is generated to switch their memory address space to a preset physical spare memory region. Finally, the isolation, migration, and switching decisions for all nodes are integrated into a complete proactive repair decision.

[0078] In specific implementations, based on the constructed memory health state field, which is a three-dimensional data field defined in the time dimension, physical number dimension, and feature dimension, the memory health state field is input to a pre-trained spatiotemporal convolutional network. The structure of the spatiotemporal convolutional network consists of alternating stacks of causal convolutional layers in the time dimension and graph convolutional layers in the spatial dimension. In some embodiments, the first layer of the spatiotemporal convolutional network can be a causal convolutional layer in the time dimension. The causal convolutional layer uses a one-dimensional convolutional kernel to perform convolution operations on the memory health state field along the time axis. Its convolutional kernel only covers the data at the current and historical moments, ensuring the causal relationship in time sequence. The output of this layer is an intermediate tensor containing the temporal evolution features. The second layer of a spatiotemporal convolutional network can be a spatial-dimensional graph convolutional layer. The graph convolutional layer's operation relies on an adjacency matrix reflecting the physical connection topology of the server cluster. This adjacency matrix is ​​constructed based on the actual data bus connections and memory channel sharing relationships between servers. The graph convolutional layer performs graph convolution operations on the intermediate tensor output from the previous layer along the physical numbering dimension based on this adjacency matrix to extract the correlation features of the memory health state field between spatial nodes. The spatiotemporal convolutional network can then continue to alternately stack multiple layers of causal convolutional layers and graph convolutional layers to extract deeper spatiotemporal features.

[0079] The final output layer of the spatiotemporal convolutional network receives the feature tensor output from the last hidden layer. This final output layer fuses the temporal evolution features with the spatial correlation features and performs a linear mapping operation. This linear mapping operation maps the fused high-dimensional features into a three-dimensional risk field with the exact same dimensions as the input memory health state field. That is, the risk field has the same dimensions as the input field in the timestamp and physical number dimensions, and a size of 1 in the third dimension, representing a single fault risk intensity value. The scalar value at each coordinate point in the risk field represents the fault risk intensity of the node at the corresponding time and physical number. The fault risk intensity is a continuous value, and its magnitude directly characterizes the probability of the node failing at that moment. The generation process of the risk field can be formally viewed as a process from the health state field... To the risk field nonlinear mapping Its expression is:

[0080]

[0081] in: This represents the input memory health state field. This represents the risk field of the output. This represents a nonlinear function parameterized by a spatiotemporal convolutional network. This represents the set of trainable weight parameters for all causal and graph convolutional layers in a spatiotemporal convolutional network.

[0082] In practical implementation, based on the generated risk field, the repair decision module initiates the logic for generating proactive repair decisions. In the three-dimensional risk field, coordinates where the fault risk intensity value exceeds a preset risk threshold are identified as potential fault source nodes. The preset risk threshold is either a static value determined through historical fault data analysis or a dynamic value dynamically adjusted according to the overall system load. Taking a small cluster containing four server nodes as an example, at the timestamp... In the risk field slice, assuming the risk intensity value of node 2 is 0.92 and the risk intensity value of node 3 is 0.45, with a preset risk threshold of 0.8, node 2 is identified as a potential fault source node, while node 3 is not identified. Taking each identified potential fault source node as the center, in the spatial dimension (i.e., within the same two-dimensional risk field slice at the same time stamp), the risk intensity values ​​of adjacent nodes directly connected to that potential fault source node in the physical topology are analyzed. Adjacent nodes with risk intensity values ​​lower than the central node and a decreasing gradient of risk intensity from the center to the adjacent node are identified. These adjacent nodes that meet the conditions are connected to form one or more fault propagation paths radiating outward from the potential fault source node. Continuing the example above, assuming nodes 1 and 4 connected to node 2 are... The risk intensities at time points are 0.7 and 0.6, respectively, and the risk intensity decreases from node 2 to node 1 and node 4. Therefore, node 2->node 1 and node 2->node 4 are identified as two fault propagation paths.

[0083] Understandably, for each identified potential fault source node, the repair decision module generates an isolation decision containing the node's physical ID. This isolation decision logically or physically marks the node as unavailable from the available memory resource pool. For all nodes along the fault propagation path, including potential fault source nodes and adjacent nodes, the repair decision module generates a migration decision to move the currently running memory workloads on these nodes to healthy nodes identified in the current risk field. Healthy nodes are those with a risk intensity value below a safety threshold. For isolated nodes or nodes whose workloads are moved out, the repair decision module generates a switching decision to switch their original memory address space mapping to a pre-defined physical spare memory region. This spare memory region consists of pre-reserved, physically isolated memory modules or memory cells in the hardware. Finally, the repair decision module integrates the isolation, migration, and switching decisions for all nodes according to their logical execution order, packaging them into a complete, serializable set of proactive repair decisions.

[0084] In one embodiment of the present invention, the repair control module encodes proactive repair decisions into a repair control instruction sequence and injects this sequence into the baseboard management controller of the server cluster to trigger a dynamic memory reconfiguration operation. Simultaneously with the dynamic reconfiguration operation, the online evaluation module collects real-time feedback performance degradation curves, reflecting changes in memory access latency and throughput during the repair process. The online evaluation module inputs the performance degradation curves to an online reinforcement learning evaluator, which evaluates the effectiveness of the executed repair control instruction sequence and outputs an evaluation reward signal. Based on the obtained evaluation reward signal, the online evaluation module dynamically adjusts the generation strategy of proactive repair decisions to optimize repair decisions in subsequent fault handling. After completing the dynamic memory reconfiguration, the closed-loop iteration module obtains updated memory health indicators from the baseboard management controller and feeds these updated indicators back into the memory health state field calculation process, thus forming a closed-loop fault prediction and repair iteration.

[0085] In practical implementation, the repair control module receives proactive repair decisions and converts them into low-level executable instructions. The repair control module encodes the proactive repair decisions into a sequence of repair control instructions, which is a combination of opcodes and operands arranged in a specific order and with clear semantics. In an example scenario, for a proactive repair decision that includes "isolate node 2, migrate the load of node 1 to node 5, and switch the address of node 2 to standby region A", the repair control module may generate the opcode sequence "ISOLATE_CHANNEL(2)", "MIGRATE_PAGES(1,5)", and "REMAP_ADDRESS(2,A)". The repair control module injects the repair control instruction sequence into the baseboard management controller of the server cluster. The injection process is completed by calling the application programming interface or command-line tool provided by the baseboard management controller to trigger dynamic memory reconfiguration operations. Dynamic reconfiguration operations include offline, online, address mapping changes, and channel configuration updates of memory units.

[0086] In practice, the online evaluation module collects real-time performance degradation curves while the baseboard management controller performs dynamic reconfiguration operations. These curves reflect the changes in key performance indicators of the memory subsystem over time, from the start to the end of the repair operation. The online evaluation module uses probes deployed in the server operating system kernel to sample real-time values ​​of memory access latency and memory throughput at a high frequency of milliseconds. The real-time values ​​of memory access latency are measured in nanoseconds, and memory throughput in gigabytes per second. Plotting sampling time on the horizontal axis and the real-time values ​​of memory access latency and memory throughput on the vertical axes respectively, the online evaluation module plots two time-series curves, which together constitute the performance degradation curve. The online evaluation module inputs the performance degradation curves into an online reinforcement learning evaluator, which is a continuously running policy evaluation model. The reinforcement learning evaluator evaluates the effectiveness of the executed repair control command sequence based on the stability and recovery speed of the system under repair disturbances, as reflected in the performance degradation curves. The reinforcement learning evaluator outputs a quantitative evaluation reward signal, and the value of the evaluation reward signal represents the overall effectiveness of the repair operation.

[0087] In some embodiments, the online evaluation module dynamically adjusts the generation strategy of proactive repair decisions based on the obtained evaluation reward signal. The adjustment process is achieved by updating the parameters of the decision model within the repair decision module. The decision model can be a neural network or a policy function. The evaluation reward signal is used as the basis for policy gradient calculation, and the model parameters of the decision model used to generate proactive repair decisions are updated through a policy gradient ascent algorithm. After the model parameters of the decision model are updated, when the system subsequently detects a risk field with similar characteristics again, the decision model in the repair decision module will tend to generate repair decisions that obtain higher evaluation reward signals. It can be understood that this process allows the system's repair strategy to learn from the effects of historical operations and continuously optimize. For example, if a decision sequence of migration followed by isolation produces a lower evaluation reward signal, while another sequence of isolation followed by migration produces a higher evaluation reward signal, the decision model will gradually adjust its internal weights to make the latter decision sequence more likely to be chosen in similar scenarios.

[0088] In practical implementation, the closed-loop iteration module starts after the baseboard management controller completes the dynamic reconfiguration of all memory. The closed-loop iteration module obtains updated memory health indicators from the baseboard management controller by sending standard management query commands. The updated memory health indicators include real-time readings of the memory module's current bit error rate, operating temperature, power supply voltage, and access latency after the repair operation. The closed-loop iteration module binds these updated memory health indicators with the new timestamp after the repair operation and the corresponding server physical number. The bound data is formatted with the same structure as the initial data acquisition module and appended as new data records to the end of the continuously acquired raw data stream. This newly appended data serves as input for the next calculation cycle, triggering a new round of memory health state field construction and risk prediction. The entire process forms a closed-loop iteration from fault prediction, decision generation, decision execution, effect evaluation to data feedback, allowing the system to continuously evolve its prediction and repair capabilities without manual intervention. In some embodiments, the process of the reinforcement learning evaluator calculating the comprehensive evaluation reward signal can be formalized as a reward function. The calculation formula is:

[0089]

[0090] in: This represents the immediate reward calculated from the current performance degradation curve, and its value is based on factors such as peak latency and recovery time. This represents a long-term reward for a period of time after a repair operation when the system operates stably without failure. It is a discount factor between 0 and 1, used to balance the weight of immediate rewards and long-term rewards.

[0091] In one embodiment of the present invention, the process by which the repair control module injects the repair control command sequence into the baseboard management controller of the server cluster specifically includes encoding the repair control command sequence into a series of executable remote command messages according to the command interface specification of the baseboard management controller. The repair control module sends these remote command messages sequentially to the baseboard management controller of the target server through the out-of-band management network. The baseboard management controller receives and parses the remote command messages, and executes low-level operations such as memory page detachment, memory address remapping, and memory channel disabling according to the instruction sequence therein. The process by which the online evaluation module collects real-time feedback performance degradation curves specifically includes sampling the performance indicators of the memory subsystem at a high frequency through the operating system kernel module during the dynamic reconfiguration operation of the baseboard management controller. The collected performance indicators include the real-time values ​​of memory access latency and memory throughput. Two time-series curves are plotted with the sampling time as the horizontal axis and the real-time values ​​of memory access latency and memory throughput as the vertical axes, respectively. These two curves together constitute the performance degradation curve. The process of inputting the performance degradation curve into the online reinforcement learning evaluator specifically includes extracting key evaluation features from the performance degradation curve. These key evaluation features include the time from the start of the repair operation to the peak memory access latency, the magnitude of the peak memory access latency, and the time required for memory throughput to recover to a steady-state level. The extracted key evaluation features are then input into the policy evaluation network of the reinforcement learning evaluator, which outputs a scalar value as an immediate reward. Combined with a pre-set long-term stable operating reward, a comprehensive evaluation reward signal for the execution of this repair control instruction sequence is calculated.

[0092] In practice, the process of the repair control module injecting the repair control command sequence into the baseboard management controller of the server cluster follows a defined procedure. The repair control command sequence is encoded into a series of executable remote command messages according to the baseboard management controller's command interface specification. The baseboard management controller's command interface specification typically defines a fixed structure for the messages, including fields such as a message header, opcode, operands, and checksum. For example, a command for memory page detachment might be encoded as a message header "0xA1", opcode "OFFLINE_PAGE", operands "starting physical address" and "number of pages", and a cyclic redundancy check (CRC) code. The repair control module sequentially sends the encoded remote command messages to the baseboard management controller of the target server via an out-of-band management network. This out-of-band management network is a dedicated management channel independent of the business data network, ensuring that the transmission of control commands is not affected by business load. At the receiving end, the baseboard management controller parses the remote command messages, verifies the message header and checksum, extracts the opcode and operands, and executes the underlying operations according to the order of the instructions. These underlying operations include memory page detachment, memory address remapping, and memory channel disabling. The memory page offline operation marks the specified physical memory page as unavailable and removes it from the operating system's available memory pool; the memory address remapping operation updates the mapping relationship between the application's logical address and the faulty physical address to a healthy spare physical address; and the memory channel disabling operation physically shuts down the signaling path to the faulty memory cell.

[0093] In practice, the online evaluation module collects real-time feedback performance degradation curves synchronously during the dynamic reconfiguration operation of the baseboard management controller. The online evaluation module samples the performance metrics of the memory subsystem at high frequency through the operating system kernel module. The kernel module acquires data by hooking the memory controller's performance monitoring counter or directly reading the memory timer register. The collected performance metrics include real-time values ​​of memory access latency and memory throughput. The real-time value of memory access latency is in nanoseconds, obtained by measuring the number of clock cycles elapsed during a cache miss memory read operation and converting it to time units. The real-time value of memory throughput is in gigabytes per second, obtained by statistically analyzing the total amount of data passing through the memory controller per unit time. With sampling time on the horizontal axis and the real-time values ​​of memory access latency and memory throughput on the vertical axes, the online evaluation module plots two time-series curves, which together constitute the performance degradation curve. The performance degradation curve records the complete process from the issuance of the repair command to the system performance reaching the peak of the disturbance and then recovering to a new steady state. Its morphological characteristics directly reflect the instantaneous impact of the repair operation on the system and the system's recovery capability. Refer to Table 1 for a simplified example of a performance degradation curve segment.

[0094] Table 1: Example Table of Performance Degradation Curve Data Segments

[0095]

[0096] In some embodiments, the process of inputting the performance degradation curve into an online reinforcement learning evaluator includes feature extraction and reward calculation, extracting key evaluation features from the performance degradation curve. Key evaluation features include the time from the start of the fix operation to the peak memory access latency. Peak memory access latency relative to baseline delay amplitude And memory throughput recovers from the disturbance to a steady-state level. Time required Baseline delay With steady-state throughput This is the average performance value of the system during its stable operation phase before the repair operation. It can be understood that the magnitude... The calculation requires reference to the baseline value; it does not directly use the peak value. Instead of calculating the absolute value, we calculate its relative growth rate. Key evaluation features are input into the policy evaluation network of the reinforcement learning evaluator, which is a multilayer perceptron that maps the input feature vectors to a scalar value as an immediate reward. Combined with pre-set long-term stable operation rewards Calculate the comprehensive evaluation reward signal of the execution of this repair control command sequence. Long-term stable operation reward This is a fixed positive reward obtained if the system does not trigger any new fault prediction events within a preset observation period after the repair operation is completed. The reward signal is evaluated comprehensively. The calculation formula is:

[0097]

[0098] in: It is a negative correlation function with respect to the time it takes for the delay to reach the peak value; the shorter the peak time, the higher the function value. It is a negative correlation function with respect to the peak delay amplitude; the smaller the amplitude, the higher the function value. It is a negative correlation function with throughput recovery time; the shorter the recovery time, the higher the function value. These are preset weighting coefficients used to balance the contribution of different features to the overall reward.

[0099] See Figure 4This is a performance degradation curve of the memory repair process, used to show the real-time changes in memory access latency and throughput during the dynamic repair of server memory. It is a performance degradation curve collected by the online evaluation module. Memory access latency reaches 398 nanoseconds at around 35 milliseconds, then gradually decreases, eventually stabilizing at around 100 nanoseconds. The latency spikes sharply after the repair operation is triggered, reflecting the instantaneous disturbance to the system caused by operations such as memory page detachment and address remapping. Memory throughput drops to approximately 42 gigabytes per second at around 25 milliseconds, then gradually recovers, eventually stabilizing at around 130 gigabytes per second. Throughput and latency are negatively correlated; throughput decreases as latency increases, demonstrating the suppression of memory subsystem performance by the repair operation. After the repair command is issued in the 0–25 millisecond range, the memory controller performs page detachment and channel disabling operations, resulting in longer access paths, limited concurrency, a rapid increase in latency, and a sharp drop in throughput. In the 25–40 millisecond range, the faulty node is isolated, the load is migrated to the spare memory area, and the system is in a transitional state of remapping and load balancing, reaching its lowest performance point. Address remapping was completed in 40–90 milliseconds, load migration ended, the system gradually established new memory access paths, latency dropped, throughput rebounded and exceeded the baseline, indicating that the system performance was optimized after the repair.

[0100] In one embodiment of the present invention, the generation strategy for proactive repair decisions is dynamically adjusted based on the evaluation reward signal. The policy gradient of the current proactive repair decision generation strategy is calculated using the evaluation reward signal. The model parameters of the decision model used to generate proactive repair decisions are updated using a policy gradient ascent algorithm. After the model parameters of the decision model are updated, when generating proactive repair decisions for similar risk fields, the decision model will tend to generate repair decisions that can obtain higher evaluation reward signals. After completing the dynamic reconfiguration of memory, the closed-loop iteration module obtains updated memory health indicators from the baseboard management controller. The acquisition process includes sending a health status query request to the baseboard management controller after the baseboard management controller completes all repair control instructions. The module receives real-time readings of the current memory module's bit error rate, operating temperature, power supply voltage, and access latency returned by the baseboard management controller and uses these readings as updated memory health indicators. The updated memory health indicators are bound with the corresponding timestamp and server physical number and appended to the original data stream. This data will be used for the next round of memory health status field construction and risk prediction.

[0101] In practical implementation, the generation strategy for proactive repair decisions is dynamically adjusted based on the evaluation reward signal. The repair decision module contains a parameterized decision model, which can be a deep neural network or a policy function. The evaluation reward signal is input to the policy optimizer, which uses the evaluation reward signal to calculate the policy gradient of the current proactive repair decision generation strategy. The policy gradient quantifies the direction and magnitude of the expected change in the evaluation reward signal when the decision model parameters change slightly. The model parameters of the decision model used to generate proactive repair decisions are updated using the policy gradient ascent algorithm. The algorithm adjusts the model parameters along the direction of the policy gradient to obtain higher future rewards when facing similar states. The model parameter update process of the decision model involves a learning rate hyperparameter, which controls the step size of each parameter update. In a specific parameter update calculation, the model parameter vector... The updates follow these rules:

[0102]

[0103] in: This represents the parameter vector of the decision model before the update. This represents the policy gradient calculated based on the current evaluation reward signal. This represents the preset learning rate. This represents the updated decision model parameter vector. After the decision model parameters are updated, when the risk prediction module outputs a risk field with similar characteristics to the historical scenario, the decision model in the repair decision module, due to the change in its internal parameters, will tend to generate a repair decision that yields a higher evaluation reward signal during decision inference. It can be understood that if the historical decision "migrate first, then isolate" resulted in a lower evaluation reward signal under a specific risk field pattern, while the decision "isolate first, then migrate" resulted in a higher evaluation reward signal, then the policy gradient update will adjust the model parameters in the direction of increasing the probability of choosing the "isolate first, then migrate" decision.

[0104] In some embodiments, after completing the dynamic reconfiguration of memory, the closed-loop iteration module obtains updated memory health indicators from the baseboard management controller. The acquisition process begins by sending a health status query request conforming to the intelligent platform management interface specification to the baseboard management controller that performed the repair operation. The health status query request is a remote command message. Upon receiving this request, the baseboard management controller reads the latest values ​​from the sensors on the memory module and organizes them into a response message to return. The closed-loop iteration module receives the response message returned by the baseboard management controller, parses the message content to obtain real-time readings of the current memory module's bit error rate, operating temperature, power supply voltage, and access latency. These real-time readings serve as the updated memory health indicators. For example, a query might return data such as "Bit Error Rate: 1.2E-12", "Operating Temperature: 48 degrees Celsius", "Power Supply Voltage: 1.2 volts", and "Access Latency: 89 nanoseconds". The closed-loop iteration module binds the updated memory health indicators with the timestamp of the query operation and the physical server ID corresponding to the query target. The bound data format is completely consistent with the format of the raw data stream collected by the data acquisition module. The bound data is appended as a new record to the end of the raw data stream that the system continuously maintains. The raw data stream is a time-series database or data file arranged in chronological order. This newly appended data contains the latest state of the system after the repair operation, and it will serve as input to the feature extraction module in the next computation cycle to drive a new round of memory health state field construction and risk prediction.

[0105] See Figure 5 This is a multi-dimensional time-series monitoring curve of server memory, showing the time-series changes of four key operating indicators of server memory over a 9-hour period. It is the raw data stream acquired by the data acquisition module, used for subsequent anomaly feature extraction and health status modeling. Memory access latency has been steadily increasing linearly, from 80 nanoseconds to 115 nanoseconds, an increase of approximately 43.8%. This continuous increase in latency is a typical precursor to memory hardware aging or failure, indicating a decline in the stability of the memory access path. Operating temperature has been slowly increasing linearly, from 45℃ to 54℃, an increase of approximately 20%. Increased temperature accelerates electron migration in memory chips, a significant contributing factor to increased bit error rate and latency. Supply voltage has been slowly decreasing linearly, from 1.20V to 1.12V, a decrease of approximately 6.7%. Voltage fluctuations affect the signal integrity of memory, and lower voltage increases the probability of data transmission errors. Bit error rate (BER) has been slowly increasing linearly, from 0 to approximately 6.5. The BER is a direct indicator of memory health; its continuous increase indicates a deterioration in the reliability of the memory chips.

[0106] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A server memory fault AI prediction and dynamic repair protection system, characterized in that, include: The data acquisition module continuously collects raw data streams corresponding to the memory running status from the deployed server cluster; The feature extraction module performs multi-dimensional abnormal fluctuation feature extraction on the original data stream, detects the abnormal fluctuation features of each time-series data point in the original data stream based on an adaptive sliding window, and integrates the extracted multiple abnormal fluctuation features into a multi-dimensional abnormal feature vector. The state modeling module uses the multidimensional anomaly feature vector to construct a time-varying memory health state field. The memory health state field uses the timestamp as the time axis, the server node number as the spatial axis, and the component values ​​of the multidimensional anomaly feature vector as the field strength dimension. The risk prediction module uses a spatiotemporal convolutional network to identify fault propagation patterns in the memory health state field. The spatiotemporal convolutional network captures the correlation of the memory health state field in the spatial dimension and the evolution of the time sequence dimension, and outputs a risk field that represents potential fault source nodes and fault propagation paths. The repair decision module generates proactive repair decisions based on the risk field. These proactive repair decisions include decisions on isolating faulty nodes, migrating loads, and switching backup memory regions.

2. The server memory fault AI prediction and dynamic repair protection system according to claim 1, characterized in that, Multi-dimensional abnormal fluctuation feature extraction is performed on the original data stream, including: The raw data stream includes the memory module's bit error rate timing, operating temperature timing, power supply voltage timing, and access latency timing; The abnormal fluctuation characteristics include sudden increase, decay gradient, and oscillation frequency; The statistical characteristics of the data within the window are calculated by applying an adaptive length sliding window to the bit error rate timing, operating temperature timing, power supply voltage timing, and access delay timing, respectively. The statistical characteristics include mean offset and standard deviation rate of change. The difference sequence between adjacent data points is calculated within a sliding window, and segments that rise or fall continuously are identified from the difference sequence. The cumulative amplitude of the segments is extracted as the burst value. Perform a second difference operation on the difference sequence and calculate the average absolute value of the second difference sequence as the decay gradient characterizing the intensity of fluctuation; Perform a Fast Fourier Transform on the original data stream to extract the main frequency components in the spectrum that exceed the background noise threshold, and calculate the average frequency of the main frequency components as the oscillation frequency.

3. The server memory fault AI prediction and dynamic repair protection system according to claim 2, characterized in that, Constructing a time-varying memory health state field using the aforementioned multidimensional anomaly feature vectors includes: Establish a three-dimensional spatial structure, where the first dimension is the timestamp sequence, the second dimension is the physical number sequence of all memory modules in the server cluster, and the third dimension is the components of the multidimensional anomaly feature vector. Assign the multidimensional anomaly feature vector corresponding to each timestamp and each physical number to the corresponding coordinate point of the three-dimensional spatial structure; For the missing coordinate points between adjacent timestamps and adjacent physical numbers in the three-dimensional spatial structure, the spatiotemporal kriging interpolation algorithm is used to interpolate and fill them based on the multidimensional abnormal feature vector values ​​of the surrounding known points, forming a continuous and smooth memory health state field.

4. The server memory fault AI prediction and dynamic repair protection system according to claim 3, characterized in that, Applying a spatiotemporal convolutional network to identify fault propagation patterns in the memory health state field includes: The memory health state field is used as input and fed into a pre-trained spatiotemporal convolutional network, which is composed of alternating stacks of causal convolutional layers in the time dimension and graph convolutional layers in the spatial dimension. The temporal causal convolutional layer of the spatiotemporal convolutional network is used to extract the evolution features of the memory health state field along the time axis; The spatial dimension graph convolutional layer of the spatiotemporal convolutional network constructs an adjacency matrix based on the physical connection topology of the server cluster, which is used to extract the correlation features between spatial nodes in the memory health state field; The final output layer of the spatiotemporal convolutional network fuses temporal evolution features with spatial correlation features and maps them into a risk field with the same dimension as the input memory health state field, where the value of each coordinate point represents the fault risk intensity of the time and physical number node.

5. The server memory fault AI prediction and dynamic repair protection system according to claim 4, characterized in that, Based on the aforementioned risk field, proactive remediation decisions are generated, including: In the risk field, coordinate points whose fault risk intensity exceeds a preset risk threshold are identified as potential fault source nodes; Taking each potential fault source node as the center, find the adjacent nodes in the risk field that are connected to it and whose risk intensity gradient is decreasing in the spatial dimension, and connect the adjacent nodes to form a fault propagation path. For each identified potential fault source node, generate an isolation decision including its physical number; For all nodes along the fault propagation path, generate migration decisions to move the memory load running on them to healthy nodes; For isolated nodes or nodes whose load is moved out, a switching decision is generated to switch their memory address space to a preset physical spare memory region; The isolation, migration, and switching decisions for all nodes are integrated into proactive repair decisions.

6. The server memory fault AI prediction and dynamic repair protection system according to claim 5, characterized in that, Also includes: The repair control module encodes proactive repair decisions into a repair control instruction sequence and injects the repair control instruction sequence into the baseboard management controller of the server cluster, triggering a dynamic memory reconfiguration operation. The online evaluation module collects real-time performance degradation curves while the dynamic reconfiguration operation is being executed. These performance degradation curves reflect changes in memory access latency and throughput during the repair process. The performance degradation curves are then input into an online reinforcement learning evaluator, which evaluates the effectiveness of the executed repair control instruction sequence and outputs an evaluation reward signal. Based on the evaluation reward signal, the generation strategy for proactive repair decisions is dynamically adjusted to optimize subsequent fault handling repair decisions. The closed-loop iteration module, after completing the dynamic reconfiguration of memory, obtains the updated memory health indicators from the baseboard management controller and feeds the updated memory health indicators back to the calculation process of the memory health status field, forming a closed-loop fault prediction and repair iteration.

7. The server memory fault AI prediction and dynamic repair protection system according to claim 6, characterized in that, The baseboard management controller that injects the repair control command sequence into the server cluster specifically includes: The repair control command sequence is encoded into a series of executable remote command messages according to the command interface specification of the baseboard management controller; Remote command messages are sent sequentially to the baseboard management controller of the target server via the out-of-band management network; The baseboard management controller receives and parses remote command messages, and executes low-level operations such as memory page detachment, memory address remapping, and memory channel disabling according to the instructions in the messages.

8. The server memory fault AI prediction and dynamic repair protection system according to claim 7, characterized in that, The performance degradation curve obtained through real-time feedback specifically includes: During the dynamic reconfiguration operation performed by the baseboard management controller, the performance metrics of the memory subsystem are sampled at a high frequency by the operating system kernel module. The collected performance metrics include real-time values ​​of memory access latency and real-time values ​​of memory throughput. With sampling time on the horizontal axis and the real-time values ​​of memory access latency and memory throughput on the vertical axes respectively, two time series curves are plotted to form a performance degradation curve. The performance degradation curve is input into an online reinforcement learning evaluator, specifically including: Key evaluation features are extracted from the performance degradation curve. These key evaluation features include the time from the start of the repair operation to the peak memory access latency, the magnitude of the peak memory access latency, and the time required for memory throughput to recover to a steady-state level. Key evaluation features are input into the policy evaluation network of the reinforcement learning evaluator, and the policy evaluation network outputs a scalar value as an immediate reward. Based on the preset long-term stable operation reward, a comprehensive evaluation reward signal for the execution of this repair control command sequence is calculated.

9. The server memory fault AI prediction and dynamic repair protection system according to claim 8, characterized in that, Based on the evaluation of reward signals, the generation strategy for proactive repair decisions is dynamically adjusted, including: The policy gradient of the current proactive repair decision generation strategy is calculated using the evaluation reward signal; The model parameters of the decision model used to generate proactive repair decisions are updated using the policy gradient ascent algorithm. After the model parameters of the decision-making model are updated, when generating proactive repair decisions for similar risk fields, the model will tend to produce repair decisions that can obtain higher evaluation reward signals.

10. The server memory fault AI prediction and dynamic repair protection system according to claim 9, characterized in that, After completing the dynamic reconfiguration of memory, the updated memory health metrics are obtained from the baseboard management controller, including: After the baseboard management controller completes all repair control instructions, a health status query request is sent to the baseboard management controller. The system receives real-time readings of the current memory module's bit error rate, operating temperature, power supply voltage, and access latency from the baseboard management controller, which are then used as updated memory health indicators. The updated memory health metrics are bound to the corresponding timestamps and server physical numbers and appended to the original data stream for use in the next round of memory health status field construction and risk prediction.

Images