PCIe device fault monitoring and dynamic processing method based on intelligent prediction

Abstract

The present application provides a method for monitoring and dynamically handling PCIe device faults based on intelligent prediction. The method obtains operating parameters from the baseboard management controller BMC, the operating system, and the PCIe device registers, processes and extracts time series features, and then inputs the parameters into a lightweight long short-term memory network LSTM model deployed in the BMC to predict the probability of faults. The device health score is calculated based on the prediction results and the operating parameters, and alarms, bandwidth degradation, or hardware isolation operations are triggered based on the score. The hardware isolation module sends control instructions to the complex programmable logic device CPLD via the I²C bus to achieve uninterrupted power-off isolation of the target device. Real-time data is uploaded to the cloud server through encrypted communication, and the updated weights are transmitted back after model fine-tuning to achieve continuous optimization and version control of the model. The present invention can be widely used in computing devices such as servers to improve system availability and fault tolerance.

Description

Technical Field

[0001] The present invention relates to the field of server hardware health management, and in particular to a PCIe device fault monitoring and dynamic processing method based on intelligent prediction. Background Art

[0002] PCIe (Peripheral Component Interconnect Express), as the core interconnect technology in server hardware systems, is widely used in scenarios with stringent requirements on system stability and availability, such as data centers, cloud computing platforms, and high-performance computing clusters.

[0003] With the increasing complexity of server architectures and device density, the reliability of PCIe devices has become a critical factor affecting overall system performance. Related technologies utilize the collaborative operation of the Baseboard Management Controller (BMC), operating system, and hardware registers to build a device status monitoring and fault response system. Specifically, this system covers the entire process from data collection and anomaly detection to fault isolation, including key steps such as IPMI protocol alarms, static threshold determination, and software logging. However, existing PCIe fault monitoring methods rely solely on fixed thresholds and software alarm mechanisms, without incorporating intelligent predictive models based on time series data. This can lead to false positives or missed positives, or require system reboots for fault isolation after a fault occurs, impacting business continuity and system availability. Furthermore, most solutions lack deep integration with Complex Programmable Logic Devices (CPLDs), preventing fast and seamless fault isolation at the hardware level. This limits the system's fault tolerance and operational efficiency.

[0004] Existing technologies have the following drawbacks: Passive fault response: Traditional solutions rely on the baseboard management controller (BMC) to log and trigger alarms (e.g., sending SNMP traps via the IPMI protocol) after a PCIe device failure. For example, CN202010125180.1 discloses a method for isolating faulty devices through host control. However, this method relies on operating system intervention, cannot achieve continuous electrical isolation at the hardware level, and lacks fault prediction capabilities. Limitations of fixed threshold alarms: Existing technologies typically use static thresholds (e.g., bit error rate > 1e-6) to identify anomalies. However, PCIe device parameters fluctuate significantly under different loads, and fixed thresholds are prone to false alarms (e.g., bit error rate naturally increases under high load) or missed alarms (e.g., intermittent faults are not detected). Insufficient hardware control capabilities: Most solutions only support fault logging at the software level and lack deep integration with programmable logic devices (CPLDs). This results in an inability to quickly disconnect the power to the faulty device, requiring a system restart to recover, impacting business continuity. Summary of the Invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] The present invention proposes a PCIe device fault monitoring and dynamic processing method based on intelligent prediction, aiming to solve the problems of delayed fault response, frequent false alarms and missed alarms, and lack of hardware-level isolation capabilities in the existing technology, thereby improving the stability, availability and maintainability of the server system.

[0007] Another object of the present invention is to provide a PCIe device fault monitoring and dynamic processing device based on intelligent prediction.

[0008] To achieve the above objectives, the present invention provides a method for monitoring and dynamically handling PCIe device faults based on intelligent prediction, comprising:

[0009] In response to a preset acquisition cycle, device operating parameters are obtained from a baseboard management controller (BMC), an operating system, and PCIe device registers;

[0010] Preprocess the equipment operating parameters and extract the time series features to generate standardized feature vectors;

[0011] Based on the standardized feature vector, a lightweight long short-term memory network (LSTM) model deployed on the BMC is used for model inference to output the probability of device failure within a preset time window in the future.

[0012] Calculate the equipment health score based on the failure probability and equipment operating parameters combined with the dynamic weight coefficient;

[0013] In response to the health score, triggering an alarm, bandwidth degradation, or hardware isolation operation based on a preset hierarchical response strategy;

[0014] In response to the hardware isolation operation instruction, a control instruction is sent to a complex programmable logic device (CPLD) via the I²C bus. After the CPLD interprets the control instruction, it controls the power switch module to disconnect the power supply line of the target device.

[0015] In response to the model update request, the encrypted real-time data is uploaded to the cloud server. After the cloud server performs model fine-tuning, the updated model weights are transmitted back to the BMC via the HTTPS protocol. The BMC updates the local model or rolls back to the stable version according to the model version control strategy.

[0016] The PCIe device fault monitoring and dynamic processing method based on intelligent prediction according to an embodiment of the present invention may also have the following additional technical features:

[0017] In one embodiment of the present invention, temperature and voltage data are obtained from the BMC via the IPMI protocol, operating system commands are called via the SSH protocol to obtain PCIe device register information, and AER registers are read via the PCIe configuration space to capture error details.

[0018] In one embodiment of the present invention, a sliding window mean method is used to fill missing values, Z-score normalization is used to eliminate parameter dimensional differences, and timing features are extracted, including the sliding mean of the bit error rate, the cumulative count of CRC errors, and the temperature deviation; wherein the Z-score represents the new value of each original data point after normalization.

[0019] In one embodiment of the present invention, the LSTM model has a hidden layer dimension of 64, a time step of 10, a model volume compressed to less than 800KB, is adapted to the ARM Cortex-A53 processor of the BMC, and has an inference frequency of once every 5 minutes.

[0020] In one embodiment of the present invention, the device health score is calculated based on the normalized bit error rate, the number of CRC errors accumulated in the past hour, and the deviation of the device temperature from the rated value, combined with a dynamic weight coefficient.

[0021] In one embodiment of the present invention, different levels of response strategies are triggered based on the numerical range of the health score, including:

[0022] When the health score is greater than or equal to 80, only an alarm message is generated;

[0023] When the health score is less than 80 and greater than or equal to 30, the bandwidth degradation operation is triggered;

[0024] When the health score is less than 30, hardware isolation is triggered.

[0025] In one embodiment of the present invention, the bandwidth degradation operation adjusts the link width through the configuration register of the PCIe switch and checks the availability of the backup link before degradation. If the backup link is unavailable, the degradation is skipped and isolation is triggered directly.

[0026] In one embodiment of the present invention, the control instruction is in a 16-bit format, including an operation code, a target PCIe slot address and a reserved field, wherein the operation code is 0xA1, which means cutting off the power supply, the address field is the physical address of the target slot, and the reserved field defaults to 0x00.

[0027] In one embodiment of the present invention, the control instruction is appended with a CRC-8 check code, and the CPLD verifies data integrity after receiving the instruction. If no ACK signal is received, the BMC resends the instruction after 100ms and retries up to 3 times.

[0028] To achieve the above objectives, the present invention further provides a PCIe device fault monitoring and dynamic processing device based on intelligent prediction, comprising:

[0029] The data acquisition module obtains device operating parameters from the baseboard management controller (BMC), the operating system, and the PCIe device registers in response to a preset acquisition cycle;

[0030] A preprocessing module, which preprocesses the equipment operating parameters and extracts time series features to generate a standardized feature vector;

[0031] The prediction module performs model inference based on the standardized feature vector using a lightweight long short-term memory network (LSTM) model deployed on the BMC, and outputs the probability of device failure within a preset time window in the future.

[0032] The health scoring module calculates the equipment health score based on the failure probability and equipment operating parameters combined with the dynamic weight coefficient;

[0033] a response control module, responsive to the health score and based on a preset hierarchical response strategy, triggering an alarm, bandwidth degradation, or hardware isolation operation;

[0034] The hardware isolation module sends a control instruction to a complex programmable logic device (CPLD) via an I²C bus in response to the hardware isolation operation instruction. After the CPLD interprets the control instruction, it controls the power switch module to disconnect the power supply line of the target device. The control instruction includes an operation code, a target PCIe slot address, and a reserved field.

[0035] The model update module responds to model update requests by uploading encrypted real-time data to the cloud server. After the cloud server performs model fine-tuning, it transmits the updated model weights back to the BMC via the HTTPS protocol. The BMC updates the local model or rolls back to the stable version according to the model version control strategy.

[0036] The intelligent prediction-based PCIe device fault monitoring and dynamic handling method and apparatus of the present invention acquire operating parameters from the baseboard management controller (BMC), operating system (OS), and PCIe device registers through data acquisition. After preprocessing to fill missing values, normalize, and extract time-series features, these parameters are input into a lightweight long-short-term memory (LSTM) model deployed in the BMC to predict fault probabilities. The predicted results are combined with operating parameters to calculate a device health score, which triggers alarms, bandwidth degradation, or hardware isolation based on the score. The hardware isolation module sends control commands to a complex programmable logic device (CPLD) via the I²C bus, achieving non-disruptive power-off isolation of the target device. Real-time data is uploaded to a cloud server via encrypted communication, and after model fine-tuning, updated weights are transmitted back, enabling continuous model optimization and version control. This invention is suitable for scenarios with stringent device reliability requirements, such as data centers, cloud computing platforms, and high-performance computing clusters, significantly improving system availability and reducing operation and maintenance costs.

[0037] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 is a flow chart of a method for monitoring and dynamically handling PCIe device faults based on intelligent prediction according to an embodiment of the present invention;

[0040] Figure 2 is an architectural diagram of a PCIe device fault monitoring and dynamic processing method based on intelligent prediction according to an embodiment of the present invention;

[0041] Figure 3 4 is a structural diagram of a PCIe device fault monitoring and dynamic processing device based on intelligent prediction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0044] The following describes a method and apparatus for monitoring and dynamically handling PCIe device faults based on intelligent prediction according to an embodiment of the present invention with reference to the accompanying drawings.

[0045] Figure 1 FIG. 1 is a flow chart of a method for monitoring and dynamically handling PCIe device faults based on intelligent prediction according to an embodiment of the present invention. Figure 1 Shown, including:

[0046] S1, in response to a preset acquisition cycle, obtaining device operating parameters from a baseboard management controller (BMC), an operating system, and a PCIe device register;

[0047] S2, preprocessing the equipment operating parameters and extracting the time series features to generate a standardized feature vector;

[0048] S3, based on the standardized feature vector, performs model inference using a lightweight long short-term memory network (LSTM) model deployed on the BMC, and outputs the probability of device failure within a future preset time window;

[0049] S4, calculates the equipment health score based on the failure probability and equipment operating parameters combined with the dynamic weight coefficient;

[0050] S5, in response to the health score, triggering an alarm, bandwidth degradation, or hardware isolation operation based on a preset hierarchical response strategy;

[0051] S6, in response to the hardware isolation operation instruction, sending a control instruction to the complex programmable logic device (CPLD) via the I²C bus. After the CPLD parses the control instruction, it controls the power switch module to disconnect the power supply line of the target device.

[0052] S7, in response to the model update request, uploads the encrypted real-time data to the cloud server. After the cloud server performs model fine-tuning, it transmits the updated model weights back to the BMC via the HTTPS protocol. The BMC updates the local model or rolls back to the stable version according to the model version control strategy.

[0053] In this embodiment of the present invention, the system uses multiple protocols and interfaces to obtain key operating status information from the hardware layer for subsequent health assessment and fault prediction. Specifically, the system obtains real-time monitoring data such as device temperature and power supply voltage from the BMC (Baseboard Management Controller) via the IPMI protocol. This data reflects the stability of the server's overall operating environment. Simultaneously, the system remotely connects to the operating system via the SSH protocol and executes specific commands to read PCIe device register information, such as link status and error counts, for analyzing device communication quality. Furthermore, the system directly accesses the AER (Advanced Error Reporting) register in the PCIe configuration space to capture detailed error types, occurrence times, and error levels, providing a basis for subsequent error classification and fault location.

[0054] In order to improve the quality and availability of collected data, the system performs a series of preprocessing operations on the raw data. First, the sliding window mean method is used to fill in missing values, effectively dealing with data breakpoints caused by abnormal sensors or acquisition channels, and ensuring the integrity of time series data. Subsequently, the Z-score standardization method is used to normalize parameters of different dimensions to eliminate model deviations caused by unit differences. On this basis, the system further extracts multiple representative time series features, including the sliding mean of the bit error rate, the cumulative count of CRC check errors, and the deviation of the device temperature from the rated value. These features together constitute the model input vector, providing a data basis for the calculation of the health score.

[0055] It's understandable that in the Z-score normalization process for multi-dimensional parameters, "score" refers to the Z-score, which represents the normalized value of each original data point. This new value reflects the position of the original data point relative to the mean of the entire dataset. Specifically, it indicates how many standard deviations away a particular data point is from the mean. This approach allows comparison not only of data with the same characteristic but different scales, but also of data with different characteristics.

[0056] The system uses a lightweight LSTM neural network model to predict device health status. This model contains one hidden layer with a dimension of 64, which effectively captures long-term dependencies in time series data. The model's time step is set to 10, corresponding to historical data from the past 10 sampling points, and is used to model the behavioral characteristics of PCIe devices. To adapt to the resource constraints of the edge computing platform, the model is optimized and compressed to a size of less than 800KB. It is adapted to the ARM Cortex-A53 processor on the BMC and meets the 2GB memory limit constraint. The model inference frequency is set to once every 5 minutes, which ensures the timeliness of the prediction while avoiding excessive consumption of system resources by frequent calculations.

[0057] Based on the model's predictions and historical data, the system further calculates a device health score. This score takes into account the normalized bit error rate, the number of CRC errors accumulated over the past hour, and the deviation between the device's current temperature and its rated value, combined with a set of dynamic weighting factors. These weights are adaptively adjusted based on the actual deployment environment and device characteristics, ensuring that the score more closely reflects actual operating conditions.

[0058] Based on the above health scores, the system has designed a multi-level response strategy mechanism to guide subsequent operation and maintenance decisions:

[0059] When the health score is ≥ 80, it indicates that the device is in normal or slightly degraded state. In this case, only alarm logs are generated for reference by operation and maintenance personnel.

[0060] When the health score is <80 and ≥30, the device is considered to have potential risks and the system will trigger bandwidth degradation, lowering its priority in the topology.

[0061] When the health score is <30, it indicates that the device is in a high-risk state, and the system will immediately initiate the hardware isolation process to prevent it from further affecting system stability.

[0062] The bandwidth downgrade operation is implemented by the BMC by accessing the configuration registers of the PCIe switch (such as the PLX PEX8796) and modifying the link width field. Before performing the downgrade, the system automatically checks for the existence of a backup link or redundant path. If a usable path is detected, the bandwidth downgrade operation is allowed; otherwise, the downgrade process is skipped and the hardware isolation phase is directly initiated to prevent service unavailability due to link interruption.

[0063] Finally, at the hardware isolation level, the system uses a CPLD (Complex Programmable Logic Device) to control a MOSFET switch to cut off the power supply to the target device. The control instruction uses a 16-bit format, including an operation code (CMD), the target PCIe slot address (ADDR), and a reserved field (DATA), where:

[0064] The operation code is 0xA1, which means "cut off the power supply".

[0065] The address field is the physical address of the target PCIe slot, which is used to accurately locate the device to be controlled;

[0066] The data field is a reserved field with a default value of 0x00 and can be used for subsequent function expansion.

[0067] To ensure reliable command transmission, the system appends a CRC-8 checksum to control commands. This CRC-8 checksum is generated by the BMC and verified for data integrity by the CPLD. If the CPLD does not return an ACK signal within the specified time, the BMC automatically resends the command after 100ms, up to three times. If the CPLD fails again, the BMC marks the device as "communication abnormal" and triggers a higher-level alarm mechanism.

[0068] In summary, this paper builds a complete PCIe device health management and fault-tolerant control system through multi-protocol collaborative hardware status information collection, data preprocessing and time series feature extraction, lightweight LSTM model health trend prediction, and multi-level response strategy linkage for bandwidth degradation and hardware isolation. This highly automated and intelligent system is suitable for applications requiring high stability and availability, such as data centers, server clusters, and edge AI accelerator cards.

[0069] In one embodiment of the present invention, data collection and preprocessing are first required.

[0070] For example, the monitoring parameters are listed in Table 1:

[0071] Table 1 Monitoring parameters list

[0072]

[0073] The data collection methods include:

[0074] BMC layer: Obtain temperature and voltage data through IPMI commands (such as ipmitool sensor list).

[0075] Operating system layer: Call the lspci -vvv command to parse the PCIe device register information and extract the LinkCapabilities / Status field.

[0076] Hardware registers: Directly read the Advanced Error Reporting (AER) registers through the PCIe configuration space to capture error details.

[0077] The data preprocessing process is as follows:

[0078] Missing value processing: The sliding window mean method is used to fill short-term missing data (window size = 5 minutes).

[0079] Normalization: Z-score standardization is performed on multi-dimensional parameters to eliminate dimensional differences.

[0080] Feature Engineering: Extract time series features (e.g., 1-hour moving average of bit error rate, cumulative count of CRC errors).

[0081] Understandably, data preprocessing is a critical first step in building high-quality data analysis and modeling. This process aims to improve data quality, eliminate noise, and enhance feature expressiveness, thereby increasing the accuracy and stability of subsequent tasks. Preprocessing typically includes core steps such as missing value handling, data normalization, and feature engineering, and is applicable to scenarios such as various monitoring data, industrial sensor data, and IT system logs.

[0082] First, when dealing with missing values ​​during data collection, using a sliding window mean method is an effective strategy. By setting a 5-minute window and filling missing points with the average value of valid data within the local time range, we can preserve the original data trend while avoiding information loss caused by missing values. For cases with long periods of continuous missing values, we can also combine linear interpolation or anomaly marking to supplement the data to ensure data integrity.

[0083] Secondly, to eliminate dimensional differences between metrics across different dimensions and improve model training efficiency, multi-dimensional parameters were normalized using Z-scores to make features comparable. Furthermore, feature engineering was performed to extract time-series features, including the one-hour moving average of the bit error rate and the cumulative count of CRC errors, to reflect system operating trends and potential degradation signals. These high-level features help the model better capture dynamic changes in the data, providing strong support for subsequent tasks such as fault prediction and anomaly detection.

[0084] In one embodiment of the present invention, a machine learning model is also designed, and then the model is trained and applied.

[0085] The model architecture includes: a lightweight LSTM (Long Short-Term Memory) network with a hidden layer dimension of 64 and a time step of 10 (corresponding to 10 sampling points). The optimization objective is to predict the probability of PCIe failures within the next 30 minutes (a binary classification task). Lightweight design: Utilizing the TensorFlow Lite framework, the model is compressed to under 800KB and compatible with the BMC's ARM Cortex-A53 processor (with a memory limit of 2GB).

[0086] The training and update mechanism includes the following: Offline training phase: Dataset construction: Historical fault data (with fault occurrence time annotated) and normal data are collected, and the training, validation, and test sets are divided into a 7:2:1 ratio. Loss function: Weighted cross-entropy loss function, giving higher weight to faulty samples (weight ratio = 5:1). Training parameters: Learning rate = 0.001, batch size = 32, and number of iterations = 100.

[0087] During the online update phase, incremental learning is performed: The BMC encrypts and uploads new data to the cloud daily. The cloud server then fine-tunes the model, and the updated model is then transmitted back to the BMC via HTTPS. Version control is performed: The three most recent model versions are retained, supporting rapid rollbacks.

[0088] Specifically, the model architecture employed in this embodiment of the present invention is based on a lightweight long short-term memory (LSTM) network, suitable for real-time fault prediction tasks on edge devices. This LSTM network structure consists of one hidden layer with a dimension of 64, effectively capturing long-term dependencies in time series data. The input data time step is set to 10 sampling points, corresponding to historical operating status data collected by the system, which is used to model the behavioral characteristics of PCIe devices. The model's optimization objective is a binary classification task: predicting whether a PCIe fault will occur within the next 30 minutes. The output represents the probability of the fault occurring.

[0089] To accommodate the resource constraints of edge computing platforms, the model was designed to be lightweight and compressed and deployed using the TensorFlow Lite framework. The final model size was kept below 800KB, allowing it to run efficiently on the ARM Cortex-A53 processor on the BMC while meeting the 2GB memory limit. This model performs inference tasks at the edge without requiring a continuous network connection, ensuring low-latency response and data security.

[0090] During offline training, the model constructs a training set using historical monitoring data, including labeled faulty and normal operation samples. This set is divided into training, validation, and test sets in a 7:2:1 ratio to ensure good generalization. To address the scarcity of faulty samples, the model uses a weighted cross-entropy loss function, weighting faulty samples five times that of normal samples to mitigate bias caused by class imbalance. During training, the learning rate is set to 0.001, the batch size is 32, and 100 iterations are repeated. Early stopping is also used to prevent overfitting.

[0091] During the online update phase, the system supports an incremental learning mechanism, with the BMC encrypting and uploading newly collected data to the cloud server daily. After receiving the data, the cloud fine-tunes the original model, leveraging the new data to improve its timeliness and accuracy. The updated model is securely transmitted back to the BMC via HTTPS, enabling dynamic updates of the local model. To enhance the system's fault tolerance and stability, the cloud retains snapshots of the three most recent model versions and supports rapid rollback to any stable version, ensuring rapid service restoration in the event of model anomalies or performance degradation.

[0092] In summary, the present invention achieves high-precision, low-resource consumption prediction capabilities for PCIe device failures on the edge side through lightweight LSTM architecture design, strict training process control, and cloud-collaborative incremental learning mechanism, which is suitable for intelligent operation and maintenance scenarios of critical infrastructure such as servers.

[0093] Among them, reasoning and scoring output: Reasoning frequency: perform model reasoning every 5 minutes. Health scoring formula:

[0094]

[0095] in, It is an overall performance score used to quantify and measure the health or performance status of the system; BER norm : Normalized bit error rate (reference value = 1 / e -6 );CRC_Error cum : The cumulative number of CRC errors in the past hour; : Deviation of device temperature from rated value (unit: °C); , , : Dynamic weight coefficient (adjusted by model output).

[0096] In one embodiment of the present invention, a multi-level response mechanism is adopted.

[0097] Among them, the classification strategy table is shown in Table 2:

[0098] Table 2 Grading strategy table

[0099]

[0100] Among them, dynamic bandwidth adjustment is achieved:

[0101] Control interface: Adjust the link width through the configuration register of the PCIe switch (such as PLX PEX8796).

[0102] Fault-tolerance mechanism: Before bandwidth degradation, the system checks the availability of the backup link. If it is unavailable, it skips the degradation and directly triggers isolation.

[0103] Specifically, in this embodiment of the present invention, the dynamic bandwidth adjustment mechanism for PCIe devices is a crucial component for achieving fault prediction and fault tolerance. This mechanism dynamically adjusts bandwidth allocation by evaluating device operating status and communication link health in real time, mitigating the impact of high-risk devices on overall system performance and allowing for subsequent isolation operations.

[0104] Control Interface and Bandwidth Adjustment: The system accesses the internal configuration registers of PCIe switches (such as the PLX PEX8796) to obtain and modify key parameters such as link width, thereby dynamically controlling the bandwidth of target devices. For example, downgrading a link from x8 to x4, or further to x1, gradually restricting its data transfer capacity. This process is initiated by the BMC and interacts with the hardware through a dedicated management interface, ensuring low latency and high reliability. Furthermore, adjustment instructions are encapsulated as atomic operations to prevent link anomalies caused by partial execution.

[0105] Fault-tolerance mechanism and safe downgrade strategy: To ensure system stability, the system automatically checks for available backup links or redundant paths before executing a bandwidth downgrade. If an alternative link is detected in the current topology, the bandwidth downgrade is allowed. Otherwise, if no backup path is available, the system skips the downgrade process and directly triggers the hardware isolation mechanism to prevent service unavailability due to link interruption. This fault-tolerance mechanism effectively prevents the risk of unrecoverable bandwidth degradation, improving the system's adaptability and robustness.

[0106] In summary, the present invention achieves refined management of PCIe device communication resources by combining a register-level bandwidth control interface with a fault-tolerant degradation strategy with path checking, providing a flexible and secure fault response method for the edge intelligent operation and maintenance system.

[0107] In one embodiment of the present invention, the CPLD hardware controls the design.

[0108] Circuit design details include: The power control module uses two MOSFETs (such as the IRF3205) in parallel to control the 3.3V and 12V power lines, respectively. Signal transmission protocol: 16-bit control commands (format: 0x[CMD][ADDR][DATA]) are sent to the CPLD via the I²C bus (at 400kHz), where: CMD = 0xA1 (disconnects power); ADDR = the physical address of the target PCIe slot; DATA = a reserved field (default 0x00).

[0109] Fault tolerance mechanism:

[0110] CRC check: The instruction is appended with a CRC-8 check code, and the CPLD verifies the data integrity after receiving it.

[0111] Timeout retransmission: If no ACK signal is received, the BMC will resend the command after 100ms and retry up to 3 times.

[0112] Among them, the isolation operation process:

[0113] 1. The BMC detects a health score <30 and generates and encrypts an isolation instruction.

[0114] 2. The command is transmitted to the CPLD via the I²C bus.

[0115] 3. The CPLD parses the instruction and triggers the MOSFET switch to disconnect the 3.3V / 12V power supply of the target device.

[0116] 4. The CPLD returns a status code to the BMC, updating the device status to "Isolated".

[0117] In a preferred embodiment of the present invention, a complex programmable logic device (CPLD) is used to implement low-level hardware control functions for PCIe devices, including key operations such as power management, signal transmission, and fault isolation. This design, through a highly integrated circuit structure and reliable communication protocols, ensures that control instructions can be quickly responded to and accurately executed at the edge, making it suitable for applications such as servers and edge computing devices that require high system stability. The power control module uses a dual-MOSFET parallel design (such as the IRF3205) to control the 3.3V and 12V power supply lines of the target PCIe device, respectively. This design offers the advantages of low on-resistance, high current carrying capacity, and fast switching response, enabling efficient and stable power-off control.

[0118] Control commands are transmitted over the I²C bus (standard 400kHz) using a standardized 16-bit format: 0x[CMD][ADDR][DATA]. CMD represents the command (for example, 0xA1 means power off), ADDR is the physical address of the target PCIe slot, used to precisely locate the device to be controlled, and DATA is a reserved field with a default value of 0x00, which can be used for future functionality expansion. To enhance communication reliability, the system incorporates two core fault-tolerance mechanisms: a CRC-8 checksum, generated by the BMC and used to verify data integrity on the CPLD; and a timeout retransmission mechanism. If no ACK signal is received, the BMC automatically resends the command after 100ms, attempting up to three times to prevent control failures due to communication anomalies.

[0119] When the system determines that a PCIe device presents a critical failure risk (for example, if its health score falls below a set threshold of 30), it initiates an automatic isolation process. The BMC calculates the device's health score based on the LSTM model's predictions and operating parameters. If the score consistently falls below the safety threshold, the BMC generates an encrypted isolation command and sends it to the CPLD via the I²C bus. After decryption and CRC verification, the CPLD parses the target device's physical address and operation command, triggering the MOSFET switch to disconnect its 3.3V and 12V power supply lines, achieving non-interruptible power isolation. Once isolation is complete, the CPLD returns a status code (such as 0x00 for success) to the BMC, which updates the device's status to "Isolated" and logs the event details. The BMC can also report the event to the management system via an API or notification mechanism, forming a complete closed-loop control process.

[0120] Further, if Figure 2 As shown, the architecture of the PCIe device fault monitoring and dynamic processing method based on intelligent prediction of the present invention is composed of the following parts:

[0121] BMC (Baseboard Management Controller): Core control and data processing unit. Cloud server: Model training and update platform. Operating system (OS): Executes data acquisition commands (such as lspci). CPLD (Complex Programmable Logic Device): Hardware control module. PCIe device: Monitors target devices (such as GPUs and NVMe SSDs).

[0122] The connection relationship between each part is as follows:

[0123] BMC and operating system:

[0124] Data flow: The BMC uses the SSH protocol to call operating system commands (such as lspci -vvv) to obtain PCIe device register information.

[0125] The BMC (Baseboard Management Controller) establishes a secure communication channel with the operating system via the SSH protocol, enabling access to underlying hardware information and remote management tasks. Regarding data flow, the BMC can proactively invoke the operating system's command-line interface (CLI), such as executing the lspci -vvv command to obtain detailed register information and status for PCIe devices, enabling real-time monitoring of hardware health. Regarding control flow, the BMC can send control commands to the operating system to trigger specific data collection tasks or diagnostic processes, such as initiating log collection and running health check scripts, thereby enhancing the automation level of system operations and maintenance.

[0126] Control flow: The BMC sends instructions to the OS to trigger data collection tasks.

[0127] BMC and cloud server:

[0128] Data flow: The BMC uploads encrypted real-time data (JSON format) to the cloud via the HTTPS protocol; the cloud returns the optimized model weights (Protobuf format).

[0129] Control flow: The cloud sends incremental learning instructions, and the BMC executes model updates.

[0130] The connection between the BMC and the cloud server forms the core path for the collaboration between edge computing and cloud intelligence. In terms of data flow, the BMC encrypts the hardware status, operation logs, and diagnostic results collected locally, and uploads them to the cloud platform in JSON format via the HTTPS protocol, providing a high-quality data source for remote monitoring and model training. At the same time, after the cloud completes model optimization, it will send the updated model weight file (usually in Protobuf format) to the BMC for upgrading the local inference engine. In terms of control flow, the cloud can guide the BMC to dynamically adjust local model behavior by issuing incremental learning instructions, configuring update strategies, and other methods, achieving closed-loop optimization and continuous improvement of intelligent decision-making capabilities.

[0131] BMC and CPLD:

[0132] Data flow: The BMC sends a 16-bit control instruction (such as 0xA10001) to the CPLD to trigger the power isolation operation.

[0133] Feedback signal: CPLD returns the operation status code (such as 0x00 for success).

[0134] Communication between the BMC and the CPLD (Complex Programmable Logic Device) primarily implements low-level hardware control and power management functions. In terms of data flow, the BMC sends 16-bit control instructions (such as 0xA10001) through dedicated interfaces (such as GPIO, I2C, or UART) to instruct the CPLD to perform specific operations, such as triggering power isolation or resetting a PCIe link. Regarding feedback signals, the CPLD returns a status code (such as 0x00 for success and 0x01 for failure) after completing the operation, allowing the BMC to determine the execution result and respond. This hardwired control method offers low latency and high reliability, making it suitable for rapid recovery in critical failure scenarios.

[0135] CPLD and PCIe devices:

[0136] Power control: Cut off the 3.3V / 12V power supply line through the MOSFET switch (IRF3205).

[0137] Bandwidth control: Limits the bandwidth of PCIe devices through the PCIe controller.

[0138] Signal monitoring: The CPLD monitors the power status signal (such as Power Good) of the PCIe device.

[0139] As the core hub of hardware control, the CPLD directly participates in the physical layer management of PCIe devices. In terms of power control, the CPLD drives MOSFET switches (such as the IRF3205) to cut off or restore the 3.3V and 12V power supply lines of the PCIe device, achieving refined power-off protection and energy-saving management. In terms of bandwidth control, the CPLD can work with the PCIe controller to dynamically adjust the link rate used by the device (for example, downgrading from Gen4 to Gen3) to address abnormal conditions such as hot plugging and overload. In terms of signal monitoring, the CPLD continuously monitors the power status signals (such as the Power Good signal) of the PCIe device. Once an anomaly is detected, it immediately notifies the BMC for intervention, thus establishing a complete closed-loop system for hardware health management.

[0140] Specifically, the training process is as follows: Data input: historical fault data and real-time collected data. Data cleaning and feature extraction: preprocessing module. Model training: LSTM network training and optimization. Weight update: incremental learning and cloud collaboration. The specific process is as follows:

[0141] Data Input: The training process relies on multi-source data, primarily consisting of two types of input data: historical fault data, used to construct the initial training set, encompasses equipment operation logs, historical fault records, and maintenance reports. Real-time data collection, which collaborates with the BMC and operating system to obtain current hardware status, performance indicators, and abnormal signals. This data includes time series characteristics, possessing strong dynamics and predictive value.

[0142] Data Cleaning and Feature Extraction: In the preprocessing module, the raw data is first filled with missing values, outliers removed, and normalized to ensure data quality. Subsequently, feature engineering is used to extract key time series features, such as sliding window mean, cumulative counts, and rates of change, to enhance the model's ability to detect system degradation trends. This stage outputs a structured feature vector, which serves as input to the LSTM network and provides a high-quality data foundation for subsequent modeling.

[0143] Model Training: LSTM Network Training and Optimization: Long Short-Term Memory (LSTM) networks are used to model time series data, capturing long-term dependencies between device operating states. Supervised learning methods are used during training, using normal and abnormal labels from historical data as target outputs to optimize network parameters and improve the model's fault identification accuracy. Hyperparameter tuning mechanisms, such as grid search or Bayesian optimization, are also introduced to further enhance model generalization.

[0144] Weight Update: Incremental Learning and Cloud Collaboration: To adapt to the continuous evolution of device status, the model supports an online incremental learning mechanism. The BMC regularly uploads newly collected data to the cloud, where the cloud platform retrains and optimizes model weights based on historical data. The optimized model is then sent to the edge BMC in Protobuf format to update the local inference engine. This mechanism enables closed-loop learning through cloud-edge collaboration, enabling the model to continuously evolve and dynamically adapt to complex scenarios such as device aging and environmental changes, improving prediction accuracy and system intelligence.

[0145] Furthermore, the data relationship between the modules is as follows:

[0146] Data input to data cleaning:

[0147] Data flow: Raw data (CSV format) is input into the cleaning module to perform missing value filling and normalization.

[0148] Logical association: The cleaned data is stored in a temporary cache area (such as a Redis database).

[0149] Data cleaning to feature extraction:

[0150] Feature engineering: Extract time series features (such as sliding window mean, cumulative error count) from the cleaned data.

[0151] Output format: Feature vector (dimension = 12) is input to the model training module.

[0152] Feature extraction to model training:

[0153] Training process: Use the TensorFlow Lite framework to update LSTM weights based on the Adam optimizer.

[0154] Model output: The trained model (.tflite file) is pushed to the BMC.

[0155] Model training and weight update:

[0156] Incremental learning: New data is uploaded to the cloud through the BMC, triggering model fine-tuning.

[0157] Version control: The cloud retains historical model versions and supports BMC rollback to stable versions.

[0158] Specifically, the data relationship between each part is as follows:

[0159] Data input to data cleaning:

[0160] Raw data (such as device operation logs and hardware monitoring metrics) is imported into the system in CSV format, serving as the starting point for the entire process. The data cleaning module preprocesses the raw data, including missing value filling (e.g., sliding window averaging), outlier filtering, and normalization (e.g., Z-score standardization) to improve data quality and eliminate dimensional discrepancies. The cleaned structured data is temporarily stored in a temporary cache, such as a Redis database, providing an efficient data access interface for the subsequent feature extraction module.

[0161] Data cleaning to feature extraction:

[0162] The feature extraction module extracts representative high-order time series features from the cleaned data to enhance the model's sensitivity and ability to identify system state changes. Extracted features include, but are not limited to, one-hour moving averages, cumulative error counts, rates of change, and moving standard deviations. These features are combined and reduced in dimension to form fixed-dimensional feature vectors (12 in this example). These are then output in a standardized format to the model training module, serving as input to the LSTM network.

[0163] Feature extraction to model training:

[0164] In the model training module, the TensorFlow Lite framework is used to build an LSTM neural network model, with the Adam optimizer used for parameter updates. This supervised learning approach is used to model time series data. During training, historical fault labels are used as supervisory signals to continuously optimize the model's ability to distinguish between normal and abnormal patterns. After training is complete, a lightweight model file (.tflite format) is generated and pushed to the BMC through the model management module for edge inference execution.

[0165] Model training and weight update:

[0166] To achieve continuous model optimization and adaptive adjustment, the system supports a collaborative incremental learning mechanism with the cloud. The BMC regularly uploads newly collected operational data to the cloud, triggering fine-tuning of the cloud-based model. The training platform then introduces new samples for local retraining based on the existing model, improving the model's generalization and timeliness. Updated model versions are centrally managed in the cloud and are identified with a version number, supporting a rollback mechanism for historical versions. Based on the system status, the BMC can choose to load the current optimal model or roll back to a stable version, thereby ensuring inference accuracy and system stability.

[0167] In summary, the present invention reduces the probability of downtime through dynamic bandwidth degradation and hardware isolation. The health score drives the bandwidth allocation strategy and improves the average utilization of PCIe links.

[0168] This paper proposes an intelligent monitoring and processing method that uses machine learning to predict PCIe device failures in real time and links them to hardware isolation. This method proactively identifies potential failures by analyzing PCIe link parameter trends. Bandwidth allocation is adjusted based on device health to maximize link utilization. Faulty devices are directly isolated using hardware control modules (such as CPLDs) to avoid system downtime.

[0169] In order to implement the above embodiment, Figure 3 As shown, this embodiment also provides a PCIe device fault monitoring and dynamic processing device 10 based on intelligent prediction, including:

[0170] The data acquisition module 100 obtains device operating parameters from the baseboard management controller BMC, the operating system, and the PCIe device registers in response to a preset acquisition cycle;

[0171] A preprocessing module 200 performs preprocessing on the device operating parameters and extracts time series features to generate a standardized feature vector;

[0172] The prediction module 300 performs model inference based on the standardized feature vector using a lightweight long short-term memory network (LSTM) model deployed on the BMC, and outputs the probability of device failure within a preset time window in the future.

[0173] Health scoring module 400 calculates the equipment health score based on the failure probability and equipment operating parameters combined with the dynamic weight coefficient;

[0174] a response control module 500 that triggers an alarm, bandwidth degradation, or hardware isolation operation based on a preset hierarchical response strategy in response to the health score;

[0175] The hardware isolation module 600 sends a control instruction to a complex programmable logic device (CPLD) via an I²C bus in response to the hardware isolation operation instruction. After the CPLD interprets the control instruction, it controls the power switch module to disconnect the power supply line of the target device. The control instruction includes an operation code, a target PCIe slot address, and a reserved field.

[0176] The model update module 700, in response to the model update request, uploads the encrypted real-time data to the cloud server. After the cloud server performs model fine-tuning, it transmits the updated model weights back to the BMC via the HTTPS protocol. The BMC updates the local model or rolls back to the stable version according to the model version control strategy.

[0177] The intelligent prediction-based PCIe device fault monitoring and dynamic processing apparatus according to an embodiment of the present invention proactively identifies potential faults by analyzing PCIe link parameter trends. It adjusts bandwidth allocation based on device health to maximize link utilization and directly isolates faulty devices through hardware control modules (such as CPLDs) to prevent system downtime.

[0178] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0179] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Claims

1. A PCIe device fault monitoring and dynamic processing method based on intelligent prediction, characterized in that: The following steps are involved: In response to a preset acquisition cycle, device operating parameters are obtained from a baseboard management controller (BMC), an operating system, and PCIe device registers; Preprocess the equipment operating parameters and extract the time series features to generate standardized feature vectors; Based on the standardized feature vector, a lightweight long short-term memory network (LSTM) model deployed on the BMC is used for model inference to output the probability of device failure within a preset time window in the future. Calculate the equipment health score based on the failure probability and equipment operating parameters combined with the dynamic weight coefficient; In response to the health score, triggering an alarm, bandwidth degradation, or hardware isolation operation based on a preset hierarchical response strategy; In response to the hardware isolation operation instruction, a control instruction is sent to a complex programmable logic device (CPLD) via the I²C bus. After the CPLD interprets the control instruction, it controls the power switch module to disconnect the power supply line of the target device. In response to the model update request, the encrypted real-time data is uploaded to the cloud server. After the cloud server performs model fine-tuning, the updated model weights are transmitted back to the BMC via the HTTPS protocol. The BMC updates the local model or rolls back to the stable version according to the model version control strategy.

2. The method according to claim 1, wherein Obtain temperature and voltage data from the BMC through the IPMI protocol, call operating system commands through the SSH protocol to obtain PCIe device register information, and read the AER register through the PCIe configuration space to capture error details.

3. The method according to claim 1, wherein The sliding window mean method is used to fill missing values, and Z-score normalization is used to eliminate parameter dimensional differences. Timing features are extracted, including the sliding mean of the bit error rate, the cumulative count of CRC errors, and temperature deviation. Among them, the Z-score represents the new value of each original data point after normalization.

4. The method according to claim 1, wherein The LSTM model has a hidden layer dimension of 64 and a time step of 10. The model size is compressed to less than 800KB, adapted to the BMC's ARM Cortex-A53 processor, and the inference frequency is once every 5 minutes.

5. The method according to claim 1, wherein The device health score is calculated based on the normalized bit error rate, the cumulative number of CRC errors in the past hour, the deviation of the device temperature from the rated value, and a dynamic weight coefficient.

6. The method according to claim 1, wherein Depending on the health score range, different levels of response strategies are triggered, including: When the health score is greater than or equal to 80, only an alarm message is generated; When the health score is less than 80 and greater than or equal to 30, the bandwidth degradation operation is triggered; When the health score is less than 30, hardware isolation is triggered.

7. The method according to claim 6, wherein The bandwidth degradation operation adjusts the link width through the configuration register of the PCIe switch and checks the availability of the backup link before degradation. If it is unavailable, the degradation is skipped and isolation is triggered directly.

8. The method according to claim 1, wherein The control instruction is in a 16-bit format, including an operation code, a target PCIe slot address, and a reserved field, wherein the operation code 0xA1 indicates cutting off the power supply, the address field is the physical address of the target slot, and the reserved field defaults to 0x00.

9. The method according to claim 8, wherein The control instruction is appended with a CRC-8 check code. The CPLD verifies the data integrity after receiving the instruction. If no ACK signal is received, the BMC resends the instruction after 100ms and retries up to 3 times.

10. A PCIe device fault monitoring and dynamic processing device based on intelligent prediction, characterized in that: include: The data acquisition module obtains device operating parameters from the baseboard management controller (BMC), the operating system, and the PCIe device registers in response to a preset acquisition cycle; A preprocessing module, which preprocesses the equipment operating parameters and extracts time series features to generate a standardized feature vector; The prediction module performs model inference based on the standardized feature vector using a lightweight long short-term memory network (LSTM) model deployed on the BMC, and outputs the probability of device failure within a preset time window in the future. The health scoring module calculates the equipment health score based on the failure probability and equipment operating parameters combined with the dynamic weight coefficient; a response control module, responsive to the health score and based on a preset hierarchical response strategy, triggering an alarm, bandwidth degradation, or hardware isolation operation; The hardware isolation module sends a control instruction to a complex programmable logic device (CPLD) via an I²C bus in response to the hardware isolation operation instruction. After the CPLD interprets the control instruction, it controls the power switch module to disconnect the power supply line of the target device. The control instruction includes an operation code, a target PCIe slot address, and a reserved field. The model update module responds to model update requests by uploading encrypted real-time data to the cloud server. After the cloud server performs model fine-tuning, it transmits the updated model weights back to the BMC via the HTTPS protocol. The BMC updates the local model or rolls back to the stable version according to the model version control strategy.

Images