Graphics processor risk processing method and device, equipment and medium

Abstract

The embodiment of the invention provides a graphics processor risk processing method and device, equipment and a medium. The method comprises the steps that running state information and environment information of at least one graphics processor are acquired; obtaining a topological graph corresponding to at least one graphics processor; nodes of the topological graph correspond to at least one graphics processor, and edges correspond to a physical connection relationship between the at least one graphics processor; determining a current weight value of an edge in the topological graph according to the running state information and the environment information of the at least one graphics processor; according to the topological graph and the current weight value of the edge in the topological graph, predicting at least one target graphic processor with a fault risk; the risk processing operation for at least one target graphics processor is determined and executed, and the problem that in the prior art, passive response can only be achieved after a fault occurs, and loss caused by a cascade fault is difficult to effectively avoid is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of hardware risk management, and in particular to a method, apparatus, device, and medium for managing graphics processor risks. Background Technology

[0002] During the operation of a graphics processing unit (GPU) cluster, due to the close physical connections between GPUs, a failure in a single GPU can propagate to other GPUs through these connections, triggering a cascading failure and causing a complete service interruption across the cluster. Current technologies for monitoring GPU failures are mostly limited to single-point status detection, failing to consider the impact of physical connections between GPUs on failure propagation. They cannot predict the failure propagation path or the potential range of affected GPUs, resulting in a reactive approach after a failure occurs, making it difficult to effectively prevent losses caused by cascading failures. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a graphics processor risk handling method, apparatus, device, and medium that overcomes or at least partially solves the above problems.

[0004] To address the aforementioned problems, this invention discloses a method for handling risks associated with graphics processors, the method comprising:

[0005] Obtain the running status and environment information of at least one graphics processor;

[0006] Obtain the topology graph corresponding to the at least one graphics processor; the nodes of the topology graph correspond to the at least one graphics processor, and the edges correspond to the physical connection relationships between the at least one graphics processor;

[0007] Based on the running status information of the at least one graphics processor and the environmental information, determine the current weight value of the edge in the topology graph;

[0008] Based on the topology graph and the current weight values ​​of the edges in the topology graph, predict at least one target graphics processor with a risk of failure;

[0009] Identify and execute risk handling operations for the at least one target graphics processor.

[0010] Optionally, the determination and execution of risk processing operations for the at least one target graphics processor includes...

[0011] Based on the operating status information and environmental information of the at least one target graphics processor, the predicted short-term risk parameters and predicted long-term loss parameters of the at least one target graphics processor are determined.

[0012] Based on the predicted short-term risk parameters and the predicted long-term loss parameters, the risk processing operations corresponding to each of the at least one target graphics processors are determined and executed.

[0013] Optionally, determining the current weight value of an edge in the topology graph based on the operating status information of the at least one graphics processor and the environment information includes:

[0014] Based on the aforementioned operational status information, the data link weights are determined;

[0015] Based on the environmental information, determine the weight of the indirect coupling link;

[0016] The current weight value of the edge in the topology graph is determined based on the data link weight and the indirect coupling link weight.

[0017] Optionally, the operational status information includes bit error rate and utilization rate, and determining the data link weight based on the operational status information includes:

[0018] Based on the bit error rate, a bit error rate influence coefficient is determined in a preset first mapping table; the preset first mapping table is a mapping table between the bit error rate and the bit error rate influence coefficient.

[0019] Based on the utilization rate, a utilization rate influence coefficient is determined in a preset second mapping table; the preset second mapping table is a mapping table between the utilization rate and the utilization rate influence coefficient.

[0020] The data link weight is determined based on the bit error rate, bit error rate impact coefficient, utilization rate, and utilization rate impact coefficient of the at least one graphics processor.

[0021] Optionally, determining the data link weight based on the bit error rate, the bit error rate impact coefficient, the utilization rate, and the utilization rate impact coefficient of the at least one graphics processor includes:

[0022] Determine a first relative offset and a second relative offset; the first relative offset is the relative offset of the bit error rate with respect to a preset bit error rate threshold, and the second relative offset is the relative offset of the utilization rate with respect to a preset utilization rate threshold;

[0023] A first weight and a second weight are determined based on the ratio of the first relative offset to the second relative offset; the first weight is used to represent the importance of the bit error rate to the risk assessment of the graphics processor, and the second weight is used to represent the importance of the utilization rate to the risk assessment of the graphics processor.

[0024] The data link weight is determined based on the bit error rate of the at least one graphics processor, the bit error rate influence coefficient, the first weight, the utilization rate, the utilization rate influence coefficient, and the second weight.

[0025] Optionally, the environmental information includes ambient temperature, the indirect coupling link weight includes thermal coupling link weight, and determining the indirect coupling link weight based on the environmental information includes:

[0026] Determine the third offset; the third offset is the relative offset of the temperature information with respect to the preset temperature reference value;

[0027] The product of the third offset and the preset temperature weight adjustment factor is used as the weight of the thermal coupling link.

[0028] Optionally, the environmental information includes the power supply voltage, the indirect coupling link weight includes the power supply link weight, and determining the indirect coupling link weight based on the environmental information includes:

[0029] Determine the fourth offset; the fourth offset is the relative offset of the power supply voltage information with respect to the preset voltage reference value;

[0030] The product of the fourth offset and the preset power supply link weight adjustment factor is used as the power supply link weight.

[0031] Optionally, the operating status information includes operating status information within a preset time window, and determining the predicted short-term risk parameters of the at least one target graphics processor based on the operating status information of the at least one target graphics processor and the environmental information includes:

[0032] Based on the operating status information and environmental information within the preset time window, a predicted short-term risk parameter is determined. The predicted short-term risk parameter is used to represent the probability of failure of the target graphics processor in the near future.

[0033] Optionally, the predicted long-term wear parameters include remaining useful life and / or aging failure risk ratio, wherein the aging failure risk ratio is the ratio of the current failure risk of the target graphics processor to the failure risk of the graphics processor under baseline conditions.

[0034] Optionally, the method further includes:

[0035] If the detected operating status information is greater than or equal to a preset operating status information threshold and / or the environmental information is greater than or equal to a preset environmental information threshold, the acquisition frequency of the operating status information and environmental information of the at least one graphics processor is increased from a preset first information acquisition frequency to a preset second information acquisition frequency.

[0036] Optionally, predicting at least one target graphics processor with a failure risk based on the topology graph and the current weight values ​​of the edges in the topology graph includes:

[0037] Determine the location of the faulty node in the topology graph; the location of the faulty node is the node location in the topology graph corresponding to the graphics processor that has malfunctioned.

[0038] The fault impact intensity of the at least one graphics processor is determined based on a preset impact intensity benchmark value, the location of the fault node, the topology graph, and the current weight values ​​of the edges in the topology graph; the fault impact intensity is a quantified value of the degree of fault impact transmitted layer by layer from the location of the fault node through the edges of the topology graph.

[0039] In the at least one graphics processor, the graphics processor whose fault impact intensity is greater than or equal to a preset fault impact intensity threshold is selected as the target graphics processor.

[0040] Optionally, determining and executing the risk processing operation corresponding to each of the at least one target graphics processor based on the predicted short-term risk parameters and the predicted long-term loss parameters includes:

[0041] For each of the at least one target graphics processors, if the predicted short-term risk parameter is greater than or equal to a preset short-term risk threshold and the predicted long-term loss parameter is greater than or equal to a preset long-term risk threshold, a data migration operation is performed on the corresponding target graphics processor.

[0042] On the other hand, embodiments of the present invention also provide a graphics processor risk processing device, the device comprising:

[0043] The information acquisition module is used to acquire the operating status information and environmental information of at least one graphics processor;

[0044] A topology graph acquisition module is used to acquire a topology graph corresponding to the at least one graphics processor; the nodes of the topology graph correspond to the at least one graphics processor, and the edges correspond to the physical connection relationships between the at least one graphics processor.

[0045] The current weight value acquisition module is used to determine the current weight value of the edge in the topology graph based on the running status information of the at least one graphics processor and the environmental information.

[0046] A graphics processor determination module is used to predict at least one target graphics processor with a risk of failure based on the topology graph and the current weight values ​​of the edges in the topology graph.

[0047] A graphics processor processing module is used to determine and execute risk processing operations for the at least one target graphics processor.

[0048] Optionally, the graphics processor processing module includes

[0049] The parameter determination submodule is used to determine the predicted short-term risk parameters and predicted long-term loss parameters of the at least one target graphics processor based on the operating status information and the environmental information of the at least one target graphics processor.

[0050] The operation determination submodule is used to determine and execute the risk processing operation corresponding to each of the at least one target graphics processors based on the predicted short-term risk parameters and the predicted long-term loss parameters.

[0051] Optionally, the current weight value acquisition module includes:

[0052] The data link weight determination submodule is used to determine the data link weight based on the running status information.

[0053] The indirect coupling link weight determination submodule is used to determine the indirect coupling link weight based on the environmental information.

[0054] The fusion weight determination submodule is used to determine the current weight value of the edge in the topology graph based on the data link weight and the indirect coupling link weight.

[0055] Optionally, the operational status information includes bit error rate and utilization rate, and the data link weight determination submodule includes:

[0056] The bit error rate impact coefficient determination unit is used to determine the bit error rate impact coefficient in a preset first mapping table based on the bit error rate; the preset first mapping table is a mapping table between the bit error rate and the bit error rate impact coefficient.

[0057] The utilization rate influence coefficient determination unit is used to determine the utilization rate influence coefficient in a preset second mapping table based on the utilization rate; the preset second mapping table is a mapping table between the utilization rate and the utilization rate influence coefficient.

[0058] The first data link weight determination unit is used to determine the data link weight based on the bit error rate, bit error rate influence coefficient, utilization rate, and utilization rate influence coefficient of the at least one graphics processor.

[0059] Optionally, the first data link weight comprehensive determination unit includes:

[0060] An offset determination subunit is used to determine a first relative offset and a second relative offset; the first relative offset is the relative offset of the bit error rate with respect to a preset bit error rate threshold, and the second relative offset is the relative offset of the utilization rate with respect to a preset utilization rate threshold.

[0061] The weighting determination subunit is used to determine a first weight and a second weight based on the ratio of the first relative offset and the second relative offset; the first weight is used to represent the importance of the bit error rate to the risk assessment of the graphics processor, and the second weight is used to represent the importance of the utilization rate to the risk assessment of the graphics processor.

[0062] The second data link weight comprehensive determination subunit is used to determine the data link weight based on the bit error rate of the at least one graphics processor, the bit error rate influence coefficient, the first weight, the utilization rate, the utilization rate influence coefficient, and the second weight.

[0063] Optionally, the environmental information includes ambient temperature, the indirect coupling link weight includes thermal coupling link weight, and the indirect coupling link weight determination submodule includes:

[0064] The third offset determination unit is used to determine the third offset; the third offset is the relative offset of the temperature information with respect to the preset temperature reference value;

[0065] The thermal coupling link weight determination unit is used to take the product of the third offset and the preset temperature weight adjustment factor as the thermal coupling link weight.

[0066] Optionally, the environmental information includes the power supply voltage, the indirect coupling link weight includes the power supply link weight, and the indirect coupling link weight determination submodule includes:

[0067] The fourth offset determination unit is used to determine the fourth offset; the fourth offset is the relative offset of the power supply voltage information with respect to the preset voltage reference value;

[0068] The power supply link weight determination unit is used to take the product of the fourth offset and the preset power supply link weight adjustment factor as the power supply link weight.

[0069] Optionally, the running status information includes running status information within a preset time window, and the parameter determination submodule includes:

[0070] The short-term risk parameter determination unit is used to determine the predicted short-term risk parameters based on the operating status information and the environmental information within the preset time window. The predicted short-term risk parameters are used to represent the probability of failure of the target graphics processor in the near future.

[0071] Optionally, the device further includes:

[0072] The frequency adjustment submodule is used to increase the acquisition frequency of the operating status information and environmental information of the at least one graphics processor from a preset first information acquisition frequency to a preset second information acquisition frequency if the detected operating status information is greater than or equal to a preset operating status information threshold and / or the environmental information is greater than or equal to a preset environmental information threshold.

[0073] Optionally, the graphics processor determining module includes:

[0074] The fault node determination submodule is used to determine the location of the corresponding fault node in the topology graph; the location of the fault node is the node location of the graphics processor that has malfunctioned in the topology graph.

[0075] The fault impact intensity submodule is used to determine the fault impact intensity of the at least one graphics processor based on a preset impact intensity benchmark value, the location of the fault node, the topology graph, and the current weight values ​​of the edges in the topology graph; the fault impact intensity is a quantified value of the degree of fault impact transmitted layer by layer from the location of the fault node through the edges of the topology graph;

[0076] A target graphics processor determination submodule is used to identify, among the at least one graphics processor, the graphics processor whose fault impact intensity is greater than or equal to a preset fault impact intensity threshold as the target graphics processor.

[0077] Optionally, the operation determination submodule includes:

[0078] The data migration unit is configured to perform a data migration operation on each of the at least one target graphics processors when the predicted short-term risk parameter is greater than or equal to a preset short-term risk threshold and the predicted long-term loss parameter is greater than or equal to a preset long-term risk threshold.

[0079] Accordingly, this invention discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various steps of the above-described embodiment of a graphics processor risk handling method.

[0080] Accordingly, this invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the various steps of the above-described embodiment of a graphics processor risk handling method.

[0081] The embodiments of this invention have the following advantages: By acquiring a topology graph corresponding to at least one graphics processor (GPU), where nodes in the topology graph correspond to at least one GPU and edges correspond to the physical connections between at least one GPU; based on the operating status and environmental information of at least one GPU, the current weight values ​​of the edges in the topology graph are determined; and based on the topology graph and the current weight values ​​of the edges in the topology graph, at least one target GPU with a failure risk is predicted. This achieves the prediction of the fault propagation path and the range of potentially affected GPUs based on physical connections and quantified fault propagation risk, effectively addressing the problem that existing technologies cannot predict the fault propagation path and the range of potentially affected GPUs. By determining and executing risk handling operations for at least one target GPU, proactive intervention is achieved for GPUs with a failure risk, effectively alleviating the problem in existing technologies where only passive responses are possible after a fault occurs, making it difficult to effectively avoid losses caused by cascading failures. Attached Figure Description

[0082] Figure 1 This is a flowchart illustrating the steps of an embodiment of a graphics processor risk handling method according to the present invention;

[0083] Figure 2 This is a structural block diagram of an embodiment of a graphics processor risk processing device according to the present invention;

[0084] Figure 3 This is a structural block diagram of a graphics processor-based risk processing electronic device according to the present invention;

[0085] Figure 4 This is a structural block diagram of a computer-readable storage medium for processing graphics processor risks according to the present invention. Detailed Implementation

[0086] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0087] One of the core concepts of this invention is to construct a topology graph with dynamically weighted edges by collecting the operating status and environmental information of the graphics processor, predict the target graphics processor with fault risk based on the topology graph and edge weights, and finally perform risk handling operations on the target GPU.

[0088] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a graphics processor risk handling method according to the present invention, which may specifically include the following steps:

[0089] Step 101: Obtain the running status information and environment information of at least one graphics processor.

[0090] A Graphics Processing Unit (GPU) is a specialized processor with highly parallel computing capabilities, primarily used for graphics rendering and large-scale parallel computing. GPUs include discrete graphics cards, integrated GPUs, and data center-grade GPUs. Their number is dynamically configured based on cluster size. Operating status information is a set of hardware performance and health data characterizing the current working state of the GPU, specifically including: Bit Error Rate (BER), the ratio of the number of erroneous bits transmitted per unit time to the total number of bits; link utilization, the percentage of actual bandwidth of high-speed interconnect links (such as NVLink and PCI Express) relative to their theoretical bandwidth; Streaming Multiprocessor (SM) utilization, the proportion of GPU core computing units occupied; Graphics Double Data Rate (GDDR) utilization, the percentage of actual used GDDR memory capacity relative to the total capacity; and the real-time operating frequency of the GPU computing cores and GDDR memory (e.g., 1.8GHz). Environmental information refers to external physical environment parameters that characterize the stable operation of a graphics processing unit (GPU). These parameters can include GPU core temperature, memory temperature, room temperature, core power supply voltage, power supply current fluctuations (e.g., ±5A), chassis air pressure, and dust content. This data can be acquired through at least one of the following methods: reading real-time data from the GPU's integrated sensors via I2C (Inter-Integrated Circuit) or SMBus (System Management Bus) interfaces; obtaining operational status information through the application programming interface provided by the GPU driver; or acquiring basic environmental information when the GPU operating system is offline via the Baseboard Management Controller (BMC) or Intelligent Platform Management Interface (IPMI) protocol.

[0091] The execution entity of this invention can be a server, a dedicated controller, etc.; the application scenario can be a high-performance computing cluster environment in which multiple graphics processors work together, including: artificial intelligence training / inference clusters, scientific computing platforms, and graphics rendering clusters, etc.

[0092] In one embodiment, the method may further include the following:

[0093] If the detected operating status information is greater than or equal to a preset operating status information threshold and / or the environmental information is greater than or equal to a preset environmental information threshold, the acquisition frequency of the operating status information and environmental information of the at least one graphics processor is increased from a preset first information acquisition frequency to a preset second information acquisition frequency.

[0094] The data acquisition frequency adjustment operation is performed when at least one of the following conditions is detected: at least one parameter in the acquired operational status information is greater than or equal to a preset operational status information threshold; or at least one parameter in the acquired environmental information is greater than or equal to a preset environmental information threshold. The preset operational status information threshold and the preset environmental information threshold are pre-stored hardware configuration parameters that can be flexibly adjusted according to the GPU model, application scenario, and business needs.

[0095] The preset first information acquisition frequency refers to the basic acquisition frequency under normal GPU operation, such as 1 time / second. The preset second information acquisition frequency refers to the high-frequency acquisition frequency when parameters exceed the limit, which is N times the preset first information acquisition frequency (N is greater than or equal to 2), such as 10 times / second. When the trigger condition is met, the interrupt period of the internal timer is switched from the first frequency period (e.g., 1 second) to the second frequency period (e.g., 0.1 seconds) through hardware logic circuitry. When the running status information and environmental information are detected as being below the corresponding threshold for M consecutive times (M can be set according to business needs), the acquisition frequency is automatically restored from the preset second information acquisition frequency to the preset first information acquisition frequency.

[0096] When GPU parameters are abnormal, high-frequency data acquisition captures transient risks, avoiding the omission of critical risk signals due to low-frequency sampling. Under normal conditions, low-frequency acquisition is maintained to reduce the GPU's communication bandwidth usage and balance real-time performance with system overhead. At the same time, high-frequency data can provide more granular parameter change trends, enabling earlier and more accurate prediction of fault propagation paths.

[0097] Step 102: Obtain the topology graph corresponding to the at least one graphics processor; the nodes of the topology graph correspond to the at least one graphics processor, and the edges correspond to the physical connection relationships between the at least one graphics processor.

[0098] A topology graph is a structured graphical model that abstractly represents multiple objects and their connections within a system using nodes and edges. Nodes represent independent objects in the system, and edges represent connections between objects. In this embodiment, each node in the topology graph corresponds one-to-one with a graphics processor (GPU), uniquely identified by hardware identifiers such as PCIe device addresses and GPU serial numbers. Nodes may contain basic GPU information for association with subsequently collected operational status information. Edges represent the physical connection between two GPU nodes, i.e., the association formed through data links or indirect coupling links. Data links can be NVLink links, PCIe links, or Ethernet links; indirect coupling links can be heat conduction paths sharing a heat dissipation module or power supply links in the same power supply circuit. Edge attributes may include connection type, link parameters, hardware identifiers, etc. Methods for constructing the topology graph include, but are not limited to: scanning and identifying the physical connections between GPUs through the GPU driver interface or PCIe bus enumeration protocol; reading a pre-stored topology description file containing the correspondence and attribute information of nodes and edges; and obtaining the connection status between nodes in real time through a link layer discovery protocol supported by the GPU.

[0099] Step 103: Determine the current weight value of the edge in the topology graph based on the running status information of the at least one graphics processor and the environment information.

[0100] Based on the operating status information of at least one graphics processor and the environmental information, the current weight value of the edges in the topology graph is determined. The current weight value is a parameter dynamically generated based on the physical connection relationships corresponding to the edges in the topology graph, combined with the operating status information of the graphics processor and the environmental information. It is used to quantify the fault propagation capability of the physical connection relationships of the graphics processor. Its core is to adjust the weight values ​​through real-time status data, so that the weight values ​​can reflect the possibility of the physical link propagating the impact of faults under the current operating conditions; the higher the value, the higher the risk of the link propagating the impact of faults. The current weight value is an attribute parameter of the edges in the topology graph, meaning that each edge corresponds to an independent current weight value. This binding relationship ensures that changes in weight values ​​correspond one-to-one with physical links; furthermore, subsequent fault propagation prediction can be based on the node-edge structure of the topology graph, performing path deduction along edges with non-zero weight values.

[0101] In one embodiment, step 103 may include the following sub-steps:

[0102] Sub-step S11: Determine the data link weight based on the running status information;

[0103] Data link weight is the current weight value of the data transmission link in the topology diagram, used to characterize the risk of the data link propagating the impact of failures in the current operating state.

[0104] In one embodiment, the operating status information includes bit error rate and utilization rate, and sub-step S11 may include the following sub-steps:

[0105] Sub-step S111: Based on the bit error rate, determine the bit error rate influence coefficient in a preset first mapping table; the preset first mapping table is a mapping table between the bit error rate and the bit error rate influence coefficient.

[0106] The bit error rate (BER) of the data link is converted into a BER impact coefficient through a preset first mapping table, enabling a quantitative assessment of the link's transmission reliability. A higher BER results in a larger BER impact coefficient, indicating a higher risk of fault propagation due to decreased link reliability. The preset first mapping table is a pre-stored data structure containing a one-to-one correspondence between the BER and the BER impact coefficient (dimensionless, ranging from 0 to 1). The BER and the impact coefficient are positively correlated. Specifically, when the BER is within the link's designed reliability range, such as meeting the normal operating standards specified by the graphics processor manufacturer, the mapping table returns a small impact coefficient, such as 0.1, indicating stable link transmission and a low risk of fault propagation. When the BER exceeds the preset reliability threshold, such as when the link error rate is significantly higher than the design standard, leading to a decrease in data transmission integrity, the mapping table returns a large impact coefficient, such as 0.7, indicating a decrease in the link's own reliability and an increased likelihood of fault propagation through the link.

[0107] Sub-step S112: Based on the utilization rate, determine the utilization rate influence coefficient in a preset second mapping table; the preset second mapping table is a mapping table between the utilization rate and the utilization rate influence coefficient.

[0108] A pre-defined second mapping table is used to convert data link utilization into a utilization impact coefficient, quantifying the impact of link load on fault propagation. Higher utilization (a larger proportion of actual link traffic to theoretical bandwidth) results in a larger utilization impact coefficient, indicating a higher risk of fault propagation due to link resource saturation. The pre-defined second mapping table is a pre-stored data structure containing a one-to-one correspondence between utilization and utilization impact coefficients (dimensionless, ranging from 0 to 1). Utilization and impact coefficients are positively correlated. Specifically, when utilization is low (less than or equal to 50%), the mapping table returns a smaller impact coefficient, such as 0.2, indicating sufficient redundancy to handle abnormal data and a low risk of fault propagation; when utilization is high (greater than 80%), the mapping table returns a larger impact coefficient, such as 0.8, indicating strained link resources and a higher risk of fault propagation to downstream nodes through saturated links.

[0109] Sub-step S113: Determine the data link weight based on the bit error rate, bit error rate influence coefficient, utilization rate, and utilization rate influence coefficient of the at least one graphics processor.

[0110] The current weight value of the data link in the topology diagram is obtained by integrating the bit error rate, bit error rate impact coefficient, utilization rate, and utilization rate impact coefficient. The data link weight can be implemented through a preset algorithm, with the core formula as follows: Data Link Weight = Bit Error Rate × Bit Error Rate Impact Coefficient + Utilization Rate × Utilization Rate Impact Coefficient. The data link weight comprehensively reflects the transmission error risk and load saturation risk of the link. By combining link reliability and load status, it avoids the one-sidedness of single-parameter evaluation and comprehensively reflects the fault propagation risk of the data link. Through a mapping table and fusion algorithm, the weight value is dynamically adjusted according to the real-time status of the link, ensuring that the risk assessment is consistent with the actual hardware operating conditions.

[0111] In one embodiment, sub-step S113 may include the following sub-step:

[0112] Sub-step S1131: Determine a first relative offset and a second relative offset; the first relative offset is the relative offset of the bit error rate relative to a preset bit error rate threshold, and the second relative offset is the relative offset of the utilization rate relative to a preset utilization rate threshold.

[0113] By calculating the offsets of the bit error rate (BER) relative to a preset BER threshold and the utilization rate relative to a preset utilization rate threshold, the degree to which the current parameters deviate from the normal range is quantified, providing a basis for subsequent dynamic adjustment of weight allocation. The preset BER threshold characterizes the maximum permissible BER when the data link is operating normally. The formula for calculating the first relative offset is: (current BER - preset BER threshold) / preset BER threshold. If the current BER is less than or equal to the threshold, the offset is 0, indicating no offset; if the current BER is greater than the threshold, the offset is positive, indicating a proportion exceeding the normal range. The preset utilization rate threshold characterizes the maximum permissible utilization rate for stable data link operation. The formula for calculating the second relative offset is: (current utilization rate - preset utilization rate threshold) / preset utilization rate threshold. If the current utilization rate is less than or equal to the threshold, the offset is 0; if the current utilization rate is greater than the threshold, the offset is positive.

[0114] Sub-step S1132: Determine a first weight and a second weight based on the ratio of the first relative offset to the second relative offset; the first weight is used to represent the importance of the bit error rate to the risk assessment of the graphics processor, and the second weight is used to represent the importance of the utilization rate to the risk assessment of the graphics processor.

[0115] Based on the ratio of the first relative offset to the second relative offset, the contribution ratios of bit error rate and utilization rate in the data link weight calculation are dynamically adjusted to achieve the effect that parameters with more severe deviations have a greater impact on the weight. For example, if the ratio of the first relative offset to the second relative offset is 1:2, then the first weight is 1 / (1+2), the second weight is 2 / (1+2), and so on for other ratios.

[0116] Sub-step S1133: Determine the data link weight based on the bit error rate of the at least one graphics processor, the bit error rate influence coefficient, the first weight, the utilization rate, the utilization rate influence coefficient, and the second weight.

[0117] The final data link weight is obtained by weighting and summing the bit error rate, bit error rate impact coefficient, first weight, utilization rate, utilization rate impact coefficient, and second weight. Data link weight = (bit error rate × bit error rate impact coefficient × first weight) + (utilization rate × utilization rate impact coefficient × second weight). The weight allocation is dynamically adjusted using an offset ratio, ensuring that the data link weight more accurately reflects the most severe risk factors and avoids secondary factors masking major risks due to fixed weight allocation.

[0118] Sub-step S12: Determine the weight of the indirect coupling link based on the environmental information;

[0119] The weights of indirect coupling links in the topology graph are calculated based on environmental information. Indirect coupling links refer to physical connections between graphics processors (GPUs) formed through non-data transmission methods, such as heat conduction paths and shared power supply loops. Their weights characterize the risk level of environmental factors causing faults to propagate through these links. Anomalies in environmental information can lead to indirect coupling risks between GPUs. For example, when the ambient temperature rises, adjacent GPUs form thermal coupling through the chassis airflow, transferring heat from the high-temperature node to the low-temperature node, causing a cascading temperature increase and triggering a cascading failure. Similarly, when the power supply voltage fluctuates, GPUs sharing the same PDU may form power coupling due to voltage anomalies, and a power supply failure in one node can affect other nodes through the loop. Therefore, the greater the deviation of the environmental information from the normal range, the higher the risk of fault propagation through indirect coupling links, and the weights need to dynamically reflect this association.

[0120] In a first exemplary embodiment, the environmental information includes ambient temperature, the indirect coupling link weight includes thermal coupling link weight, and sub-step S12 may include the following:

[0121] Determine the third offset; the third offset is the relative offset of the temperature information with respect to the preset temperature reference value; the product of the third offset and the preset temperature weight adjustment factor is used as the weight of the thermal coupling link.

[0122] Environmental information specifically includes ambient temperature, such as the ambient temperature of the computer room and the internal temperature of the chassis. Indirect coupling link weights specifically include thermal coupling link weights. Thermal coupling links refer to the links between graphics processors in the topology diagram that exchange heat through heat conduction and convection due to their physical proximity. Thermal coupling link weights are used to quantify the risk of faults propagating through these thermally coupled links due to changes in ambient temperature. The third offset refers to the relative deviation of the ambient temperature from the preset temperature reference value, used to quantify the proportion of the ambient temperature deviating from the normal range. The preset temperature reference value refers to the baseline ambient temperature for stable operation of the graphics processor; the third offset = (current ambient temperature - preset temperature reference value) / preset temperature reference value. If the current ambient temperature is less than or equal to the preset temperature reference value, the third offset is 0; if the current ambient temperature is greater than the preset temperature reference value, the third offset is positive. For example, if the current temperature is 30℃ and the preset reference value is 25℃, then the third offset = (30-25) / 25 = 0.2, i.e., an offset of 20%. The third offset directly reflects the stress intensity of ambient temperature on the thermally coupled link: the larger the offset, the higher the ambient temperature, the more intense the heat exchange between graphics processors, and the higher the risk of fault propagation through heat conduction. The preset temperature weight adjustment factor is a proportional coefficient that converts the third offset into the thermally coupled link weight, preset according to the thermal sensitivity characteristics of the graphics processor. Its value must ensure that the thermally coupled link weight ultimately falls within the [0,1] interval. Thermally coupled link weight = third offset × preset temperature weight adjustment factor; if the calculation result is greater than 1, the thermally coupled link weight is set to 1.

[0123] By using a third offset and an adjustment factor, the absolute value of the ambient temperature is converted into a weight value that can be used for fault propagation calculation. When the ambient temperature rises, the third offset increases, and the weight of the thermally coupled link increases synchronously, ensuring that the weight value reflects the risk of thermally related fault propagation under the current temperature condition in real time.

[0124] In a second exemplary embodiment, the environmental information includes the power supply voltage, the indirect coupling link weight includes the power supply link weight, and sub-step S12 may include the following:

[0125] Determine the fourth offset; the fourth offset is the relative offset of the power supply voltage information with respect to the preset voltage reference value; the product of the fourth offset and the preset power supply link weight adjustment factor is used as the power supply link weight.

[0126] In this embodiment, environmental information may include power supply voltage, such as the input voltage of the power supply circuit where the graphics processor is located, and indirect coupling link weight specifically includes power supply link weight. The power supply link refers to the associated link between graphics processors formed in the topology diagram due to shared power supply circuits. The power supply link weight is used to quantify the risk of fault propagation through such power supply associated links caused by abnormal power supply voltage. The fourth offset refers to the relative deviation of the power supply voltage from the preset voltage reference value, used to quantify the proportion of the power supply voltage deviating from the normal range. The preset voltage reference value refers to the reference power supply voltage for stable operation of the graphics processor; the fourth offset = (current power supply voltage - preset voltage reference value) / preset voltage reference value; if the deviation between the current power supply voltage and the preset voltage reference value is within the allowable range (e.g., ±5%), the fourth offset is 0; if the deviation exceeds the allowable range, the fourth offset is positive. For example, if the current voltage is 13.2V and the preset reference value is 12V, then the fourth offset = (13.2-12) / 12 = 0.1, i.e., an offset of 10%. The preset power supply link weight adjustment factor is a proportional coefficient that converts the fourth offset into power supply link weights, preset according to the voltage sensitivity characteristics of the power supply circuit. Its value must ensure that the power supply link weight ultimately falls within the range of [0,1]. Power supply link weight = fourth offset × preset power supply link weight adjustment factor; if the calculation result is greater than 1, the power supply link weight is set to 1.

[0127] The fluctuation of the power supply voltage is converted into a calculable weight value through the fourth offset and adjustment factor. The greater the deviation of the power supply voltage from the reference value, the greater the fourth offset and the higher the weight of the power supply link, ensuring that the weight value reflects the risk of fault propagation under the current power supply status in real time. The weight of the indirect coupling link corresponding to other environmental information can be derived by referring to the logic of the two exemplary embodiments above, and will not be repeated here.

[0128] Sub-step S13: Determine the current weight value of the edge in the topology graph based on the data link weight and the indirect coupling link weight.

[0129] The current weight value of an edge in the topology graph is calculated by integrating the weights of data link links and indirect coupling links. The current weight value is the final risk quantification parameter for the edge in the topology graph, comprehensively reflecting the total risk of fault propagation caused by both data transmission links and indirect coupling links. The current weight value refers to the final weight value of the edge in the topology graph (range [0,1]), combining the weights of data link links and indirect coupling links; a higher value indicates a higher total risk of fault propagation for that edge. Data link weights and indirect coupling link weights reflect fault propagation risk from different dimensions. The current weight value needs to consider the contributions of both. By integrating the weights of data link links and indirect coupling links, a comprehensive quantification of the current weight value of the edge in the topology graph is achieved, solving the problem of the one-sidedness of risk assessment for a single link type.

[0130] For example, the current weight value can be calculated using a weighted summation as the fusion algorithm, or it can be adjusted according to the actual scenario, such as taking the maximum value, weighted product, etc. The specific formula for weighted summation is: Current weight value = (data link weight × α) + (indirect coupling link weight × β). Weight coefficients: α is the contribution ratio of the data link weight, β is the contribution ratio of the indirect coupling link weight, and α + β = 1; the values ​​of α and β can be adjusted according to the GPU type, business needs, usage scenarios, etc.; if there is only one link type on the topology graph edge, the current weight value is directly equal to the weight value of that link; the current weight value needs to be limited to the range [0,1]: if the weighted summation result is greater than 1, the current weight value = 1; comprehensively considering data transmission risk and environmental coupling risk, avoiding the one-sidedness of single link type evaluation, and fully reflecting the actual fault propagation risk of the topology edge; by adjusting the weight coefficients α and β, the risk characteristics of different types of topology edges can be adapted to improve the evaluation accuracy.

[0131] In one embodiment, the current weight value of an edge in the topology graph can also be calculated according to the following formula:

[0132] W' ij =W ij ×f(C ij )

[0133] Among them, C ij The parameter represents the real-time state parameter of the target graphics processor, and f(·) is the weight adjustment function W based on this parameter. i j represents the baseline weight, which can be the weight after the last adjustment or a preset fixed weight. All adjusted weights W' ij All values ​​are limited to the range [0.1, 1.0] to ensure model stability.

[0134] Step 104: Based on the topology graph and the current weight values ​​of the edges in the topology graph, predict at least one target graphics processor with a risk of failure.

[0135] Based on the topology graph and the current edge weights, a pre-defined algorithm predicts at least one target graphics processor (GPU) in the cluster that poses a risk of failure. A target GPU is defined as a GPU that may fail in the present or future, or may fail due to the failure of other nodes. The core logic uses the topology graph as its structural basis and the current edge weights as propagation probabilities. Through the transmission and accumulation of the impact intensity of the failure, GPUs with risks exceeding a pre-defined threshold are identified. Specifically, this involves first identifying nodes in the topology graph that initially pose a risk of failure, such as GPUs that have already malfunctioned or whose operating status / environment information is severely out of control. Starting from the source node, the impact intensity of the failure is propagated layer by layer along the edges of the topology graph. During propagation, the impact intensity is attenuated or amplified according to the current edge weight (the higher the weight, the higher the efficiency of the impact transmission). Finally, the accumulated impact intensity of each node is evaluated, and nodes exceeding the pre-defined risk threshold are identified as target GPUs with a risk of failure. The specific pre-defined algorithm can employ the influence diffusion algorithm from graph theory, abstracting the topology graph as a directed or undirected graph, with nodes representing GPUs and the current edge weights representing the probability of impact propagation. The initial impact strength of the fault source node is set to a baseline value, such as 1.0. Then, the total impact strength of each node is calculated layer by layer, using the formula: downstream node impact strength = upstream node impact strength × current edge weight. This total impact strength is the sum of the impact strengths transmitted by all upstream nodes through edges. If the total impact strength is greater than or equal to a preset risk threshold, such as 0.5, the node is marked as a target graphics processor. Alternatively, the preset algorithm can be designed based on the flood filling algorithm concept. For example, a faulty or high-risk graphics processor can be set as a seed node with an initial impact strength of 1.0. Starting from the seed node, all its adjacent nodes (those connected by edges in the topology graph) are traversed, and the impact strength of each adjacent node is calculated, specifically equal to the seed node's impact strength × the current edge weight. Adjacent nodes are then used as new seed nodes, and the calculation is repeated, propagating the impact strength layer by layer to its adjacent nodes until the impact strength decays below the preset threshold. After accumulating the impact strength of all propagation paths, nodes with a total strength greater than or equal to the preset risk threshold are identified as target graphics processors with a fault risk. This embodiment ensures that fault risk prediction conforms to the actual hardware layout of the graphics processor cluster by using the physical connection relationship of the topology graph and combining the current weight value of the edges, thus avoiding abstract predictions that are detached from physical links. Through the transmission and accumulation of influence intensity, the target graphics processor that may be affected can be identified before the fault occurs, providing a basis for decision-making for proactive maintenance or task migration.

[0136] In one embodiment, step 104 may include the following sub-steps:

[0137] Sub-step S21: Determine the location of the faulty node in the topology graph; the location of the faulty node is the node location of the graphics processor that has malfunctioned in the topology graph.

[0138] Locating the faulty node in the topology graph, i.e., the graphics processor that has failed, can be done by identifying the faulty graphics processor through hardware fault detection signals, and then matching its hardware identifier to the node position in the topology graph, which serves as the starting point for the propagation of the fault's impact intensity. The logic for locating high-risk faulty nodes is similar and will not be elaborated here.

[0139] Sub-step S22: Determine the fault impact intensity of the at least one graphics processor based on the preset impact intensity benchmark value, the location of the fault node, the topology graph, and the current weight values ​​of the edges in the topology graph; the fault impact intensity is a quantified value of the degree of fault impact transmitted layer by layer from the location of the fault node through the edges of the topology graph.

[0140] Starting from the location of the faulty node and using a preset influence strength benchmark value, such as 1.0, as the initial strength, the fault influence strength is propagated layer by layer along the edges of the topology graph to adjacent nodes. During the propagation process, the influence strength is attenuated or amplified according to the current weight value of the edge, ultimately yielding the fault influence strength for each graphics processor.

[0141] Sub-step S23: Among the at least one graphics processor, the graphics processor whose fault influence intensity is greater than or equal to a preset fault influence intensity threshold is selected as the target graphics processor.

[0142] Graphics processors whose fault impact intensity is greater than or equal to a preset fault impact intensity threshold are marked as target graphics processors. Specifically, by traversing all nodes in the topology graph, the fault impact intensity of each node is compared with the preset threshold, and graphics processors corresponding to nodes exceeding the threshold are selected as targets with fault risk. By locating faulty nodes and propagating the intensity layer by layer, the fault propagation path and boundary are clearly defined; objective judgment is achieved based on the threshold, avoiding subjective assessment bias; the current weight value of the edge is dynamically updated to ensure that the fault impact intensity calculation is consistent with the current hardware state, improving prediction accuracy.

[0143] In one embodiment, the environmental information may include the computer room temperature. The method may also individually label target graphics processors based on the computer room temperature. When the computer room temperature exceeds a threshold, the graphics processor in that environment is labeled as the target processor. The threshold can be calculated as follows:

[0144] T alert =T base +K e ×(E-E0)

[0145] Among them, T alertFor dynamic alarm thresholds, T base Based on the alarm threshold, K e Here, E is the environmental compensation factor, E0 is the current environmental parameter value, and E0 is the reference environmental parameter value. The advantage of setting the threshold in this way is that when the computer room encounters high temperatures, the marking threshold can be dynamically adjusted to avoid marking a large number of invalid target graphics processors under known harsh environmental conditions, ensuring that the marked target graphics processors are those that have actually caused the failure.

[0146] Step 105: Determine and execute risk handling operations for the at least one target graphics processor.

[0147] Based on the predicted risk characteristics of at least one target graphics processor, corresponding risk mitigation operations are determined and automatically executed to reduce the probability of failure and minimize its impact. Risk mitigation operations include, but are not limited to, task migration, preventative maintenance, and performance limiting; the specific operation type is dynamically adjusted according to the risk level of the target graphics processor.

[0148] In one embodiment, step 105 may include the following sub-steps:

[0149] Sub-step S31: Based on the operating status information and environmental information of the at least one target graphics processor, determine the predicted short-term risk parameters and predicted long-term loss parameters of the at least one target graphics processor.

[0150] For at least one target graphics processor (GPU), based on its operating status and environmental information, two types of core risk parameters are calculated: predicted short-term risk parameters and predicted long-term degradation parameters. Predicted short-term risk parameters quantify the probability (range [0,1]) of the target GPU failing within a short period, such as 30 minutes. Higher values ​​indicate a higher risk of failure in the short term. This can be achieved by: extracting short-term fault-sensitive features from time window data, such as temperature change rate and the number of times the bit error rate exceeds the limit; and then using a preset algorithm (such as logistic regression or LSTM network) to obtain the predicted short-term risk parameters. Predicted long-term degradation parameters quantify the risk of aging failure due to accumulated damage from long-term operation, such as remaining lifetime hours and risk ratio. Higher values ​​indicate more severe long-term aging and a shorter remaining reliable operating time. The degree of hardware aging can be calculated using long-term aging models (such as the Cox proportional hazards model or the Arrhenius model).

[0151] In one embodiment, the running status information includes running status information within a preset time window, and sub-step S31 may include the following sub-steps:

[0152] Sub-step S311: Based on the running status information and the environmental information within the preset time window, determine the predicted short-term risk parameter, which is used to represent the probability of the target graphics processor experiencing a failure in the near future.

[0153] By utilizing the operational status information and corresponding environmental information collected from the target graphics processor within a preset time window, a specific algorithm is used to calculate a predicted short-term risk parameter. This parameter quantifies the likelihood of the target graphics processor failing within a short period. By focusing on the operational status and environmental information within the preset time window, a dynamic and real-time assessment of the target graphics processor's short-term failure risk is achieved, ensuring that the predicted short-term risk parameter accurately reflects the immediate probability of failure under its current state.

[0154] In one embodiment, the predicted long-term wear parameters include remaining useful life and / or aging failure risk ratio, wherein the aging failure risk ratio is the ratio of the current failure risk of the target graphics processor to the failure risk of the graphics processor under a baseline state.

[0155] Predicting long-term degradation parameters specifically includes remaining useful life and / or aging failure risk ratio, used to quantify the long-term aging degree of the target graphics processor from two dimensions: absolute lifespan and relative risk. Remaining useful life refers to the expected length of time the target graphics processor can maintain reliable operation under its current cumulative damage state. It can be calculated based on the target graphics processor's long-term cumulative damage indicators and environmental information. Cumulative damage indicators include, but are not limited to: high load duration; the number of complete temperature change cycles from room temperature to high temperature and back to room temperature; and the cumulative extent to which the supply voltage exceeds the rated range. The lifetime degradation model used in the calculation can employ accelerated aging models such as the Arrhenius model or the Coffin-Manson model, converting the cumulative damage indicators into a lifetime degradation ratio. The aging failure risk ratio (HR) is the ratio of the target graphics processor's current failure risk to the graphics processor's failure risk under a baseline state. The baseline state refers to the graphics processor's brand-new, unaged state (factory test state), with its failure risk being the baseline risk value; HR = 1.0 indicates that the current risk is consistent with the baseline state; HR greater than 1.0 indicates that the current risk is higher than the baseline state. Based on the cumulative stress parameters of the target graphics processor, survival analysis models (such as the Cox proportional hazards model) can be used to calculate the aging risk. These stress parameters include, but are not limited to, average operating temperature, voltage fluctuation standard deviation, and high load percentage. The HR value directly reflects the failure risk multiple caused by current aging; for example, HR = 3.0 means that the current failure risk of the target graphics processor is 3 times that of a brand-new state. Remaining useful life provides an absolute time reference, while the aging failure risk ratio provides a relative risk comparison, jointly supporting long-term maintenance decisions. Through cumulative damage indicators and survival analysis models, the impact of stress factors such as temperature, voltage, and load on hardware lifespan is accurately reflected, conforming to the aging physical laws of semiconductor devices.

[0156] Sub-step S32: Based on the predicted short-term risk parameters and the predicted long-term loss parameters, determine and execute the risk processing operations corresponding to each of the at least one target graphics processors.

[0157] For at least one target graphics processor (GPU), corresponding risk handling operations are matched and executed based on calculated short-term and long-term loss parameters. Differentiated operations can be performed on GPUs with different risk levels. By combining short-term and long-term loss parameters, differentiated risk handling operations are matched and executed for each GPU, avoiding excessive intervention in low-risk nodes or insufficient intervention in high-risk nodes, improving resource utilization efficiency, and reducing business interruption and data loss through data migration and preventative maintenance before failures occur, thereby enhancing system reliability.

[0158] For example, risk management operations can be preventative maintenance. When the short-term failure probability is within a preset threshold, and the predicted long-term wear parameters are also within the corresponding preset threshold, the target processor is determined to be of medium risk. At this time, a series of preventative maintenance operations will be proactively executed to attempt to mitigate the current risk and extend stable operation time. These operations may include: reloading or updating device drivers, dynamically adjusting fan speed strategies to enhance heat dissipation, moderately limiting hardware peak performance to reduce instantaneous stress, or performing deeper memory / cache diagnostics and cleanup, etc.

[0159] In another example, risk management could involve continuous monitoring. When the probability of a short-term failure falls below a monitoring threshold, the corresponding target processor is classified as low-risk. In this state, no proactive intervention is performed to avoid unnecessary overhead and disturbance. However, all relevant hardware metrics data are continuously recorded and analyzed to maintain routine monitoring of the hardware's health.

[0160] In one embodiment, sub-step S32 may include the following sub-steps:

[0161] Sub-step S321: For each of the at least one target graphics processors, if the predicted short-term risk parameter is greater than or equal to a preset short-term risk threshold and the predicted long-term loss parameter is greater than or equal to a preset long-term risk threshold, a data migration operation is performed on the corresponding target graphics processor.

[0162] Predicting short-term risk parameters refers to quantifying the probability of the target graphics processor failing in the near future; preset short-term risk thresholds refer to the short-term risk thresholds that trigger emergency data migration; predicting long-term wear parameters refers to quantifying the risk of the target graphics processor failing due to long-term cumulative damage; preset long-term risk thresholds refer to the long-term aging thresholds that trigger emergency data migration.

[0163] For example, a data migration operation can take the following steps: First, query the cluster resource management module to select a standby graphics processor that is in normal condition and has a low load; then, migrate the running task data and computing context on the target graphics processor to the standby hardware via a high-speed data link (such as NVLink or PCIe); after the migration is complete, switch business traffic or task scheduling to the standby hardware, and the target graphics processor stops receiving new tasks, thus completing the data migration. By migrating data only to nodes that are highly likely to fail in the short term and are severely aged in the long term, resource waste is avoided, intervention is ensured to be targeted, and proactive data migration is achieved before failure occurs, thus ensuring business continuity.

[0164] By acquiring a topology graph corresponding to at least one graphics processor (GPU), where each node corresponds to at least one GPU and each edge represents a physical connection between GPUs, and based on the operating status and environmental information of at least one GPU, the current weight values ​​of the edges in the topology graph are determined. Based on the topology graph and the current weight values ​​of its edges, at least one target GPU with a risk of failure is predicted. This achieves the prediction of the fault propagation path and the range of potentially affected GPUs based on physical connections and quantified fault propagation risk, effectively addressing the problem of existing technologies being unable to predict fault propagation paths and the range of potentially affected GPUs. By identifying and executing risk handling operations for at least one target GPU, proactive intervention is achieved for GPUs with a risk of failure, effectively mitigating the problem in existing technologies where only passive responses are possible after a fault occurs, making it difficult to effectively prevent losses caused by cascading failures.

[0165] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0166] Reference Figure 2 The diagram illustrates a structural block diagram of an embodiment of a graphics processor risk processing device according to the present invention, which may specifically include the following modules:

[0167] The information acquisition module 201 is used to acquire the operating status information and environmental information of at least one graphics processor;

[0168] The topology graph acquisition module 202 is used to acquire the topology graph corresponding to the at least one graphics processor; the nodes of the topology graph correspond to the at least one graphics processor, and the edges correspond to the physical connection relationships between the at least one graphics processor;

[0169] The current weight value acquisition module 203 is used to determine the current weight value of the edge in the topology graph based on the running status information of the at least one graphics processor and the environmental information.

[0170] The graphics processor determination module 204 is used to predict at least one target graphics processor with a risk of failure based on the topology graph and the current weight values ​​of the edges in the topology graph.

[0171] The graphics processor processing module 205 is used to determine and execute risk processing operations for the at least one target graphics processor.

[0172] Optionally, the graphics processor processing module includes

[0173] The parameter determination submodule is used to determine the predicted short-term risk parameters and predicted long-term loss parameters of the at least one target graphics processor based on the operating status information and the environmental information of the at least one target graphics processor.

[0174] The operation determination submodule is used to determine and execute the risk processing operation corresponding to each of the at least one target graphics processors based on the predicted short-term risk parameters and the predicted long-term loss parameters.

[0175] Optionally, the current weight value acquisition module includes:

[0176] The data link weight determination submodule is used to determine the data link weight based on the running status information.

[0177] The indirect coupling link weight determination submodule is used to determine the indirect coupling link weight based on the environmental information.

[0178] The fusion weight determination submodule is used to determine the current weight value of the edge in the topology graph based on the data link weight and the indirect coupling link weight.

[0179] Optionally, the operational status information includes bit error rate and utilization rate, and the data link weight determination submodule includes:

[0180] The bit error rate impact coefficient determination unit is used to determine the bit error rate impact coefficient in a preset first mapping table based on the bit error rate; the preset first mapping table is a mapping table between the bit error rate and the bit error rate impact coefficient.

[0181] The utilization rate influence coefficient determination unit is used to determine the utilization rate influence coefficient in a preset second mapping table based on the utilization rate; the preset second mapping table is a mapping table between the utilization rate and the utilization rate influence coefficient.

[0182] The first data link weight determination unit is used to determine the data link weight based on the bit error rate, bit error rate influence coefficient, utilization rate, and utilization rate influence coefficient of the at least one graphics processor.

[0183] Optionally, the first data link weight comprehensive determination unit includes:

[0184] An offset determination subunit is used to determine a first relative offset and a second relative offset; the first relative offset is the relative offset of the bit error rate with respect to a preset bit error rate threshold, and the second relative offset is the relative offset of the utilization rate with respect to a preset utilization rate threshold.

[0185] The weighting determination subunit is used to determine a first weight and a second weight based on the ratio of the first relative offset and the second relative offset; the first weight is used to represent the importance of the bit error rate to the risk assessment of the graphics processor, and the second weight is used to represent the importance of the utilization rate to the risk assessment of the graphics processor.

[0186] The second data link weight comprehensive determination subunit is used to determine the data link weight based on the bit error rate of the at least one graphics processor, the bit error rate influence coefficient, the first weight, the utilization rate, the utilization rate influence coefficient, and the second weight.

[0187] Optionally, the environmental information includes ambient temperature, the indirect coupling link weight includes thermal coupling link weight, and the indirect coupling link weight determination submodule includes:

[0188] The third offset determination unit is used to determine the third offset; the third offset is the relative offset of the temperature information with respect to the preset temperature reference value;

[0189] The thermal coupling link weight determination unit is used to take the product of the third offset and the preset temperature weight adjustment factor as the thermal coupling link weight.

[0190] Optionally, the environmental information includes the power supply voltage, the indirect coupling link weight includes the power supply link weight, and the indirect coupling link weight determination submodule includes:

[0191] The fourth offset determination unit is used to determine the fourth offset; the fourth offset is the relative offset of the power supply voltage information with respect to the preset voltage reference value;

[0192] The power supply link weight determination unit is used to take the product of the fourth offset and the preset power supply link weight adjustment factor as the power supply link weight.

[0193] Optionally, the running status information includes running status information within a preset time window, and the parameter determination submodule includes:

[0194] The short-term risk parameter determination unit is used to determine the predicted short-term risk parameters based on the operating status information and the environmental information within the preset time window. The predicted short-term risk parameters are used to represent the probability of failure of the target graphics processor in the near future.

[0195] Optionally, the device further includes:

[0196] The frequency adjustment submodule is used to increase the acquisition frequency of the operating status information and environmental information of the at least one graphics processor from a preset first information acquisition frequency to a preset second information acquisition frequency if the detected operating status information is greater than or equal to a preset operating status information threshold and / or the environmental information is greater than or equal to a preset environmental information threshold.

[0197] Optionally, the graphics processor determining module includes:

[0198] The fault node determination submodule is used to determine the location of the corresponding fault node in the topology graph; the location of the fault node is the node location of the graphics processor that has malfunctioned in the topology graph.

[0199] The fault impact intensity submodule is used to determine the fault impact intensity of the at least one graphics processor based on a preset impact intensity benchmark value, the location of the fault node, the topology graph, and the current weight values ​​of the edges in the topology graph; the fault impact intensity is a quantified value of the degree of fault impact transmitted layer by layer from the location of the fault node through the edges of the topology graph;

[0200] A target graphics processor determination submodule is used to identify, among the at least one graphics processor, the graphics processor whose fault impact intensity is greater than or equal to a preset fault impact intensity threshold as the target graphics processor.

[0201] Optionally, the operation determination submodule includes:

[0202] The data migration unit is configured to perform a data migration operation on each of the at least one target graphics processors when the predicted short-term risk parameter is greater than or equal to a preset short-term risk threshold and the predicted long-term loss parameter is greater than or equal to a preset long-term risk threshold.

[0203] By acquiring a topology graph corresponding to at least one graphics processor (GPU), where each node corresponds to at least one GPU and each edge represents a physical connection between GPUs, and based on the operating status and environmental information of at least one GPU, the current weight values ​​of the edges in the topology graph are determined. Based on the topology graph and the current weight values ​​of its edges, at least one target GPU with a risk of failure is predicted. This achieves the prediction of the fault propagation path and the range of potentially affected GPUs based on physical connections and quantified fault propagation risk, effectively addressing the problem of existing technologies being unable to predict fault propagation paths and the range of potentially affected GPUs. By identifying and executing risk handling operations for at least one target GPU, proactive intervention is achieved for GPUs with a risk of failure, effectively mitigating the problem in existing technologies where only passive responses are possible after a fault occurs, making it difficult to effectively prevent losses caused by cascading failures.

[0204] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0205] Reference Figure 3 The diagram illustrates a structural block diagram of an electronic device 30 provided by an embodiment of the present invention, comprising:

[0206] The processor 301, the memory 302, and the computer program 3021 stored in the memory 302 and capable of running on the processor 301, when the computer program 3021 is executed by the processor 301, implement the various processes of the above-described graphics processor risk handling method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0207] Reference Figure 4 The diagram shows a structural block diagram of a computer-readable storage medium 40 provided in an embodiment of the present invention. The computer-readable storage medium 40 stores a computer program 401. When the computer program 401 is executed by a processor, it implements the various processes of the above-described graphics processor risk processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0208] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0209] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0210] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0211] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0213] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0214] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0215] The above provides a detailed description of a graphics processor risk processing method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for handling risks in graphics processing units, characterized in that, The method includes: Obtain the running status and environment information of at least one graphics processor; Obtain the topology graph corresponding to the at least one graphics processor; the nodes of the topology graph correspond to the at least one graphics processor, and the edges correspond to the physical connection relationships between the at least one graphics processor; Based on the running status information of the at least one graphics processor and the environmental information, determine the current weight value of the edge in the topology graph; Based on the topology graph and the current weight values ​​of the edges in the topology graph, predict at least one target graphics processor with a risk of failure; Identify and execute risk handling operations for the at least one target graphics processor.

2. The graphics processor risk handling method according to claim 1, characterized in that, The process of determining and executing risk handling operations for the at least one target graphics processor includes... Based on the operating status information and environmental information of the at least one target graphics processor, the predicted short-term risk parameters and predicted long-term loss parameters of the at least one target graphics processor are determined. Based on the predicted short-term risk parameters and the predicted long-term loss parameters, the risk processing operations corresponding to each of the at least one target graphics processors are determined and executed.

3. The graphics processor risk handling method according to claim 1, characterized in that, Determining the current weight value of an edge in the topology graph based on the operating status information of the at least one graphics processor and the environment information includes: Based on the aforementioned operational status information, the data link weights are determined; Based on the environmental information, determine the weight of the indirect coupling link; The current weight value of the edge in the topology graph is determined based on the data link weight and the indirect coupling link weight.

4. The graphics processor risk handling method according to claim 3, characterized in that, The operational status information includes bit error rate and utilization rate. Determining the data link weight based on the operational status information includes: Based on the bit error rate, a bit error rate influence coefficient is determined in a preset first mapping table; the preset first mapping table is a mapping table between the bit error rate and the bit error rate influence coefficient. Based on the utilization rate, a utilization rate influence coefficient is determined in a preset second mapping table; the preset second mapping table is a mapping table between the utilization rate and the utilization rate influence coefficient. The data link weight is determined based on the bit error rate, bit error rate impact coefficient, utilization rate, and utilization rate impact coefficient of the at least one graphics processor.

5. The graphics processor risk handling method according to claim 4, characterized in that, The step of determining the data link weight based on the bit error rate, the bit error rate impact coefficient, the utilization rate, and the utilization impact coefficient of the at least one graphics processor includes: Determine a first relative offset and a second relative offset; the first relative offset is the relative offset of the bit error rate with respect to a preset bit error rate threshold, and the second relative offset is the relative offset of the utilization rate with respect to a preset utilization rate threshold; A first weight and a second weight are determined based on the ratio of the first relative offset to the second relative offset; the first weight is used to represent the importance of the bit error rate to the risk assessment of the graphics processor, and the second weight is used to represent the importance of the utilization rate to the risk assessment of the graphics processor. The data link weight is determined based on the bit error rate of the at least one graphics processor, the bit error rate influence coefficient, the first weight, the utilization rate, the utilization rate influence coefficient, and the second weight.

6. The graphics processor risk handling method according to claim 3, characterized in that, The environmental information includes ambient temperature, the indirect coupling link weight includes thermal coupling link weight, and determining the indirect coupling link weight based on the environmental information includes: Determine the third offset; the third offset is the relative offset of the temperature information with respect to the preset temperature reference value; The product of the third offset and the preset temperature weight adjustment factor is used as the weight of the thermal coupling link.

7. The graphics processor risk handling method according to claim 3, characterized in that, The environmental information includes the power supply voltage, the indirect coupling link weight includes the power supply link weight, and determining the indirect coupling link weight based on the environmental information includes: Determine the fourth offset; the fourth offset is the relative offset of the power supply voltage information with respect to the preset voltage reference value; The product of the fourth offset and the preset power supply link weight adjustment factor is used as the power supply link weight.

8. The graphics processor risk handling method according to claim 2, characterized in that, The operational status information includes operational status information within a preset time window. The step of determining the predicted short-term risk parameters of the at least one target graphics processor based on the operational status information and the environmental information includes: Based on the operating status information and environmental information within the preset time window, a predicted short-term risk parameter is determined. The predicted short-term risk parameter is used to represent the probability of failure of the target graphics processor in the near future.

9. The graphics processor risk handling method according to claim 2, characterized in that, The predicted long-term wear parameters include remaining service life and / or aging failure risk ratio, whereby the aging failure risk ratio is the ratio of the current failure risk of the target graphics processor to the failure risk of the graphics processor under baseline conditions.

10. The graphics processor risk handling method according to claim 1, characterized in that, The method further includes: If the detected operating status information is greater than or equal to a preset operating status information threshold and / or the environmental information is greater than or equal to a preset environmental information threshold, the acquisition frequency of the operating status information and environmental information of the at least one graphics processor is increased from a preset first information acquisition frequency to a preset second information acquisition frequency.

11. The graphics processor risk handling method according to claim 1, characterized in that, The step of predicting at least one target graphics processor with a risk of failure based on the topology graph and the current weight values ​​of the edges in the topology graph includes: Determine the location of the faulty node in the topology graph; the location of the faulty node is the node location in the topology graph corresponding to the graphics processor that has malfunctioned. The fault impact intensity of the at least one graphics processor is determined based on a preset impact intensity benchmark value, the location of the fault node, the topology graph, and the current weight values ​​of the edges in the topology graph; the fault impact intensity is a quantified value of the degree of fault impact transmitted layer by layer from the location of the fault node through the edges of the topology graph. In the at least one graphics processor, the graphics processor whose fault impact intensity is greater than or equal to a preset fault impact intensity threshold is selected as the target graphics processor.

12. The graphics processor risk handling method according to claim 2, characterized in that, The step of determining and executing risk processing operations corresponding to each of the at least one target graphics processors based on the predicted short-term risk parameters and the predicted long-term loss parameters includes: For each of the at least one target graphics processors, if the predicted short-term risk parameter is greater than or equal to a preset short-term risk threshold and the predicted long-term loss parameter is greater than or equal to a preset long-term risk threshold, a data migration operation is performed on the corresponding target graphics processor.

13. A graphics processor risk processing device, characterized in that, The device includes: The information acquisition module is used to acquire the operating status information and environmental information of at least one graphics processor; A topology graph acquisition module is used to acquire a topology graph corresponding to the at least one graphics processor; the nodes of the topology graph correspond to the at least one graphics processor, and the edges correspond to the physical connection relationships between the at least one graphics processor. The current weight value acquisition module is used to determine the current weight value of the edge in the topology graph based on the running status information of the at least one graphics processor and the environmental information. A graphics processor determination module is used to predict at least one target graphics processor with a risk of failure based on the topology graph and the current weight values ​​of the edges in the topology graph. A graphics processor processing module is used to determine and execute risk processing operations for the at least one target graphics processor.

14. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of a graphics processor risk handling method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of a graphics processor risk handling method as described in any one of claims 1-12.

Images