GPU virtualization fault root cause analysis and self-healing method and system based on AI assistance

Abstract

The invention relates to a GPU virtualization fault root cause analysis and self-healing method and system based on AI assistance, and the method comprises the steps: collecting multi-source heterogeneous data of different hierarchies in a GPU virtualization environment in real time, marking the multi-source heterogeneous data collected at the same time with a uniform timestamp and a corresponding resource label, and transmitting the data to a preset fault analysis positioning model, the fault analysis positioning model analyzes and judges whether the multi-source heterogeneous data is abnormal or not in real time, and abnormal judgment information is generated; when the abnormity judgment information shows that abnormity exists, the fault analysis positioning model judges fault information of the multi-source heterogeneous data and outputs a plurality of fault root causes to be selected based on the fault information; based on a preset fault inference map, performing logic verification of corresponding multi-source heterogeneous data on the fault root cause to be selected, and outputting fault root cause information of which the confidence coefficient is higher than a confidence coefficient threshold value; and generating a repair strategy based on the fault root cause information. The method has the effect of improving the self-healing capability of the GPU virtualization fault.

Description

Technical Field

[0001] This application relates to the technical field of GPU virtualization, and in particular to an AI-assisted GPU virtualization fault root cause analysis, self-healing method and system. Background Technology

[0002] Modern cloud computing and data centers widely adopt GPU virtualization technology to securely and efficiently allocate physical GPU resources to multiple virtual machines (VMs) or containers. However, GPU virtualization environment failures have many root causes and are difficult to locate, such as vGPU instances failing to start, sudden performance drops, application errors, and device pass-through failures.

[0003] Existing methods mostly use monitoring tools to monitor and alert on multi-source metrics data. Although log analysis tools can collect data centrally, they cannot understand the fault context of GPU virtualization. In other words, the data linkage between monitoring tools is poor, making it difficult to pinpoint the accurate root cause of faults when faced with multi-source heterogeneous data. Therefore, it is also difficult to repair faults, which needs to be improved. Summary of the Invention

[0004] To improve the accuracy of root cause analysis and localization of GPU virtualization faults, and to enhance the self-healing capability of GPU virtualization faults, this application provides an AI-assisted method and system for GPU virtualization fault root cause analysis and self-healing.

[0005] The above-mentioned objective of this application is achieved through the following technical solution:

[0006] An AI-assisted method for root cause analysis and self-healing of GPU virtualization faults, comprising the following steps:

[0007] Real-time acquisition of multi-source heterogeneous data at different levels in the GPU virtualization environment, including performance metrics, configuration information, log information, and topology information;

[0008] The multi-source heterogeneous data collected simultaneously are marked with a unified timestamp and corresponding resource tags and sent to the preset fault analysis and location model. The fault analysis and location model analyzes and determines in real time whether there are any anomalies in the multi-source heterogeneous data and generates anomaly judgment information.

[0009] When the anomaly judgment information indicates the presence of an anomaly, the fault analysis and localization model determines the fault information of the multi-source heterogeneous data and outputs several candidate root causes of the fault based on the fault information.

[0010] Based on the preset fault reasoning graph, the selected fault root cause is logically verified by corresponding multi-source heterogeneous data, and fault root cause information with confidence higher than the confidence threshold is output; and a repair strategy is generated based on the fault root cause information to automatically repair the fault information.

[0011] By adopting the above technical solution, multi-source heterogeneous data is collected in real time during the operation of the GPU virtualization environment. This involves acquiring different data dimensions at the hardware and virtualization levels and assigning the same timestamp and identifier to the multi-source heterogeneous data. This allows for the creation of a data profile of the GPU virtualization environment at the same time dimension, facilitating subsequent fault analysis. Furthermore, the fault analysis and localization model identifies the specific fault information by analyzing the abnormal portions of the multi-source heterogeneous data and outputs one or more possible root causes based on the specific fault symptoms. To make the determination of root cause information more accurate, a pre-set fault reasoning graph is used. Multi-source heterogeneous data is mapped to a fault reasoning graph, and the reliability of the selected root causes of faults is logically verified. Logical verification can trace the causes of fault information back based on causal, dependency, and symbiotic relationships. The confidence level of the data in the backtracking process is proved by multi-source heterogeneous data. Finally, the root cause information of faults with a confidence level higher than the confidence level threshold is output to improve the accuracy and reliability of fault root cause judgment. Finally, a repair strategy is generated based on the fault root cause information to realize the autonomous intelligent repair of GPU virtualization anomalies. This improves the accuracy of GPU virtualization fault root cause analysis and location, and enhances the self-healing capability of GPU virtualization faults.

[0012] Optionally: The step of the fault analysis and location model analyzing and determining whether there are anomalies in multi-source heterogeneous data in real time and generating anomaly judgment information includes:

[0013] The fault analysis and localization model acquires a set of vector features from multi-source heterogeneous data;

[0014] The historical vector features pre-stored in the vector feature set are compared; the historical vector features include abnormal historical vector features and normal historical vector features.

[0015] Based on the similarity between the current vector feature set and the historical vector features, it is determined whether there is an anomaly in the current vector feature set, and anomaly judgment information is generated.

[0016] By adopting the above technical solution, multi-source heterogeneous data is converted into vector features using NLP technology, and time-series indicators are converted into statistical features to obtain the corresponding vector feature set. By pre-storing vector feature sets that meet the abnormal and normal conditions, namely abnormal vector features and normal historical vector features, unsupervised learning is used to compare the approximation of the vector features. The approximation is used to determine whether there is an anomaly in the current multi-source heterogeneous data and generate anomaly judgment information, which includes the two cases of being judged as abnormal and normal.

[0017] Optionally: When the anomaly judgment information indicates the existence of an anomaly, the step of the fault analysis and localization model judging the fault information of multi-source heterogeneous data and outputting several candidate root causes based on the fault information includes:

[0018] The fault analysis and localization model identifies abnormal vectors in the vector feature set and locates the corresponding fault information based on the abnormal vectors.

[0019] Determine whether there are any classic anomaly vector features in the historical anomaly vector features that are more or less similar to the current anomaly vector than the maximum threshold.

[0020] If so, the classic root cause corresponding to the classic anomaly vector feature will be the only candidate root cause of the current fault information.

[0021] If not, then determine whether there is an approximate abnormal vector feature in the abnormal historical vector features that is greater than or equal to the minimum threshold with respect to the current abnormal vector, and take the fault root cause corresponding to the approximate abnormal vector feature as the candidate fault root cause.

[0022] By adopting the above technical solution, when an anomaly is detected, the fault analysis and localization model identifies and obtains the anomaly vector that caused the anomaly. Based on the anomaly vector, it locates the fault information caused by the current anomaly, such as GPU stopping working, error codes appearing, or abnormal performance indicators. Furthermore, if the similarity between the anomaly vector and the classic anomaly vector in the historical anomaly vector features is greater than the maximum threshold, then the current anomaly is determined to be a fault situation with classic anomaly vector features. This can generate a high-confidence root cause of the fault. Here, the anomaly vector can be a single vector feature or a combination of vector features. When the anomaly vector feature cannot match the classic anomaly vector feature, it first filters out the approximate anomaly vector features based on the comparison of similarity. The root causes of the faults of the filtered anomaly vector features are used as the candidate root causes of the current fault situation, providing a more comprehensive range of fault selection possibilities. This provides a reliable data foundation for subsequent fault root cause screening and makes the confirmation of the fault root cause more accurate.

[0023] Optionally, the step of performing logical verification of the selected root cause of the fault based on a preset fault reasoning graph and outputting root cause information with a confidence level higher than a confidence threshold includes:

[0024] When the candidate root cause is the only classic root cause, the candidate root cause is given the highest confidence and output as the final confirmed root cause.

[0025] When there are several candidate root causes of failure, obtain the vector feature set of each candidate root cause of failure.

[0026] Based on the fault reasoning graph, each feature vector in the vector feature set is verified with the corresponding logical node, and the confidence level of each candidate fault root cause is output. The types of logical nodes include hardware component nodes, software entity nodes, configuration item nodes, and error code nodes.

[0027] By adopting the above technical solution, when the root cause of the fault is the only classic root cause, the root cause of the current abnormal situation is immediately output so that staff can perform rapid maintenance. When there are several candidate root causes, it is necessary to further verify the vectors represented by each type of data in the vector feature set. Using a preset fault reasoning graph that includes multiple relationship types such as causality, dependency, and co-occurrence, the root cause of the current fault information is traced back to its source. The reliability of the data of the relationship nodes of hardware, software, configuration, code, etc. is recorded during the backtracking process. Based on the mutual verification and corroboration of the relationships of multiple nodes, the reliability and accuracy of fault root cause backtracking are improved.

[0028] Optionally: The step of verifying each feature vector in the vector feature set with its corresponding logical node based on the fault reasoning graph, and outputting the confidence level of each candidate fault root cause, includes:

[0029] The fault information corresponding to the candidate fault root cause is mapped to the fault node of the fault reasoning graph, and logical nodes are filtered based on preset relationship rules to generate an associated sub-graph. Each logical node is pre-bound with a corresponding vector interval.

[0030] Each vector feature in the vector feature set is compared with the vector range of the corresponding logical node to generate the confidence score of that logical node.

[0031] Based on the confidence level of the faulty node and each logical node, output the confidence level of each candidate root cause of the fault.

[0032] Identify whether the confidence level of each candidate root cause is higher than the confidence level threshold, and output the root cause information of the root cause with a confidence level higher than the confidence level threshold.

[0033] By adopting the above technical solution, the fault information indicating an abnormal situation is mapped to the fault nodes in the fault reasoning graph. Based on preset relational rules, the fault reasoning graph can filter out relational nodes that have no causal, dependency, or co-occurrence relationship with the fault information, generating only the associated sub-graph with logical node connections. This data screening improves the efficiency of subsequent fault root cause verification. Furthermore, by mapping each vector feature in the vector feature set to its corresponding logical node and comparing it with the vector interval, it is possible to know the collaborative performance of various related data types under the current fault condition. For example, if the physical vGPU performance suddenly drops to an abnormal stage, then... By tracing the GPU frequency data, the characteristic of GPU frequency reduction is found in the corresponding relationship nodes. Further backtracking is performed on the cause of GPU frequency reduction, including the path nodes of GPU temperature and PCIe bandwidth detection. It is found that the bandwidth is smooth but the physical GPU is overheating. At this time, the root cause of the failure can be determined as physical GPU overheating. Thus, the failure root cause information with the highest confidence can be found on the data backtracking path. Based on the comparison of confidence values, all failure root causes with confidence values ​​higher than the confidence threshold are output. When faced with failure information caused by multiple failure root causes, a complete and reliable maintenance suggestion can be provided for subsequent maintenance staff.

[0034] Optionally: the step of generating a repair strategy based on the root cause information to automatically repair the fault information includes:

[0035] Based on the output root cause information of the fault, obtain the historical repair methods corresponding to the output root cause of the fault.

[0036] Based on the current GPU resource operating status, determine the repair cost of the acquired historical repair methods; and generate a list of repair actions to execute the historical repair methods.

[0037] By adopting the above technical solution, after screening and verifying the candidate root causes of faults through the fault reasoning graph, the output fault root cause information is matched with repair methods. That is, the historical repair methods corresponding to the fault root cause are screened out for preliminary matching. It is also necessary to consider the current GPU resource operation status, as different repair methods will have different degrees of impact on the current GPU resource operation status, such as task interruption or long-term GPU idleness, in order to reduce the loss caused by repair. Finally, a repair action list is generated to automatically repair the fault information according to the list, providing an efficient and low-cost repair method.

[0038] Optionally: The step of determining the repair cost of the acquired historical repair methods based on the current GPU resource operating status and generating a list of repair actions to send to automatically repair fault information includes:

[0039] Based on the cost minimization rule, each historical repair method is individually repaired and decomposed to obtain several repair action nodes with a preset order;

[0040] Based on the current GPU resource operating status, determine the impact scope and execution cost of each repair action node;

[0041] Analyze the impact range and execution cost of each repair action node in each historical repair method, and select the historical repair method with the lowest repair cost;

[0042] Based on a preset order of repair action nodes, a list of repair actions is generated, and repair actions are automatically performed in the order of each repair action node.

[0043] By adopting the above technical solution, the candidate historical repair methods are broken down, and each node in the decomposition process is selected based on the current GPU resource running status to find the repair action with the lowest cost. The impact range and execution cost of each repair action node are listed and compared in parallel with the nodes of each historical repair method. The historical repair method with the lowest cost is selected to generate a repair action list. The repair action of each node is automatically executed according to the repair action list to achieve self-healing of fault information.

[0044] The second objective of this invention is achieved through the following technical solution:

[0045] An AI-assisted GPU virtualization fault root cause analysis and self-healing system includes:

[0046] The data acquisition module is used to collect multi-source heterogeneous data at different levels in the GPU virtualization environment in real time. The multi-source heterogeneous data includes performance indicators, configuration information, log information, and topology information.

[0047] The anomaly detection module is used to mark the simultaneously collected multi-source heterogeneous data with a unified timestamp and corresponding resource tags, and send them to the preset fault analysis and location model. The fault analysis and location model analyzes and determines in real time whether there are anomalies in the multi-source heterogeneous data and generates anomaly detection information.

[0048] The root cause localization module is used to determine the fault information of multi-source heterogeneous data when the anomaly judgment information indicates that an anomaly exists, and output several candidate root causes of the fault based on the fault information.

[0049] The reasoning and verification module is used to perform logical verification of the selected root causes of faults based on the preset fault reasoning graph, and output the root cause information of faults with a confidence level higher than the confidence level threshold; and generate a repair strategy based on the root cause information of faults to automatically repair the fault information.

[0050] The above-mentioned objective three of this application is achieved through the following technical solution:

[0051] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described AI-assisted GPU virtualization fault root cause analysis and self-healing method.

[0052] The fourth objective of this application is achieved through the following technical solution:

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned AI-assisted GPU virtualization fault root cause analysis and self-healing method.

[0054] In summary, this application includes at least one of the following beneficial technical effects:

[0055] 1. The fault analysis and localization model locates the current fault information by identifying anomalies in multi-source heterogeneous data. Based on specific fault symptoms, it outputs one or more possible root causes. To improve the accuracy of root cause identification, a pre-set fault reasoning graph maps the multi-source heterogeneous data to the graph. The reliability of the selected root causes is logically verified. This verification traces the causes of the fault information back to their origins based on causal, dependency, and symbiotic relationships. The confidence level of the data during the backtracking process is demonstrated using multi-source heterogeneous data. Finally, root cause information with a confidence level higher than a threshold is output, improving the accuracy and reliability of root cause identification. Finally, a repair strategy is generated based on the root cause information to achieve autonomous and intelligent repair of GPU virtualization anomalies; improving the accuracy of GPU virtualization fault root cause analysis and localization, and enhancing the GPU virtualization fault self-healing capability.

[0056] 2. Multi-source heterogeneous data is converted into vector features using NLP technology, and time-series indicators are converted into statistical features to obtain corresponding vector feature sets. By pre-storing vector feature sets that meet the abnormal and normal conditions, i.e. abnormal vector features and normal historical vector features, unsupervised learning is used to compare the approximation of the vector features. The approximation is used to determine whether there are anomalies in the current multi-source heterogeneous data and generate anomaly judgment information, which includes the two cases of being judged as abnormal and normal.

[0057] 3. If the similarity between the anomaly vector and the classic anomaly vector in the anomaly history vector features is greater than the maximum threshold, then the current anomaly is determined to be the fault situation of the classic anomaly vector features. This can generate a high-confidence root cause of the fault. The anomaly vector here can be a single vector feature or a combination of vector features. When the anomaly vector features cannot match the classic anomaly vector features, then based on the comparison of similarity, the approximate anomaly vector features are first selected, and the root causes of the faults of the selected anomaly vector features are used as the candidate root causes of the current fault situation, providing a more comprehensive range of fault selection possibilities.

[0058] 4. By mapping each vector feature in the vector feature set to the corresponding logical node and comparing it with the vector interval, we can know the collaborative performance of various related data types under the current fault condition. For example, if the physical vGPU performance drops sharply to the abnormal stage, we can trace back the GPU frequency data and find the characteristics of GPU frequency reduction in the corresponding relationship node. We can further trace back the cause of GPU frequency reduction, including the path node of GPU temperature and the node of PCIe bandwidth detection. If we find that the bandwidth is smooth but the physical GPU is overheating, we can determine that the root cause of the fault is physical GPU overheating. Thus, we can find the fault root cause information with the highest confidence on the data backtracking path. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of a method for root cause analysis and self-healing of GPU virtualization faults based on AI assistance, as proposed in this application.

[0060] Figure 2 This is a flowchart of an embodiment of an AI-assisted GPU virtualization fault root cause analysis and self-healing method according to this application;

[0061] Figure 3 This is another implementation flowchart of step S43 in an embodiment of an AI-assisted GPU virtualization fault root cause analysis and self-healing method of this application;

[0062] Figure 4 This is a schematic block diagram of a computer device according to this application. Detailed Implementation

[0063] The following is in conjunction with the appendix Figures 1-4 This application will be described in further detail.

[0064] In one embodiment, such as Figure 1 and Figure 2 As shown, this application discloses an AI-assisted method for GPU virtualization fault root cause analysis and self-healing, which specifically includes the following steps:

[0065] S10: Real-time acquisition of multi-source heterogeneous data at different levels in the GPU virtualization environment, including performance indicators, configuration information, log information, and topology information;

[0066] In this embodiment, GPU virtualization environments such as NVIDIA vGPU, MxGPU, and open-source solutions such as GVirtu-S and VFIO-MDEV in QEMU / KVM securely and efficiently allocate physical GPU resources to multiple virtual machines (VMs) or containers. This technology stack involves hardware, host drivers, virtualization hypervisors, in-VM drivers, and upper-layer applications, making it complex in structure.

[0067] Performance metrics, configuration information, log information, and topology information represent data types. Performance metrics include GPU utilization, video memory usage, SM activity, PCIe bandwidth, temperature, power consumption, and ECC error count. Log information includes host kernel logs, NVIDIA GRID / driver logs, Hypervisor logs, and virtual machine event logs. Topology information includes the GPU physical location (NUMA node) and the affinity relationship between the PCIe topology and the CPU and memory.

[0068] S20: Mark the simultaneously collected multi-source heterogeneous data with a unified timestamp and corresponding resource tags, and send them to the preset fault analysis and location model. The fault analysis and location model analyzes and determines in real time whether there are any anomalies in the multi-source heterogeneous data, and generates anomaly judgment information.

[0069] In this embodiment, each piece of data is given a unified timestamp and resource tag, and then associated and aligned to form a complete data panorama. In addition, feature engineering is required to vectorize the multi-source heterogeneous data.

[0070] The fault analysis and localization model adopts an unsupervised learning approach, such as a time series prediction model (e.g., LSTM), which learns patterns from normal historical data to detect indicators and log patterns in multi-source heterogeneous data, and identifies anomalies in the data through learning, generating information to determine anomalies.

[0071] Specifically, step S20 includes the following steps:

[0072] S21: The fault analysis and localization model acquires a set of vector features from multi-source heterogeneous data;

[0073] S22: Perform feature comparison on the historical vector features pre-stored in the vector feature set; historical vector features include abnormal historical vector features and normal historical vector features;

[0074] S23: Based on the similarity between the current vector feature set and the historical vector features, determine whether there is an anomaly in the current vector feature set and generate anomaly judgment information.

[0075] Among them, the vector feature set is a feature engineering process that transforms unstructured log text, such as error messages, from multi-source heterogeneous data into feature vectors using NLP technology; and transforms time-series indicators into statistical features, such as mean, variance, and trend data types.

[0076] The approximation comparison of the vector feature set with the pre-stored historical vector features is an approximate comparison of the overall feature distribution profile. The abnormal historical vector features include abnormal vector features of different fault types, including faults such as vGPU instance failure to start, performance drop, application error, and device pass-through failure.

[0077] If the approximation exceeds a preset threshold, an anomaly can be determined. Optionally, if the index of a single vector in the vector feature set exceeds the threshold, it is also considered an anomaly. By using both the overall vector feature set distribution and single vector feature index identification to make anomaly judgments, anomalies can be identified more accurately.

[0078] The anomaly detection information includes both normal and abnormal cases. Furthermore, the anomaly detection information will mark abnormal vectors or combinations of abnormal vectors in the vector feature set.

[0079] S30: When the anomaly judgment information indicates that an anomaly exists, the fault analysis and localization model judges the fault information of the multi-source heterogeneous data and outputs several candidate root causes of the fault based on the fault information.

[0080] In this embodiment, the fault analysis and localization model employs a supervised learning model based on deep neural networks. It uses historical vector feature data pre-matched with the root cause of the fault as a training set for learning and training. When a set of vector features from multi-source heterogeneous data is input, it can output the predefined root cause of the fault.

[0081] Specifically, step S30 includes:

[0082] S31: The fault analysis and localization model identifies abnormal vectors in the vector feature set and locates the corresponding fault information based on the abnormal vectors;

[0083] S32: Determine whether there are any classic anomaly vector features in the anomaly history vector features that are more or less similar to the current anomaly vector than the maximum threshold.

[0084] S33: If so, the classic root cause corresponding to the classic anomaly vector feature shall be the only candidate root cause of the current fault information.

[0085] S34: If not, determine whether there is an approximate abnormal vector feature in the abnormal historical vector features that is greater than or equal to the minimum threshold with respect to the current abnormal vector, and take the fault root cause corresponding to the approximate abnormal vector feature as the candidate fault root cause.

[0086] Anomaly vectors include single anomaly vector features or combinations of multiple anomaly vector features, while fault information represents the specific fault manifestation shown by the current anomaly. The identification of classic anomaly vector features is determined by whether the approximation of the current anomaly vector and its distribution to a specific anomaly vector feature in historical vector features reaches a user-defined maximum approximation threshold. In other words, the accurate root cause of the fault in the current vector feature set can be determined, i.e., the classic root cause. The approximation to a specific anomaly vector feature in historical vector features includes approximation between the anomaly vector and a single vector feature or a combination of some anomaly vector features.

[0087] If there are vector features in the current feature set that are more similar to the abnormal vector features than the minimum threshold, it means that the current fault information can find the root cause of the fault. However, if the number of associated root causes is greater than or equal to two, the system can initially locate the possible root causes of the fault in the current feature set as candidate root causes. For example, the fault information of CUDA application error in virtual machine VM1 may have multiple different root causes, including memory leak in Guest OS and vGPU contention for video memory.

[0088] S40: Based on the preset fault reasoning graph, perform logical verification of the selected fault root cause using corresponding multi-source heterogeneous data, output fault root cause information with a confidence level higher than the confidence level threshold, and generate a repair strategy based on the fault root cause information to automatically repair the fault information.

[0089] Specifically, step S40 includes the following steps:

[0090] S41: When the candidate root cause is the only classic root cause, assign the candidate root cause the highest confidence and output it as the final confirmed root cause.

[0091] S42: When there are several candidate root causes of failure, obtain the vector feature set of each candidate root cause of failure.

[0092] S43: Based on the fault reasoning graph, each feature vector in the vector feature set is verified with the corresponding logical node, and the confidence of each candidate fault root cause is output. The types of logical nodes include hardware component nodes, software entity nodes, configuration item nodes, and error code nodes.

[0093] In this embodiment, if the fault analysis and localization model outputs only one unique classic root cause, it is assigned the highest confidence level as the final confirmed root cause information, saving subsequent fault reasoning steps.

[0094] The pre-built fault reasoning graph is a dedicated knowledge graph of incidents and faults for GPU virtualization environments. It contains various types of logical nodes, including hardware entity types, software entity types, configuration items, symptom indicator types, root cause types, and repair action types. Hardware entity type nodes include, but are not limited to, physical GPUs, GPU memory, PCIe slots, CPUs, NUMA nodes, and power supply units. Software entity type nodes include, but are not limited to, host GPU drivers, virtual machine GPU drivers, and Hypervisors. (KVM), virtualization management program, container runtime, AI training framework; configuration items include but are not limited to vGPU type, memory quota, compute unit quota, QoS policy, driver version number; root causes of failure include but are not limited to uncorrectable ECC errors in physical GPU memory, incompatibility between host driver version and vGPU management service, vGPU configuration conflicts causing device initialization failure; physical GPU hardware errors, driver defects such as host driver version mismatch, vGPU configuration conflicts, virtual machine internal driver failure, resource overload; repair actions include but are not limited to restarting the host GPU driver, hot-migrating the virtual machine to another physical host, rolling back the host driver to a specified version, QoS bandwidth limiting of the entire vGPU, restarting the vGPU instance, migrating the virtual machine, rolling back the driver, adjusting resource configuration, etc. The confidence of node verification is determined based on the degree of data conformity of the corresponding vector features in the vector feature set at that node.

[0095] Furthermore, refer to Figure 3 Step S43 further includes the following steps:

[0096] S431: Map the fault information corresponding to the candidate fault root cause to the fault node of the fault reasoning graph, and filter the logical nodes based on the preset relationship rules to generate the associated sub-graph. Each logical node is pre-bound with a corresponding vector interval.

[0097] S432: Compare each vector feature in the vector feature set with the vector interval of the corresponding logical node to generate the confidence score of the logical node;

[0098] S433: Based on the confidence level of the faulty node and each logical node, output the confidence level of each candidate root cause of the fault;

[0099] S434: Identify whether the confidence level of each candidate root cause is higher than the confidence level threshold, and output the root cause information of the root cause with a confidence level higher than the confidence level threshold.

[0100] Among them, relationship rules refer to other related abnormal data nodes that cause the current fault information to be sent or occur together with the current fault information. Relationship rules include causal relationships, dependency relationships, co-occurrence relationships, and repair relationships. Causal relationships include relationships that cause or trigger, such as physical GPU overheating causing GPU frequency reduction, which in turn causes a sharp drop in vGPU performance. Dependency relationships include relationships that depend on or run on, such as CUDA applications within a virtual machine depending on NVIDIA drivers within the virtual machine, and NVIDIA drivers within the virtual machine depending on the allocated vGPU instance. Co-occurrence relationships include relationships that often occur together, such as PCIeAER errors often appearing alongside GPU link training failure logs in the host machine's dmesg. Repair relationships include two types of repair relationships: repairing through some means and mitigating through some means, such as vGPU instance freezing can be repaired by resetting the vGPU instance.

[0101] The confidence score of each logical node is calculated by comparing its vector features with a predefined vector range representing anomalies. A high confidence score indicates a feature within the range, while a low confidence score indicates a feature outside the range, with the confidence decreasing the further away from the range. Finally, the confidence scores of each logical node are weighted to obtain the final confidence score for each candidate root cause of the fault. Root causes with confidence scores higher than a certain threshold are considered credible by the system, and these may include one or more such root causes.

[0102] For example, the fault information is that the CUDA application in virtual machine VM1 reports the error "CUDA error: out of memory, vGPU memory usage is consistently 100%". The fault analysis and localization model outputs the following potential root causes: memory leak in the Guest OS, insufficient vGPU memory quota, or other vGPUs competing for memory. Verification of logical nodes using the fault reasoning graph reveals that the confidence level of the memory leak node in the Guest OS is low. However, backtracking reveals a high confidence level for a certain vGPU memory over-utilization scenario. Further backtracking leads to the conclusion that the vGPU memory in VM1 is being squeezed, thus confirming vGPU memory contention as the root cause.

[0103] Step S40 also includes the steps of obtaining historical repair methods corresponding to the output fault root cause based on the output fault root cause information; determining the repair cost of the obtained historical repair methods based on the current GPU resource running status; and generating a repair action list to execute the historical repair methods.

[0104] Specifically:

[0105] S44: Based on the cost minimization rule, each historical repair method is individually repaired and decomposed to obtain several repair action nodes with a preset order;

[0106] S45: Based on the current GPU resource operating status, determine the impact range and execution cost of each repair action node;

[0107] S46: Analyze the impact range and execution cost of each repair action node in each historical repair method, and select the historical repair method with the lowest repair cost;

[0108] S47: Based on the preset order of repair action nodes, generate a list of repair actions and automatically perform repair actions according to the order of each repair action node.

[0109] In this embodiment, the cost minimization principle needs to be determined in conjunction with the current GPU resource operating status. This means that, given the current GPU resource operating status and the historical repair methods being divided into multiple repair action nodes, the method that minimizes resource waste for each GPU operating state is selected based on the statistical analysis of these multiple repair action nodes. The repair action list includes the order and specific operation method of each repair action node, allowing the drive engine to automatically repair faults according to the repair action list. Successfully repaired cases are sent to the fault analysis and localization model for training, optimizing the model's fault root cause matching. If the repair fails, a repair failure message is sent to the user terminal requesting manual intervention.

[0110] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0111] In one embodiment, an AI-assisted GPU virtualization fault root cause analysis and self-healing system is provided, which corresponds to an AI-assisted GPU virtualization fault root cause analysis and self-healing method described in the above embodiment. The AI-assisted GPU virtualization fault root cause analysis and self-healing system includes:

[0112] The data acquisition module is used to collect multi-source heterogeneous data at different levels in the GPU virtualization environment in real time. The multi-source heterogeneous data includes performance indicators, configuration information, log information, and topology information.

[0113] The anomaly detection module is used to mark the simultaneously collected multi-source heterogeneous data with a unified timestamp and corresponding resource tags, and send them to the preset fault analysis and location model. The fault analysis and location model analyzes and determines in real time whether there are anomalies in the multi-source heterogeneous data and generates anomaly detection information.

[0114] The root cause localization module is used to determine the fault information of multi-source heterogeneous data when the anomaly judgment information indicates that an anomaly exists, and output several candidate root causes of the fault based on the fault information.

[0115] The reasoning and verification module is used to perform logical verification of the selected root causes of faults based on the preset fault reasoning graph, and output the root cause information of faults with a confidence level higher than the confidence level threshold; and generate a repair strategy based on the root cause information of faults to automatically repair the fault information.

[0116] Optionally, the anomaly detection module includes:

[0117] The feature acquisition submodule is used by the fault analysis and localization model to acquire a set of vector features from multi-source heterogeneous data.

[0118] The feature comparison submodule is used to perform feature comparison on the historical vector features pre-stored in the vector feature set; the historical vector features include abnormal historical vector features and normal historical vector features;

[0119] The approximity judgment submodule is used to determine whether there are anomalies in the current vector feature set based on the approximation between the judgment vector feature set and the historical vector features, and to generate anomaly judgment information.

[0120] Optionally, the root cause localization module includes:

[0121] An anomaly vector identification submodule is used by the fault analysis and localization model to identify anomaly vectors in the vector feature set and locate the corresponding fault information based on the anomaly vectors.

[0122] The classic anomaly detection submodule is used to determine whether there are any classic anomaly vector features in the anomaly history vector features that have a similarity to the current anomaly vector that is greater than or equal to the maximum threshold.

[0123] The first root cause determination submodule is used to, if any, select the classical root cause corresponding to the classical anomaly vector features as the unique candidate root cause of the current fault information.

[0124] The second root cause determination submodule is used to determine, if none exists, whether there is an approximate abnormal vector feature in the abnormal historical vector features that is greater than or equal to the minimum threshold with respect to the current abnormal vector, and to take the fault root cause corresponding to the approximate abnormal vector feature as the candidate fault root cause.

[0125] Optionally, the inference verification module includes:

[0126] The first reasoning submodule is used to assign the highest confidence to the candidate fault root cause and output it as the final confirmed fault root cause when the candidate fault root cause is the only classic root cause.

[0127] The second reasoning submodule is used to obtain the vector feature set of each candidate fault root cause when there are several candidate fault root causes.

[0128] The verification submodule is used to verify each feature vector in the vector feature set with the corresponding logical node based on the fault reasoning graph, and output the confidence of each candidate fault root cause. The types of logical nodes include hardware component nodes, software entity nodes, configuration item nodes, and error code nodes.

[0129] Optionally, the verification submodule includes:

[0130] The subgraph generation unit is used to map the fault information corresponding to the candidate fault root cause to the fault node of the fault reasoning graph, and filter logical nodes based on preset relational rules to generate associated subgraphs. Each logical node is pre-bound with a corresponding vector interval.

[0131] The confidence acquisition unit is used to compare each vector feature in the vector feature set with the vector interval of the corresponding logical node to generate the confidence of the logical node.

[0132] The root cause calculation unit is used to output the confidence level of each candidate root cause based on the confidence level of the faulty node and each logical node.

[0133] The root cause determination unit is used to identify whether the confidence level of each candidate fault root cause is higher than the confidence level threshold, and outputs fault root cause information with a confidence level higher than the confidence level threshold.

[0134] Optionally, the inference verification module also includes:

[0135] The repair identification submodule is used to obtain the historical repair methods corresponding to the output fault root cause based on the output fault root cause information.

[0136] The repair list generation submodule is used to determine the repair cost of the acquired historical repair methods based on the current GPU resource running status; and to generate a list of repair actions to execute the historical repair methods.

[0137] Optionally, the fix list generation submodule includes:

[0138] The node identification unit is used to individually decompose each historical repair method based on the cost minimization rule to obtain several repair action nodes with a preset order.

[0139] The cost assessment unit is used to determine the impact range and execution cost of each repair action node based on the current GPU resource operating status.

[0140] The repair method generation unit is used to statistically analyze the impact range and execution cost of each repair action node in each historical repair method, and to select the historical repair method with the lowest repair cost.

[0141] The repair unit is used to generate a list of repair actions based on a preset order of repair action nodes, and to automatically perform repair actions in the order of each repair action node.

[0142] For specific limitations regarding an AI-assisted GPU virtualization fault root cause analysis and self-healing system, please refer to the limitations of an AI-assisted GPU virtualization fault root cause analysis and self-healing method described above, which will not be repeated here. The modules in the aforementioned AI-assisted GPU virtualization fault root cause analysis and self-healing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the memory of the computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0143] In one embodiment, a computer device, which may be a server, is provided. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an AI-assisted GPU virtualization fault root cause analysis and self-healing method.

[0144] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an AI-assisted GPU virtualization fault root cause analysis and self-healing method.

[0145] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements an AI-assisted GPU virtualization fault root cause analysis and self-healing method.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0148] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An AI-assisted GPU virtualization fault root cause analysis and self-healing method, characterized in that, The method comprises the steps of: real-time collection of multi-source heterogeneous data at different levels in a GPU virtualization environment, the multi-source heterogeneous data including performance indicators, configuration information, log information, and topology information; labeling the simultaneously collected multi-source heterogeneous data with a uniform timestamp and corresponding resource labels, and sending the multi-source heterogeneous data to a preset fault analysis and positioning model, the fault analysis and positioning model performing real-time analysis and judgment on the multi-source heterogeneous data to determine whether the multi-source heterogeneous data is abnormal, and generating abnormality judgment information; when the abnormality judgment information indicates that the multi-source heterogeneous data is abnormal, the fault analysis and positioning model judges fault information of the multi-source heterogeneous data, and outputs a plurality of candidate fault root causes based on the fault information; based on a preset fault reasoning graph, logical verification of the candidate fault root causes is performed on corresponding multi-source heterogeneous data, and fault root cause information with a confidence level higher than a confidence threshold is outputted; and a repair strategy is generated based on the fault root cause information to automatically repair the fault information.

2. The AI-assisted GPU virtualization fault root cause analysis and self-healing method according to claim 1, characterized in that, The step of the fault analysis and positioning model performing real-time analysis and judgment on the multi-source heterogeneous data to determine whether the multi-source heterogeneous data is abnormal, and generating abnormality judgment information, comprises: the fault analysis and positioning model acquires a vector feature set of the multi-source heterogeneous data; feature comparison is performed on historical vector features pre-stored in the vector feature set, the historical vector features including abnormal historical vector features and normal historical vector features; based on the approximation degree of the vector feature set and the historical vector features, it is determined whether the current vector feature set is abnormal, and abnormality judgment information is generated.

3. The AI-assisted GPU virtualization fault root cause analysis and self-healing method of claim 2, wherein, The step of, when the abnormality judgment information indicates that the multi-source heterogeneous data is abnormal, the fault analysis and positioning model judges fault information of the multi-source heterogeneous data, and outputs a plurality of candidate fault root causes, comprises: the fault analysis and positioning model identifies an abnormal vector in the vector feature set, and locates corresponding fault information based on the abnormal vector; it is determined whether there is a classical abnormal vector feature in the abnormal historical vector features, the approximation degree of which is greater than or equal to a maximum threshold value, with respect to the current abnormal vector; if so, a classical root cause corresponding to the classical abnormal vector feature is taken as the only candidate fault root cause of the current fault information; if not, it is determined whether there is an approximate abnormal vector feature in the abnormal historical vector features, the approximation degree of which is greater than or equal to a minimum threshold value, with respect to the current abnormal vector, and a fault root cause corresponding to the approximate abnormal vector feature is taken as a candidate fault root cause.

4. The AI-assisted GPU virtualization fault root cause analysis and self-healing method according to claim 3, characterized in that, In the step of, based on a preset fault reasoning graph, logical verification of the candidate fault root causes is performed on corresponding multi-source heterogeneous data, and fault root cause information with a confidence level higher than a confidence threshold is outputted, comprises: when the candidate fault root cause is a unique classical root cause, the candidate fault root cause is assigned a highest confidence level and outputted as a final confirmed fault root cause; when the candidate fault root cause includes a plurality of candidate fault root causes, a vector feature set of each candidate fault root cause is acquired; based on the fault reasoning graph, each feature vector in the vector feature set is verified with a corresponding logical node, and the confidence level of each candidate fault root cause is outputted, the type of the logical node including a hardware component node, a software entity node, a configuration item node, and an error code node.

5. The AI-assisted GPU virtualization fault root cause analysis and self-healing method according to claim 4, characterized in that, The step of verifying each feature vector in the vector feature set with the corresponding logical node and outputting the confidence of each candidate fault root cause based on the fault reasoning graph, comprises: mapping the fault information corresponding to the candidate fault root cause to the fault node of the fault reasoning graph, and filtering the logical nodes based on the preset relationship rules to generate an associated subgraph, each logical node being pre-bound with a corresponding vector interval; comparing each vector feature in the vector feature set with the vector interval of the corresponding logical node to generate the confidence of the logical node; outputting the confidence of each candidate fault root cause based on the fault node and the confidence of each logical node; identifying whether the confidence of each candidate fault root cause is higher than the confidence threshold, and outputting the fault root cause information with a confidence higher than the confidence threshold.

6. The AI-assisted GPU virtualization fault root cause analysis and self-healing method according to claim 4, characterized in that, The step of generating a repair strategy based on the fault root cause information to automatically repair the fault information, comprises: based on the output fault root cause information, obtaining the historical repair means corresponding to the output fault root cause; based on the current GPU resource running state, judging the repair cost of the obtained historical repair means; and generating a repair action list to execute the historical repair means.

7. The AI-assisted GPU virtualization fault root cause analysis and self-healing method according to claim 6, characterized in that, The step of generating a repair strategy based on the fault root cause information to automatically repair the fault information, comprises: based on the cost minimization rule, repairing and disassembling each obtained historical repair means to obtain a plurality of repair action nodes with a preset order; based on the current GPU resource running state, judging the influence range and execution cost of each repair action node; statistically analyzing the influence range and execution cost of each repair action node in each historical repair means, and screening out the historical repair means with the lowest repair cost; based on the repair action nodes with the preset order, generating a repair action list, and automatically repairing the action according to the order of each repair action node. The step of generating a repair strategy based on the fault root cause information to automatically repair the fault information, comprises:

8. An AI-assisted GPU virtualization fault root cause analysis and self-healing system, characterized in that, a data collection module for collecting real-time multi-source heterogeneous data at different levels in a GPU virtualization environment, the multi-source heterogeneous data including performance indicators, configuration information, log information, and topology information; an abnormality judgment module for marking the simultaneously collected multi-source heterogeneous data with a unified timestamp and corresponding resource label, and sending it to a preset fault analysis and positioning model, which analyzes and judges whether the multi-source heterogeneous data is abnormal in real time, and generates abnormality judgment information; a root cause positioning module for judging the fault information of the multi-source heterogeneous data based on the fault analysis and positioning model when the abnormality judgment information is abnormal, and outputting a plurality of candidate fault root causes based on the fault information; an inference verification module for logically verifying the candidate fault root causes based on the preset fault reasoning graph, and outputting the fault root cause information with a confidence higher than the confidence threshold; and generating a repair strategy based on the fault root cause information to automatically repair the fault information. The processor executes the computer program to realize the steps of the AI-assisted GPU virtualization fault root cause analysis and self-healing method according to any one of claims 1 to 7.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the AI-assisted GPU virtualization fault root cause analysis and self-healing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the AI auxiliary-based GPU virtualization fault root cause analysis and self-recovery method according to any one of claims 1 to 7.

Images