A computer GPU computing power detection and maintenance method and system based on data analysis

Abstract

The application discloses a kind of computer GPU computing power detection maintenance method and system based on data analysis, it is related to GPU detection technical field, the limitation of breaking through single point observation and static threshold, by constructing the individualized dynamic efficiency baseline of division work load type, in combination with the correlation analysis of multidimensional time series data, accurately capture the dynamic change and deep coupling relationship of GPU running state, effectively avoid false alarm and miss report problem, realize early identification to initial performance degradation and potential fault, without waiting to problem deterioration can be active perception;It has precise causal inference ability simultaneously, by predefining fault chain model and real-time data feature matching, can locate fault root cause and the development stage where it is, replace artificial investigation, substantially shorten average repair time;While relying on fault chain modeling, identify fault development trend in advance, seize the best intervention opportunity, avoid slight problem evolves into serious business interruption.

Description

Technical Field

[0001] This invention relates to the field of GPU testing technology, and specifically to a method and system for testing and maintaining the computing power of computer GPUs based on data analysis. Background Technology

[0002] In modern computing, especially in cutting-edge applications such as artificial intelligence, machine learning, big data analytics, and high-performance computing, graphics processing units (GPUs) have become an indispensable core computing power support. Their powerful parallel computing capabilities have greatly promoted technological progress and industrial development, and they play an increasingly important role in cloud computing, edge computing, and various terminal devices. Therefore, ensuring a stable and efficient supply of GPU computing power, and accurately monitoring and promptly maintaining its operational status, is crucial for ensuring business continuity, improving overall system performance, and optimizing operating costs.

[0003] Existing technologies, such as the invention application patent with announcement number CN115827402A, disclose a method for detecting and analyzing the graphics processing performance of GPU servers based on pipeline control. It can be seen from the existing technologies that the methods for detecting and maintaining the computing power of computer GPUs usually adopt a strategy based on a combination of hardware sensors and operating system-level monitoring tools. These methods mainly collect various operating indicators of the target GPU periodically. Once any indicator exceeds its corresponding threshold range, an alarm mechanism will be triggered immediately to prompt maintenance personnel to intervene manually. However, with the increasing complexity of modern computing architecture, the diversification of GPU load types, and the rapid expansion of data center scale, the inherent limitations of the above-mentioned traditional detection and maintenance methods based on single indicator thresholds and passive responses have gradually been exposed, making them inadequate in dealing with current and future challenges. Specifically, this is reflected in the following aspects: (1) Relying on single-point observation and static threshold judgment, it is difficult to adapt to the dynamic coupling operating environment of GPUs. GPU health is related to many factors such as workload, resource allocation, and hardware aging. Static thresholds cannot capture the deep correlation and dynamic changes of multidimensional data, and are prone to misjudgment, resulting in alarm fatigue or missing early performance degradation and potential faults. They can only be passively detected after the problem has significantly worsened.

[0004] (2) It focuses on state description but lacks causal inference ability, resulting in low fault diagnosis efficiency. Traditional systems can only inform about abnormal indicators, but cannot locate the root cause. Furthermore, they do not model the fault chain, relying on the experience of maintenance personnel for manual troubleshooting, which leads to excessively long average repair time. They also make it difficult to identify the development trend in the early stage of the fault, thus missing the best intervention opportunity.

[0005] (3) It is based on passive alarms and lacks proactive prevention and intelligent decision-making capabilities. Maintenance work lags behind the occurrence of faults, there is no early warning mechanism, and it cannot automatically generate the optimal maintenance strategy by combining cluster resource status and business priorities. This not only affects the continuity of high-requirement businesses, but also fails to give full play to the performance of GPU resources, increasing operational risks and costs. Summary of the Invention

[0006] The purpose of this invention is to provide a computer GPU computing power detection and maintenance method and system based on data analysis, which solves the problems existing in the background technology.

[0007] To solve the above technical problems, the present invention adopts the following technical solution: The first aspect of the present invention provides a computer GPU computing power detection and maintenance method based on data analysis, including: S1, multi-dimensional operation data acquisition: acquiring historical periodic and real-time multi-dimensional time series data of the target GPU when running workload through a data acquisition module, wherein the multi-dimensional time series data includes workload data characterizing computing power demand, physical sensor data characterizing hardware status, and error data characterizing operation stability.

[0008] S2. Establishment of Individualized Dynamic Efficiency Baseline: Based on the multi-dimensional time-series data of historical periods collected in S1, an individualized dynamic efficiency baseline is constructed for the target GPU. The dynamic efficiency baseline is used to describe the quantitative mapping relationship model between computing power output and power consumption, temperature, execution time and error data when the target GPU processes a specific workload under healthy conditions.

[0009] S3, Computing Power Supply Efficiency Analysis and Fault Chain Reasoning: Compare and analyze the real-time multi-dimensional time-series data collected in S1 with the dynamic efficiency baseline established in S2, and execute the sub-steps of S31, Efficiency Decline Analysis and S32, Fault Chain Matching Analysis.

[0010] S4. Comprehensive Health Score and Risk Quantification: Based on the analysis results of S3, calculate the comprehensive health score of the target GPU. The health score includes the rate of performance degradation, the stage of the failure chain, and the potential impact on business continuity.

[0011] S5. Maintenance Decision Generation and Triggering: Based on the health score obtained in S4, combined with the cluster resource status and business scheduling strategy, a targeted maintenance decision is automatically generated and output. The maintenance decision includes specific maintenance actions, recommended execution time windows, and task migration schemes to ensure business continuity.

[0012] A second aspect of the present invention provides a system for performing the data analysis-based computer GPU computing power detection and maintenance method described in the present invention, comprising: a data acquisition module for acquiring historical periodic and real-time multidimensional time-series data of the target GPU when running a workload.

[0013] Individualized Dynamic Efficiency Baseline Establishment Module: Based on the collected multi-dimensional time-series data from historical periods, an individualized dynamic efficiency baseline is built for the target GPU.

[0014] The computing power supply efficiency analysis and fault chain reasoning module compares and analyzes the collected real-time multi-dimensional time-series data with the established dynamic efficiency baseline, and performs sub-steps of efficiency degradation analysis and fault chain matching analysis.

[0015] The comprehensive health score and risk quantification module is used to calculate the comprehensive health score of the target GPU, which includes the rate of performance degradation, the stage of the failure chain, and the potential impact on business continuity.

[0016] The maintenance decision generation and triggering module is used to automatically generate and output targeted maintenance decisions based on health scores, combined with cluster resource status and business scheduling strategies.

[0017] The beneficial effects of the present invention are as follows: (1) The present invention breaks through the limitations of single-point observation and static threshold. By constructing individualized dynamic efficiency baselines for different workload types and combining the correlation analysis of multi-dimensional time series data, it accurately captures the dynamic changes and deep coupling relationship of GPU running status, effectively avoids false alarms and missed alarms, and realizes early identification of initial performance degradation and potential faults. It can proactively detect problems without waiting for them to worsen.

[0018] (2) This invention has the ability to accurately infer cause and effect. By matching the predefined fault chain model with real-time data features, it can locate the root cause of the fault and its development stage, replacing manual investigation and significantly shortening the average repair time. At the same time, based on fault chain modeling, it can identify the fault development trend in advance, seize the best intervention time, and prevent minor problems from evolving into serious business interruptions.

[0019] (3) This invention realizes the upgrade from passive alarm to active decision-making. Based on comprehensive health score and risk quantification, combined with cluster resource status and business priority, it automatically generates maintenance actions, execution time windows and task migration plans. It achieves proactive prevention through early warning and ensures the continuity of high-requirement business through intelligent maintenance decision-making, giving full play to the efficiency of GPU resources and reducing operational risks and costs. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention.

[0022] Figure 2 This is a schematic diagram of the system structure connection of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Reference Figure 1 As shown, this invention provides a data analysis-based method for detecting and maintaining the computing power of a computer GPU. It overcomes the limitations of single-point observation and static thresholds by constructing individualized dynamic efficiency baselines for different workload types. Combined with correlation analysis of multi-dimensional time-series data, it accurately captures the dynamic changes and deep coupling relationships of the GPU's operating status, effectively avoiding false alarms and missed alarms. This enables early identification of initial performance degradation and potential faults, allowing proactive detection without waiting for problems to worsen. The method includes: S1, Multi-dimensional operational data acquisition: A data acquisition module obtains historical and real-time multi-dimensional time-series data of the target GPU during workload operation. This multi-dimensional time-series data includes workload data representing computing power requirements, physical sensor data representing hardware status, and error data representing operational stability.

[0025] It should be noted that workload data includes load type labels, load intensity parameters, resource requirement characteristics, and execution time; physical sensor data includes power consumption data, parameters of different temperature types, and other hardware status data; and error data includes the frequency of different error types.

[0026] S2. Establishment of Individualized Dynamic Efficiency Baseline: Based on the multi-dimensional time-series data of historical periods collected in S1, an individualized dynamic efficiency baseline is constructed for the target GPU. The dynamic efficiency baseline is used to describe the quantitative mapping relationship model between computing power output and power consumption, temperature, execution time and error data when the target GPU processes a specific workload under healthy conditions. For example, under 1080P graphics rendering load, the specific quantitative correlation between computing power output such as the number of frames rendered per second and the corresponding healthy power consumption range, temperature range, single frame completion time and error data will be clearly defined.

[0027] In a specific embodiment of the present invention, the step of S2 in constructing an individualized dynamic efficiency baseline for the target GPU includes the following steps: S21, historical data segmentation processing: the multi-dimensional time-series data of the historical period collected in S1 is first classified according to the workload type, and then the time-series data corresponding to each type of workload is divided into multiple data segments according to the time series, and each data segment is normalized.

[0028] For example, the data can be divided according to the time series, such as dividing the data into segments based on continuous operation data for 7 days. The normalization method can be the Min-Max normalization method, which maps physical quantities such as power consumption, temperature, and execution time in each data segment to the interval [0, 1]. The formula is: normalized value = (original value - minimum value of the data segment) / (maximum value of the data segment - minimum value of the data segment). Alternatively, the Z-Score normalization method can be selected.

[0029] S22. Baseline Model Training: For the normalized data corresponding to each type of workload, a machine learning algorithm is used to train and generate an initial dynamic efficiency baseline model for the corresponding workload type. The initial dynamic efficiency baseline model includes the power consumption range, the safe range for different temperature types, the execution time range, and the allowable parameter range for different error types corresponding to different workload data.

[0030] It should be noted that the workload data, such as 1080P resolution graphics rendering, deep learning training, etc., the different temperature types, such as core temperature and memory temperature, and the different error types, include but are not limited to memory errors, computing unit errors, and bus transmission errors.

[0031] For example, machine learning algorithms can employ regression algorithms based on gradient boosting trees or time-series prediction algorithms based on long short-term memory networks to establish a correlation model between computing power output and parameters such as power, temperature, execution time, and error type. For periodic workload scenarios, time-series prediction algorithms based on long short-term memory networks are preferred, while for non-periodic workload scenarios, regression algorithms based on gradient boosting trees are preferred. Furthermore, details regarding algorithm parameter debugging and training iteration termination conditions are well-known to those skilled in the art, and no additional limitations are required to ensure the implementation and reproducibility of the solution.

[0032] S23. Baseline Model Validation: The initial dynamic efficiency baseline model corresponding to each type of workload is validated by cross-validation method, and the model that meets the preset accuracy requirements under each type of workload is selected as the final dynamic efficiency baseline for the corresponding workload type.

[0033] For example, preset accuracy requirements such as the average absolute error between the power consumption value predicted by the initial dynamic efficiency baseline model and the actual historical data being less than or equal to 5%, the maximum deviation between the predicted temperature value and the actual value being less than or equal to 3 degrees Celsius, the relative error between the predicted execution time value and the actual value being ≤8%, the deviation between the predicted frequency range of each type of error and the actual value being less than or equal to 1 time / hour, and the maximum deviation between the predicted frequency range and the actual error frequency of the corresponding type of error in the historical normal samples not exceeding 1 time / hour, can be set by the user according to the usage scenario, and are not restricted here.

[0034] S24. Baseline Model Update: Periodically collect new data on the target GPU running for each type of workload, and incrementally update the dynamic efficiency baseline for the corresponding workload type based on the new data to ensure that the dynamic efficiency baseline model for each type of workload can reflect the latest state of the target GPU.

[0035] S25. Baseline Model Storage: Store the final dynamic efficiency baseline corresponding to each type of workload in the database for subsequent real-time data analysis.

[0036] In a specific embodiment of the present invention, the baseline model update in S2 includes the following steps: S241, new data acquisition and preprocessing: periodically acquire new data of the target GPU running under various workloads, classify the data according to the workload type, and perform preprocessing operations on the new data corresponding to each type of workload. The preprocessing operations include data cleaning, normalization processing and time series alignment.

[0037] S242. Incremental training data construction: The preprocessed new data corresponding to each type of workload is merged with the historical data, and incremental training datasets are constructed according to the workload type. The incremental training datasets are used to update the dynamic efficiency baseline model of the corresponding workload type.

[0038] S243. Incremental Model Training: Use machine learning algorithms to train the incremental training dataset for each type of workload, generating an updated dynamic efficiency baseline model for the corresponding workload type.

[0039] S244. Model Validation and Selection: The updated dynamic efficiency baseline model for each type of workload is validated using cross-validation. If the validation results meet the preset accuracy requirements, the updated model is adopted as the new dynamic efficiency baseline model for the corresponding workload type.

[0040] S245, Model Storage and Synchronization: Store the new dynamic efficiency baseline model for each type of workload in the database and synchronize it to all relevant analysis modules to ensure the consistency of real-time data analysis.

[0041] S3, Computing Power Supply Efficiency Analysis and Fault Chain Reasoning: Compare and analyze the real-time multi-dimensional time-series data collected in S1 with the dynamic efficiency baseline established in S2, and execute the sub-steps of S31, Efficiency Decline Analysis and S32, Fault Chain Matching Analysis.

[0042] In a specific embodiment of the present invention, the specific content of S31, efficiency degradation analysis, includes: identifying whether the computing power supply efficiency of the target GPU has degraded under the same or similar types of workload. The degradation is manifested as the power consumption required to complete the task exceeding the power consumption range, different temperature types exceeding the corresponding safe range, execution time exceeding the range, or the frequency of different types of errors exceeding the allowable range.

[0043] It should be noted that the specific identification method for identifying the same or similar types of workloads is as follows: extract key indicators that distinguish the types of workloads from the workload data representing computing power requirements obtained from S1, including but not limited to workload type labels, workload intensity parameters, and resource requirement characteristics.

[0044] For example, the load type label is directly associated with the identifier of the task attribute, the load intensity parameter reflects the indicator of the task magnitude, and the resource requirement feature reflects the load's demand pattern on GPU hardware.

[0045] If the target GPU's current load has the same load type label as a historical load, the load intensity parameter difference rate is between 10% and 30%, and the resource requirement feature similarity is greater than or equal to 80%, then it is determined that the target GPU's current load is the same as or similar to that historical load. Furthermore, a comparison is made based on the target GPU's current power consumption, parameters for different temperature types, execution time, and the frequency of different types of errors, as well as the dynamic efficiency baseline of that historical load, to determine whether the target GPU's computing power supply efficiency has degraded. In practical application scenarios, those skilled in the art can adaptively modify this based on the application scenario requirements; no limitations are imposed here.

[0046] The specific sub-steps include: S311, data alignment processing: aligning the real-time multi-dimensional time-series data of the target GPU collected in S1 with the dynamic efficiency baseline of the same or similar workload type established in S2 on the time axis to ensure that the time dimensions of the two are consistent.

[0047] S312. Efficiency Deviation Calculation: Calculate the deviation value between real-time multi-dimensional time-series data and dynamic efficiency baseline. The deviation value includes power consumption deviation, deviation for different temperature types, execution time deviation, and deviation for each error type.

[0048] S313. Degradation Detection: If any deviation value exceeds the preset threshold, the computing power supply efficiency of the target GPU is determined to have degraded.

[0049] It should be noted that the preset thresholds are determined based on historical health data statistics for the target GPU's corresponding workload type. Specifically, they are the upper limit of the 95% confidence interval of the deviation of each indicator under the healthy state of the workload. For example, the preset threshold for power consumption deviation of 1080P rendering workload is 20%, and the preset threshold for core temperature deviation is 5℃. Alternatively, they can be set by those skilled in the art in combination with the GPU hardware specifications, such as the safe deviation range recommended by the manufacturer. No specific restrictions are imposed here.

[0050] S314. Degradation Assessment: Based on the magnitude of each deviation value, assess the degree of degradation of the target GPU's computing power supply efficiency and generate a degradation report.

[0051] It should be noted that the content of S314 above specifically involves: First, performing individual deviation quantization, that is, mapping each deviation value to a quantization score, based on various deviation-score mapping tables stored in the database. Second, assigning weights to each deviation based on its impact on computing power supply efficiency, based on various deviation-weight mapping tables stored in the data warehouse. Then, performing a comprehensive score calculation, calculating the contribution of each deviation by multiplying the individual quantization score by its corresponding weight, summing them to obtain a comprehensive degradation score, and mapping the target GPU's computing power supply efficiency degradation degree to the comprehensive degradation score range corresponding to the preset computing power supply efficiency degradation degree.

[0052] For example, each deviation value is mapped to a quantified score. For instance, a power consumption deviation exceeding the baseline allowable range by 10%-30% earns 2 points, exceeding it by 30%-50% earns 4 points, an uncorrectable memory error deviation of 1-3 times / hour earns 4 points, and a core temperature deviation of 2-5℃ earns 2 points, etc. Weights are assigned based on the degree of impact of each deviation on computing power supply efficiency. For example, uncorrectable memory error deviation has a weight of 0.3, core temperature deviation has a weight of 0.25, compute unit error deviation has a weight of 0.2, execution time deviation has a weight of 0.15, and power consumption deviation and bus transmission error deviation each have a weight of 0.05.

[0053] It should also be noted that the decline report includes the specific value of each deviation, the individual quantitative score, the weight, the comprehensive score, and the degree of decline in computing power supply efficiency.

[0054] S315. Decline Record Storage: Store decline reports in a database for subsequent failure chain matching analysis.

[0055] In a specific embodiment of the present invention, the specific content of S32, fault chain matching analysis, includes: when efficiency decline or abnormal indicators are identified, matching the linkage relationship between multiple indicators in real-time data with a predefined fault chain model to locate the root cause of efficiency decline or abnormality.

[0056] The specific sub-steps include: S321. Loading Fault Chain Models: Load predefined fault chain models from the database. The fault chain models describe the chronological order and logical relationship of a series of symptoms caused by different root causes.

[0057] It should be noted that the fault chain model includes, but is not limited to, the memory aging fault chain model, the cooling fan fault chain model, and the power supply module abnormal fault chain model.

[0058] S322. Real-time data feature extraction: Extract the linkage relationship features between multiple indicators from the real-time multi-dimensional time-series data of the target GPU. The linkage relationship features include correlation coefficients and changing trends.

[0059] It should be noted that the real-time data feature extraction in S3 includes the following steps: S3221, Indicator Data Grouping: Divide multiple indicators in real-time multidimensional time series data into several data groups according to their correlation. Each data group contains interrelated indicators.

[0060] S3222, Feature Statistical Analysis: Perform statistical analysis on the indicators in each data group to extract statistical features, including mean, variance, maximum value and minimum value.

[0061] S3223. Trend Calculation: Based on time series data, calculate the trend of change of indicators in each data group, and the trend of change is expressed in the form of slope or growth rate.

[0062] S3224. Correlation coefficient calculation: Calculate the correlation coefficient between indicators in different data groups to quantify the linkage between indicators.

[0063] It should be noted that in the S3224 correlation coefficient calculation step, the correlation coefficient between different data groups can be solved using correlation analysis techniques known in the field of computer data processing, including but not limited to conventional methods such as the Pearson correlation coefficient method and the Spearman rank correlation coefficient method. The core principles of the above methods, such as the coefficient derivation logic based on covariance and standard deviation, the ranking calculation rules of rank correlation, the specific calculation process such as data standardization preprocessing, outlier removal strategies, and parameter settings such as significance test thresholds, are all conventional application methods of those skilled in the art in data association analysis scenarios. The calculation process of this step can be accurately reproduced without additional detailed explanation, and it does not affect the implementation effect of the overall technical solution of the present invention.

[0064] S3225. Feature Storage and Indexing: Store the extracted statistical features, trends, and correlation coefficients in the database, and create feature indexes for subsequent matching analysis.

[0065] S323. Matching degree calculation: The extracted linkage relationship features are matched with the symptom features of each fault chain model at each fault development stage, and the matching degree value between the target GPU and each fault chain model at each fault development stage is calculated.

[0066] It should be noted that the extracted linkage features, namely the correlation coefficients and trends of multiple indicators obtained in S32, are compared one by one with the predefined symptom features corresponding to each stage of fault development in each fault chain model. These symptom features are the reasonable range of correlation coefficients and the slope interval of trends that multiple indicators should exhibit under the corresponding fault. A similarity calculation method known in the field of computer data processing is used, including but not limited to cosine similarity algorithms and Euclidean distance algorithms. The comparison results between the real-time linkage features and the symptom features corresponding to each stage of fault development in a single fault chain model are converted into a quantitative matching value. The value range is exemplarily set to 0-1, with a value closer to 1 indicating a higher degree of fit between the real-time features and the model's symptom features. The specific parameter settings for similarity calculation, such as the vector construction rules for cosine similarity and the dimensional weight allocation for Euclidean distance, are conventional methods used by those skilled in the art, and no additional limitations are required to ensure the reproducibility of the calculation process.

[0067] S324. Root cause location: If the matching degree between the target GPU and a certain fault chain model at a certain fault development stage is greater than a preset threshold, then the fault chain model is determined to be the fault chain of the current target GPU, and the root cause of efficiency degradation or abnormality and the fault development stage of the fault chain in which the target GPU is located are located.

[0068] S325, Root Cause Record Storage: Store the identified root cause and the specific stage of the fault chain in which the target GPU is located in the database for subsequent health scoring and risk quantification.

[0069] In a specific embodiment of the present invention, loading the fault chain model in S3 includes the following steps: S3211, Model library initialization: Loading a predefined fault chain model library from the database, wherein the model library contains multiple fault chain models that describe the order and logical relationship of the occurrence of symptoms caused by different root causes.

[0070] S3212, Model Version Verification: Verify the version of the loaded fault chain model library to ensure that the loaded model version is the latest version.

[0071] S3213. Model compatibility check: Based on the hardware configuration and operating environment of the target GPU, select the fault chain model that is suitable for the target GPU.

[0072] It should be noted that the hardware configuration is obtained through the hardware information interface built into the target GPU, and the operating environment is obtained through the cluster GPU monitoring module or system logs.

[0073] S3214. Model Loading Priority Ranking: Based on the historical matching success rate and applicable scope of the fault chain model, the selected fault chain models are prioritized.

[0074] Specifically, historical matching records of GPU faults of the same type are extracted from the database for each fault chain model, and the success rate is calculated. The success rate is calculated by dividing the number of successful matches by the total number of calls, and the fault chain models are sorted in descending order of success rate.

[0075] If multiple failure chain models have similar success rates, then compare the compatibility of each failure chain model with the target GPU and sort the failure chain models in descending order of compatibility.

[0076] The specific content of the fit between each fault chain model and the target GPU is as follows: The pre-defined GPU hardware parameters for each fault chain model are extracted from the database and compared with the target GPU hardware parameters. Matches are counted to obtain the number of matches. Considering the workload of the target GPU, if a fault chain model's usage scenario only includes the target GPU's workload type, the second fit is denoted as A; otherwise, it is denoted as A'. The number of matches and the second fit are weighted according to the cluster GPU usage characteristics, and a weighted sum is obtained to obtain the fit.

[0077] It should also be noted that the weights assigned according to the characteristics of cluster GPU usage are as follows: for example, the weight of hardware configuration is higher than that of scene adaptation.

[0078] S3215, Model Loading and Caching: Load the fault chain model according to priority and cache it in memory to improve the efficiency of real-time matching analysis.

[0079] This invention possesses accurate causal inference capabilities. By matching predefined fault chain models with real-time data features, it can pinpoint the root cause and development stage of a fault, replacing manual troubleshooting and significantly shortening the average repair time. Simultaneously, relying on fault chain modeling, it can identify fault development trends in advance, seize the optimal intervention opportunity, and prevent minor issues from escalating into serious business disruptions.

[0080] S4. Comprehensive Health Score and Risk Quantification: Based on the analysis results of S3, calculate the comprehensive health score of the target GPU. The health score includes the rate of performance degradation, the stage of the failure chain, and the potential impact on business continuity.

[0081] In a specific embodiment of the present invention, the comprehensive health score calculation in S4 includes the following steps: S41, performance degradation rate assessment: based on the efficiency degradation analysis results, the performance degradation rate of the target GPU is calculated, wherein the performance degradation rate is the weighted average of power consumption deviation, deviation of different temperature types, execution time deviation, and deviation of each error type per unit time.

[0082] S42. Fault Chain Stage Determination: Based on the root cause of the target GPU and the fault development stage of the fault chain in the fault chain matching analysis results, and combined with the risk weight values ​​of the root cause and fault development stage stored in the database, the risk weight value of the fault development stage of the target GPU is mapped.

[0083] It should be noted that the risk weight value for the root cause of a fault – the stage of fault development – ​​is a preset value that quantifies the risk level of different root causes of faults, such as memory aging, cooling fan failure, and power supply module malfunction, at each stage of fault development, such as the early, middle, and late stages. It is established based on historical fault data, combined with the actual impact of the fault on GPU computing power output and business continuity. For example, the risk of hardware damage is higher and the probability of business interruption is greater in the later stages of a fault, so the corresponding weight value is higher than in the early stages. Within the same stage, memory aging faults have a greater impact on computing power than fan failures, so a higher weight value is set. The specific values ​​and ranges of these weight values ​​can be adjusted by those skilled in the art based on the characteristics of the cluster GPU hardware and the needs of the business scenario to ensure accurate mapping of the risk level at different fault stages; these details will not be elaborated here.

[0084] S43. Business Impact Analysis: Evaluate the potential impact of the target GPU's performance degradation and failure chain stages on business continuity, with the potential impact quantified by the probability and duration of business interruption.

[0085] Specifically, the failure chain stage is correlated with the degree of performance degradation. Referring to historical service interruption records of similar GPUs at the same failure stage and performance degradation level, and using well-known probabilistic statistical methods, the probability of current service interruption is mapped. For example, the later the failure and the more severe the performance degradation, the higher the likelihood of service interruption due to hardware failure or insufficient computing power. Combining the maintenance action type corresponding to the failure development stage (e.g., hardware replacement required in the later stages, or only parameter adjustment required in the early stages), and referring to the historical execution time of similar maintenance actions, the dependence of the service on the GPU is added. For example, core services need to be prioritized for recovery. The estimated interruption duration is obtained by multiplying the service type carried by the target GPU with the interruption duration reduction coefficient corresponding to each service type stored in the database.

[0086] It should be noted that the interruption duration reduction coefficient for each business type is a quantitative parameter set according to the business's dependence on GPU, such as core / non-core and real-time / offline attributes. It is used to correct the basic interruption duration of maintenance actions and ensure that core businesses are restored first. The specific value can be adjusted by those skilled in the art in combination with the business scenario.

[0087] S44. Health Score Calculation: Based on the performance degradation rate, fault chain stage risk weight value, and business impact analysis results, a weighted summation method is used to calculate the comprehensive health score of the target GPU.

[0088] It should be noted that the weights are set based on business priority and GPU usage scenarios. For example, for real-time response businesses, the business impact weight is set to 0.4, the fault chain stage risk weight is set to 0.3, and the performance degradation rate weight is set to 0.3; for offline processing businesses such as offline deep learning training, the performance degradation rate weight is set to 0.4, the fault chain stage risk weight is set to 0.3, and the business impact weight is set to 0.3. The specific weights can be optimized by those skilled in the art based on historical maintenance results.

[0089] S45. Score Storage and Update: Store the comprehensive health score in the database and update the score value regularly based on the latest analysis results to reflect the real-time health status of the target GPU.

[0090] S5. Maintenance Decision Generation and Triggering: Based on the health score obtained in S4, combined with the cluster resource status and business scheduling strategy, a targeted maintenance decision is automatically generated and output. The maintenance decision includes specific maintenance actions, recommended execution time windows, and task migration schemes to ensure business continuity.

[0091] In a specific embodiment of the present invention, the generation of maintenance decision in S5 includes the following steps: S51, resource status assessment: obtain real-time status information of the cluster resources where the target GPU is located, including idle computing resources, network bandwidth and storage capacity.

[0092] S52. Business scheduling strategy analysis: Analyze the current business scheduling strategy and determine the priority of business continuity.

[0093] It should be noted that the specific implementation method of business scheduling strategy analysis is as follows: Through the business scheduling management module, the current workload type carried by the target GPU is extracted, such as 1080P resolution graphics rendering, deep learning training, etc., and each workload type is associated with its corresponding business scenario to clarify the business attributes served by each workload, such as real-time response business, offline processing business, etc. Based on historical business operation data, the operational characteristics of the businesses associated with each workload type are analyzed to determine whether the business needs to continuously provide services to meet real-time requirements. Interruption of real-time response business is likely to affect user experience or business revenue, while offline processing business can be delayed and resumed. Combined with the pre-built business type-priority mapping table in the database, business continuity priorities are assigned accordingly. Simultaneously, the peak operation characteristics of each business are identified to determine typical periods when the business is in a high-load operation state. Finally, the business continuity priority assignment results and peak period characteristics are integrated to form a business scheduling strategy analysis report, providing a business-level decision-making basis for subsequent maintenance execution time window optimization and task migration priority setting.

[0094] It should also be noted that the mapping rules of the business type-priority mapping table are set based on the characteristics of the impact of business interruption and whether continuous service is required in the historical business operation data. The specific settings, such as the quantitative standards for the impact of interruption and the judgment logic for continuous service requirements, are well known to those skilled in the art and will not be described in detail here.

[0095] S53. Maintenance Action Selection: Based on the health score of the target GPU, and in conjunction with the maintenance action and health score mapping table stored in the database, a suitable maintenance action is selected. The maintenance actions include hardware replacement, software update, and parameter adjustment.

[0096] For example, if the target GPU health score is in a high range, such as slightly higher temperature or slightly excessive power consumption due to parameter configuration deviations, without any signs of hardware performance degradation, then parameter adjustment maintenance is selected. Specifically, this can involve adjusting hardware operating parameters such as the GPU core frequency threshold and the cooling fan speed trigger condition. If the health score is moderate, such as a slight increase in error frequency or minor fluctuations in computing power due to software version compatibility issues, without physical damage to the hardware itself, then software update maintenance is selected. Specifically, this can involve updating the GPU driver, optimizing the operating system kernel, or adjusting the computing power scheduling related software configuration. If the health score is in a low range, such as frequent errors and significant degradation in computing power due to hardware aging or physical damage, and parameter adjustments and software updates cannot improve the situation, then hardware replacement maintenance is selected. Specifically, this can involve replacing hardware components such as faulty memory chips, aging cooling modules, or abnormal power supply modules.

[0097] It should be noted that the mapping rules for the maintenance action and health score mapping table can be formulated by those skilled in the art in combination with the characteristics of the cluster GPU hardware and the needs of the business scenario. The specific adjustment details are well known to those skilled in the art and will not be described in detail here.

[0098] S54. Execution Time Window Optimization: Based on cluster resource status, business continuity priority, and maintenance actions, calculate the recommended execution time window for maintenance actions.

[0099] In a specific embodiment of the present invention, several time periods corresponding to non-core services are selected to exclude peak operating periods of core services, ensuring that maintenance actions do not interfere with the continuity of high-priority services. Within the aforementioned time periods corresponding to non-core services, the cluster resource availability periods of S51 are matched, i.e., the idle computing resources within the cluster are selected to meet the migration requirements of the target GPU task, i.e., the network bandwidth is uncongested and the storage capacity is suitable for the temporary storage of task data, to avoid maintenance interruptions or task migration failures due to insufficient resources. Based on the maintenance actions determined in S53, and combined with the normal duration of each maintenance action type stored in the data warehouse, an additional buffer time with a set threshold is reserved within the aforementioned matched time periods to cope with unexpected problems. Finally, the recommended maintenance action execution time window that meets the criteria is selected. The non-core services are specifically services with a priority lower than the separation priority between core and non-core services.

[0100] It should be noted that the target GPU task migration requirement specifically refers to the minimum resource standard required for the task to run normally when migrating the workload currently carried by the target GPU to other candidate target nodes within the cluster.

[0101] S55. Task migration plan generation: To ensure business continuity, a task migration plan is generated, which includes the task migration target node and migration path planning.

[0102] In a specific embodiment of the present invention, the task migration scheme generation in S5 includes the following steps: S551, target node screening: based on the real-time status information of the cluster resources where the target GPU is located, each candidate target node that meets the task migration requirements is screened out. The candidate target node must meet the requirements that its idle computing resources are higher than a preset resource threshold and its network bandwidth is higher than a preset bandwidth threshold.

[0103] S552. Migration Path Planning: Based on the network topology and current network traffic distribution of each candidate target node, calculate the optimal migration path from the node where the target GPU is located to each candidate target node. The optimal migration path aims to minimize migration latency.

[0104] Specifically, 1. Collect basic data through the cluster network management module: Based on the network topology, clarify the physical connection relationship between the node where the target GPU is located and the candidate target nodes, i.e., direct links, indirect links that require intermediate nodes for forwarding, link types such as Ethernet, InfiniBand, and the bandwidth limit of each link. Based on the current network traffic distribution, obtain the bandwidth occupied by each link, data transmission latency, and packet loss rate in real time to determine the available transmission capacity of the links.

[0105] 2. Enumerate feasible paths: Based on the above topology, list all potential transmission paths from the node where the target GPU is located to the candidate target node, and exclude invalid paths with link failure or bandwidth of 0.

[0106] 3. Optimal Path Selection: With minimizing migration latency as the optimization objective, a path latency calculation model is constructed. Combining available bandwidth, transmission delay, and intermediate node forwarding latency for each link, the estimated total migration latency for each feasible path is calculated. The path with the minimum total latency is selected as the optimal migration path. If multiple paths have latency differences less than a preset threshold, the historical link failure rate is used as a supplementary selection criterion, prioritizing the path with the highest stability to ensure smooth task migration.

[0107] S553. Migration Cost Assessment: For each candidate target node and its corresponding optimal migration path, assess the migration cost, which comprehensively considers migration latency, network resource utilization, and the impact of migration on business continuity.

[0108] Specifically, 1. The total migration delay is estimated based on the path calculated by S552. The longer the delay, the higher the corresponding cost score. In actual implementation, if the delay exceeds the preset threshold, the cost score will increase by a fixed proportion for each additional unit of delay.

[0109] 2. The network bandwidth utilization rate of the calculation path during the migration period is calculated as the migration bandwidth / total link bandwidth. The higher the utilization rate, such as exceeding the preset utilization threshold, the more serious the network resource squeeze on other services in the cluster will be, and the higher the corresponding cost score will be.

[0110] 3. Based on the priority of the business to be migrated determined by S52, the risk of interruption and the difficulty of ensuring data consistency in the migration of core businesses such as 1080P real-time rendering are higher, and the corresponding cost score is higher than that of non-core businesses such as offline training.

[0111] The migration latency, network resource utilization, and impact of migration on business continuity are each assigned a weight based on the adapted business scenario. The estimated migration cost of the optimal migration path for each candidate target node is obtained by weighted summation. The weights of the adapted business scenarios can be set by those skilled in the art based on the characteristics of the cluster business, and are not restricted here.

[0112] S554. Migration Scheme Determination: Based on the migration cost assessment results, select the candidate target node with the lowest migration cost and its corresponding optimal migration path as the final task migration scheme.

[0113] S555. Migration record storage: Store the final task migration plan in the database for use in subsequent maintenance decisions.

[0114] This invention upgrades from passive alarms to proactive decision-making. Based on comprehensive health scores and risk quantification, combined with cluster resource status and business priorities, it automatically generates maintenance actions, execution time windows, and task migration plans. It achieves proactive prevention through early warning and ensures high-demand business continuity through intelligent maintenance decision-making, fully leveraging GPU resource efficiency and reducing operational risks and costs.

[0115] Reference Figure 2 As shown, a second aspect of the present invention provides a system for performing the computer GPU computing power detection and maintenance method based on data analysis described in the present invention, comprising: a data acquisition module for acquiring historical periodic and real-time multidimensional time-series data of the target GPU when running a workload.

[0116] Individualized Dynamic Efficiency Baseline Establishment Module: Based on the collected multi-dimensional time-series data from historical periods, an individualized dynamic efficiency baseline is built for the target GPU.

[0117] The computing power supply efficiency analysis and fault chain reasoning module compares and analyzes the collected real-time multi-dimensional time-series data with the established dynamic efficiency baseline, and performs sub-steps of efficiency degradation analysis and fault chain matching analysis.

[0118] The comprehensive health score and risk quantification module is used to calculate the comprehensive health score of the target GPU, which includes the rate of performance degradation, the stage of the failure chain, and the potential impact on business continuity.

[0119] The maintenance decision generation and triggering module is used to automatically generate and output targeted maintenance decisions based on health scores, combined with cluster resource status and business scheduling strategies.

[0120] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

[0123] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A method for detecting and maintaining the computing power of computer GPUs based on data analysis, characterized in that, include: S1. Multidimensional operational data acquisition: The data acquisition module acquires historical and real-time multidimensional time-series data of the target GPU when running workloads. The multidimensional time-series data includes workload data representing computing power requirements, physical sensor data representing hardware status, and error data representing operational stability. S2. Establishment of Individualized Dynamic Efficiency Baseline: Based on the multi-dimensional time-series data of historical periods collected in S1, an individualized dynamic efficiency baseline is constructed for the target GPU. The dynamic efficiency baseline is used to describe the quantitative mapping relationship model between computing power output and power consumption, temperature, execution time and error data when the target GPU processes a specific workload under healthy conditions. S3, Computing power supply efficiency analysis and fault chain reasoning: Compare and analyze the real-time multi-dimensional time series data collected in S1 with the dynamic efficiency baseline established in S2, and execute the sub-steps of S31, efficiency degradation analysis and S32, fault chain matching analysis. S4. Comprehensive Health Score and Risk Quantification: Based on the analysis results of S3, calculate the comprehensive health score of the target GPU. The health score includes the rate of performance degradation, the stage of the failure chain, and the potential impact on business continuity. S5. Maintenance Decision Generation and Triggering: Based on the health score obtained in S4, combined with the cluster resource status and business scheduling strategy, a targeted maintenance decision is automatically generated and output. The maintenance decision includes specific maintenance actions, recommended execution time windows, and task migration schemes to ensure business continuity.

2. The method for detecting and maintaining computer GPU computing power based on data analysis according to claim 1, characterized in that, The steps in S2 for building an individualized dynamic efficiency baseline for the target GPU include: S21. Historical data segmentation: The multi-dimensional time series data of the historical period collected by S1 is first classified according to the workload type, and then the time series data corresponding to each type of workload is divided into multiple data segments according to the time series, and each data segment is normalized. S22. Baseline Model Training: For the normalized data corresponding to each type of workload, a machine learning algorithm is used to train and generate an initial dynamic efficiency baseline model for the corresponding workload type. The initial dynamic efficiency baseline model includes the power consumption range, the safe range for different temperature types, the execution time range, and the allowable parameter range for different error types corresponding to different workload data. S23. Baseline Model Validation: The initial dynamic efficiency baseline model corresponding to each type of workload is validated by cross-validation method, and the model that meets the preset accuracy requirements under each type of workload is selected as the final dynamic efficiency baseline for the corresponding workload type. S24. Baseline Model Update: Regularly collect new data on the target GPU running for each type of workload, and incrementally update the dynamic efficiency baseline for the corresponding workload type based on the new data to ensure that the dynamic efficiency baseline model for each type of workload can reflect the latest state of the target GPU. S25. Baseline Model Storage: Store the final dynamic efficiency baseline corresponding to each type of workload in the database for subsequent real-time data analysis.

3. The method for detecting and maintaining computer GPU computing power based on data analysis according to claim 1, characterized in that, The specific content of S31, the efficiency degradation analysis, includes: Identify whether the computing power supply efficiency of the target GPU degrades under the same or similar types of workloads; The specific sub-steps include: S311, data alignment processing: aligning the real-time multi-dimensional time-series data of the target GPU collected in S1 with the dynamic efficiency baseline of the same or similar workload type established in S2 on the time axis to ensure that the time dimensions of the two are consistent. S312, Efficiency Deviation Calculation: Calculate the deviation value between real-time multi-dimensional time-series data and dynamic efficiency baseline. The deviation value includes power consumption deviation, deviation for different temperature types, execution time deviation, and deviation for each error type. S313. Degradation Detection: If any deviation value exceeds the preset threshold, it is determined that the computing power supply efficiency of the target GPU has degraded. S314. Degradation Assessment: Based on the magnitude of each deviation value, assess the degree of degradation in the computing power supply efficiency of the target GPU and generate a degradation report; S315. Decline Record Storage: Store decline reports in a database for subsequent failure chain matching analysis.

4. The method for detecting and maintaining computer GPU computing power based on data analysis according to claim 1, characterized in that, The specific content of S32, the fault chain matching analysis, includes: S321. Loading Fault Chain Models: Load predefined fault chain models from the database. The fault chain models describe the chronological order and logical relationship of a series of symptoms caused by different root causes. S322. Real-time data feature extraction: Extract the linkage relationship features between multiple indicators from the real-time multi-dimensional time series data of the target GPU. The linkage relationship features include correlation coefficient and change trend. S323. Matching degree calculation: The extracted linkage relationship features are matched with the symptom features of each fault chain model at each stage of fault development, and the matching degree value between the target GPU and each fault chain model at each stage of fault development is calculated. S324. Root cause location: If the matching degree between the target GPU and a certain fault chain model at a certain fault development stage is greater than a preset threshold, then the fault chain model is determined to be the fault chain of the current target GPU, and the root cause of efficiency degradation or abnormality and the fault development stage of the fault chain in which the target GPU is located are located. S325, Root Cause Record Storage: Store the identified root cause and the specific stage of the fault chain in which the target GPU is located in the database for subsequent health scoring and risk quantification.

5. The method for detecting and maintaining computer GPU computing power based on data analysis according to claim 1, characterized in that, The generation of maintenance decisions in S5 includes the following steps: S51. Resource Status Assessment: Obtain real-time status information of the cluster resources where the target GPU is located, including idle computing resources, network bandwidth, and storage capacity. S52. Service scheduling strategy analysis: Analyze the current service scheduling strategy and determine the priority of service continuity; S53. Maintenance Action Selection: Based on the health score of the target GPU, and in conjunction with the maintenance action and health score mapping table stored in the database, a suitable maintenance action is selected. The maintenance actions include hardware replacement, software update, and parameter adjustment. S54. Execution time window optimization: Calculate the recommended execution time window for maintenance actions based on cluster resource status, business continuity priority, and maintenance actions; S55. Task migration plan generation: To ensure business continuity, a task migration plan is generated, which includes the task migration target node and migration path planning.

6. The method for detecting and maintaining computer GPU computing power based on data analysis according to claim 5, characterized in that, The task migration scheme generation in S5 includes the following steps: S551. Target node selection for migration: Based on the real-time status information of the cluster resources where the target GPU is located, each candidate target node that meets the task migration requirements is selected. The candidate target node must have idle computing resources higher than a preset resource threshold and network bandwidth higher than a preset bandwidth threshold. S552. Migration Path Planning: Based on the network topology and current network traffic distribution of each candidate target node, calculate the optimal migration path from the node where the target GPU is located to each candidate target node. The optimal migration path aims to minimize migration latency. S553. Migration Cost Assessment: For each candidate target node and its corresponding optimal migration path, assess the migration cost, which comprehensively considers migration latency, network resource utilization, and the impact of migration on business continuity. S554. Migration Scheme Determination: Based on the migration cost assessment results, select the candidate target node with the lowest migration cost and its corresponding optimal migration path as the final task migration scheme. S555. Migration record storage: Store the final task migration plan in the database for use in subsequent maintenance decisions.

7. The method for detecting and maintaining computer GPU computing power based on data analysis according to claim 1, characterized in that, The calculation of the comprehensive health score in S4 includes the following steps: S41. Performance degradation rate assessment: Based on the efficiency degradation analysis results, calculate the performance degradation rate of the target GPU. The performance degradation rate is a weighted average of the power consumption deviation, the deviation of different temperature types, the execution time deviation, and the deviation of each error type per unit time. S42. Fault Chain Stage Determination: Based on the root cause of the target GPU and the fault development stage of the fault chain in the fault chain matching analysis results, and combined with the risk weight values ​​of the root cause and fault development stage stored in the database, the risk weight value of the fault development stage of the target GPU is mapped. S43. Business Impact Analysis: Evaluate the potential impact of the target GPU's performance degradation and failure chain stages on business continuity, with the potential impact quantified by the probability and duration of business interruption. S44. Health score calculation: Based on the performance degradation rate, the risk weight value of the fault chain stage, and the business impact analysis results, the comprehensive health score of the target GPU is calculated using a weighted summation method. S45. Score Storage and Update: Store the comprehensive health score in the database and update the score value regularly based on the latest analysis results to reflect the real-time health status of the target GPU.

8. A system for performing the computer GPU computing power detection and maintenance method based on data analysis as described in any one of claims 1-7, characterized in that, include: Data acquisition module: Acquires historical and real-time multi-dimensional time-series data of the target GPU while running workloads; Individualized Dynamic Efficiency Baseline Establishment Module: Based on the collected historical multi-dimensional time-series data, it constructs an individualized dynamic efficiency baseline for the target GPU; Computing power supply efficiency analysis and fault chain reasoning module: compares and analyzes the collected real-time multi-dimensional time series data with the established dynamic efficiency baseline, and performs sub-steps of efficiency degradation analysis and fault chain matching analysis. The comprehensive health score and risk quantification module is used to calculate the comprehensive health score of the target GPU, which includes the comprehensive performance degradation rate, the stage of the failure chain, and the potential impact on business continuity. The maintenance decision generation and triggering module is used to automatically generate and output targeted maintenance decisions based on health scores, combined with cluster resource status and business scheduling strategies.

Images