A graphics processing unit monitoring method and device, electronic equipment and storage medium

Abstract

The disclosure provides a graphics processing unit monitoring method and device, electronic equipment and storage medium, and relates to the technical field of computers. The method comprises the following steps: obtaining a state parameter of a graphics processing unit (GPU) memory; generating fusion feature data of the GPU memory based on the state parameter; the fusion feature data comprises GPU memory usage, memory bandwidth occupancy and memory usage fluctuation coefficient; training a support vector machine (SVM) model based on the fusion feature data to obtain a GPU state prediction model; the GPU state prediction model can predict the GPU memory state; predicting the real-time fusion feature of the GPU memory based on the GPU state prediction model to obtain a GPU state prediction result; the state prediction result comprises the real-time memory state and confidence of the GPU. Thus, the spatial features of memory usage and the time fluctuation correlation can be captured in multiple dimensions, the false positive rate is significantly reduced, and the system stability and performance optimization capability are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a graphics processing unit monitoring method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence, scientific computing and graphics rendering, the application scenarios of graphics processing units (GPUs) are increasingly complex, and the reasonable management and state monitoring of GPU memory resources have become a key link for system stability and performance optimization. The existing GPU monitoring solutions mainly use monitoring tools provided by hardware manufacturers and open source monitoring frameworks, and mainly realize the following functions: (1) real-time collection of basic indicators such as GPU memory usage, temperature, and computing power load; (2) triggering an alarm based on a preset threshold (such as marking as "severe" when the memory usage is greater than or equal to 90%); (3) providing data visualization and historical trend analysis.

[0003] However, the existing GPU monitoring solutions often have the following defects: (1) static threshold defect: fixed threshold cannot adapt to different application scenarios (such as normal fluctuations in memory usage in deep learning training may be higher than in traditional graphics rendering scenarios); (2) single-dimensional evaluation: no evaluation model is constructed for multi-feature fusion, which cannot capture the potential correlation between memory usage and bandwidth (such as high memory usage accompanied by low bandwidth may represent a data read-write bottleneck), resulting in inaccurate monitoring results; (3) passive monitoring mode: only state monitoring is realized, lacking the ability to predict future abnormalities and dynamic strategy optimization mechanisms. SUMMARY

[0004] The present disclosure provides a graphics processing unit monitoring method, device, electronic equipment and storage medium to at least solve the above technical problems in the prior art.

[0005] According to a first aspect of the present disclosure, a graphics processing unit monitoring method is provided, comprising: obtaining state parameters of a graphics processing unit (GPU) memory; the state parameters comprising total memory capacity, memory usage, memory bandwidth and memory bandwidth occupancy of the GPU; based on the state parameters, generating fusion feature data of the GPU memory; the fusion feature data comprising memory usage rate, memory bandwidth occupancy rate and memory usage rate fluctuation coefficient of the GPU; based on the fusion feature data, training a support vector machine (SVM) model to obtain a GPU state prediction model; the GPU state prediction model being capable of predicting the GPU memory state; based on the GPU state prediction model, predicting real-time fusion features of the GPU memory to obtain a state prediction result of the GPU; the state prediction result comprising real-time memory state and confidence of the GPU.

[0006] In an implementation, the obtaining the state parameter of the GPU memory includes at least one of the following: obtaining the state parameter of the GPU memory based on a monitoring interface of the GPU; obtaining the state parameter of the GPU memory based on a baseboard management controller (BMC); obtaining the state parameter of the GPU memory based on a kernel log of the GPU; and obtaining the state parameter of the GPU memory based on a register corresponding to the GPU.

[0007] In an implementation, the generating the fusion feature data of the GPU memory based on the state parameter includes: generating a GPU memory usage rate based on the total GPU memory capacity and the GPU memory usage; generating a GPU memory bandwidth occupancy rate based on the GPU memory bandwidth and the GPU memory bandwidth occupancy; and generating a GPU memory usage rate fluctuation coefficient based on a random coefficient and the GPU memory usage rate, where the random coefficient represents a fluctuation of the GPU memory usage rate over time.

[0008] In an implementation, the generating the fusion feature data of the GPU memory based on the state parameter includes: determining a mean vector and a covariance matrix of the fusion feature data in different GPU memory states based on the state parameter, where the GPU memory state represents a risk level of the GPU memory; and generating the fusion feature data in each GPU memory state through normal distribution simulation based on the mean vector and the covariance matrix.

[0009] In an implementation, the determining the mean vector and the covariance matrix of the fusion feature data in different GPU memory states based on the state parameter includes: determining the fusion feature data in different GPU memory states based on the state parameter; and determining the mean vector and the covariance matrix based on the fusion feature data in different GPU memory states.

[0010] In an implementation, after the obtaining the state parameter of the GPU memory, the method further includes at least one of the following: in response to an simulation environment variable being set to 1, determining that the fusion feature data of the GPU memory needs to be generated based on the state parameter; the simulation environment variable is used to indicate whether the fusion feature data is obtained through simulation; and in response to detecting that there is no GPU, determining that the fusion feature data of the GPU memory needs to be generated based on the state parameter.

[0011] In an implementation, the GPU memory state includes a normal state, a warning state, and a serious state; the mean vector includes a mean of the GPU memory usage rate, a mean of the GPU memory bandwidth occupancy rate, and a mean of the GPU memory usage rate fluctuation coefficient; and the covariance matrix includes a variance of the GPU memory usage rate, a variance of the GPU memory bandwidth occupancy rate, and a variance of the GPU memory usage rate fluctuation coefficient.

[0012] In an implementation, after the fusion feature data of the GPU memory is generated based on the state parameters, the method further includes: performing outlier filtering on the fusion feature data; and converting the filtered fusion feature data into a target format supported by the SVM model, wherein the target format is a two-dimensional array, and a first dimension is a number of the fusion feature data, and a second dimension is a number of features in the fusion feature data.

[0013] In an implementation, the training of the SVM model based on the fusion feature data to obtain the GPU state prediction model includes: labeling the fusion feature data with corresponding GPU memory states; and inputting the labeled fusion feature data into the SVM model to train the SVM model and obtain the GPU state prediction model.

[0014] In an implementation, the prediction of the real-time fusion feature of the GPU memory based on the GPU state prediction model includes: obtaining the real-time fusion feature of the GPU memory; inputting the real-time fusion feature into the GPU state prediction model, wherein the GPU state prediction model outputs prediction probabilities of different GPU memory states; determining a GPU memory state corresponding to a maximum value of the prediction probabilities as the real-time memory state of the GPU, and determining the maximum value of the prediction probabilities as the confidence.

[0015] In an implementation, the method further includes: determining that the real-time memory state is unknown in response to the confidence being less than a confidence threshold.

[0016] According to a second aspect of the present disclosure, a GPU monitoring apparatus is provided, which includes: an obtaining module configured to obtain state parameters of a GPU memory; the state parameters include a total memory capacity, a memory usage, a memory bandwidth, and a memory bandwidth occupancy of the GPU; a generating module configured to generate fusion feature data of the GPU memory based on the state parameters; the fusion feature data includes a memory usage rate, a memory bandwidth occupancy rate, and a memory usage rate fluctuation coefficient of the GPU; a training module configured to train an SVM model based on the fusion feature data to obtain a GPU state prediction model; the GPU state prediction model is capable of predicting a GPU memory state; and a prediction module configured to predict a real-time fusion feature of the GPU memory based on the GPU state prediction model to obtain a state prediction result of the GPU; the state prediction result includes a real-time memory state and a confidence of the GPU.

[0017] In an implementation, the obtaining module is further configured to: obtain the state parameter of the GPU memory based on a monitoring interface of the GPU; obtain the state parameter of the GPU memory based on a baseboard management controller (BMC); obtain the state parameter of the GPU memory based on a kernel log of the GPU; and obtain the state parameter of the GPU memory based on a register corresponding to the GPU.

[0018] In an implementation, the generating module is further configured to: generate the GPU memory usage rate based on the total GPU memory capacity and the GPU memory usage amount; generate the GPU memory bandwidth occupancy rate based on the GPU memory bandwidth and the GPU memory bandwidth occupancy amount; and generate the GPU memory usage rate fluctuation coefficient based on a random coefficient and the GPU memory usage rate, where the random coefficient represents fluctuation of the GPU memory usage rate over time.

[0019] In an implementation, the generating module is further configured to: determine the mean vector and the covariance matrix of the fused feature data in different GPU memory states based on the state parameter, where the GPU memory state represents a risk degree of the GPU memory; and generate the fused feature data in each GPU memory state through normal distribution simulation based on the mean vector and the covariance matrix.

[0020] In an implementation, the generating module is further configured to: determine the fused feature data in different GPU memory states based on the state parameter; and determine the mean vector and the covariance matrix based on the fused feature data in different GPU memory states.

[0021] In an implementation, the GPU monitoring apparatus further includes a determining module configured to: determine that the fused feature data of the GPU memory needs to be generated based on the state parameter in response to an simulation environment variable being set to 1, where the simulation environment variable is used to indicate whether the fused feature data is obtained through simulation; and determine that the fused feature data of the GPU memory needs to be generated based on the state parameter in response to detection of a non-existing GPU.

[0022] In an implementation, the GPU memory state includes a normal state, a warning state, and a serious state; the mean vector includes a mean of the GPU memory usage rate, a mean of the GPU memory bandwidth occupancy rate, and a mean of the GPU memory usage rate fluctuation coefficient; and the covariance matrix includes a variance of the GPU memory usage rate, a variance of the GPU memory bandwidth occupancy rate, and a variance of the GPU memory usage rate fluctuation coefficient.

[0023] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0024] at least one processor; and a memory connected with the at least one processor in communication;

[0025] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the present disclosure.

[0026] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of the present disclosure.

[0027] The GPU state prediction model predicts the real-time fusion features of the GPU memory to obtain a state prediction result including a real-time memory state and a confidence level. Thus, the static threshold defects, single-dimensional evaluation, and limitations of the passive monitoring mode of the existing monitoring scheme are effectively overcome, a dynamic and adaptive intelligent evaluation system is realized, the spatial features and time fluctuation correlation of the memory usage can be captured in multiple dimensions, the false positive rate is significantly reduced, and the system stability and performance optimization capability are improved.

[0028] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0029] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0030] In the drawings, identical or corresponding reference numerals indicate identical or corresponding parts.

[0031] Figure 1 A flowchart of a GPU state prediction method according to an embodiment of the present disclosure is shown Figure 1 ;

[0032] Figure 2 A flowchart of a GPU state prediction method according to an embodiment of the present disclosure is shown Figure 2 ;

[0033] Figure 3 FIG. 1 shows a flow diagram of a method for monitoring a graphics processing unit according to an embodiment of the present disclosure. Figure 3 ;

[0034] Figure 4 FIG. 2 shows a structural diagram of a device for monitoring a graphics processing unit according to an embodiment of the present disclosure.

[0035] Figure 5 FIG. 3 shows a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to make the objectives, features and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0037] Figure 1 FIG. 1 shows a flow diagram of a method for monitoring a graphics processing unit according to an embodiment of the present disclosure. Figure 1 As shown in FIG. 1, a method for monitoring a graphics processing unit (GPU) includes the following steps. Figure 1

[0038] In step S101, a state parameter of GPU memory is acquired.

[0039] In the embodiment, the state parameter can reflect the current running state of the GPU memory. The state parameter can include total GPU memory capacity, GPU memory usage, GPU memory bandwidth and GPU memory bandwidth occupancy. The total GPU memory capacity refers to the maximum available capacity of the GPU memory, reflecting the maximum memory resource that the GPU can support in design. The GPU memory usage refers to the size of the memory currently occupied by a program or a task, which can directly reflect the current usage of the memory. The GPU memory bandwidth refers to the data transfer rate of the GPU memory, which is usually measured by the amount of data transferred per second, reflecting the data transfer capability of the memory. The GPU memory bandwidth occupancy refers to the size of the GPU memory bandwidth currently occupied, which can reflect the current usage of the GPU memory bandwidth.

[0040] In step S102, fusion feature data of the GPU memory is generated based on the state parameter.

[0041] ​In this embodiment, the fusion feature data can better reflect the use state and performance of the GPU display memory. The fusion feature data can include GPU display memory usage, display memory bandwidth occupancy, and display memory usage fluctuation coefficient. The display memory usage can be the ratio of display memory usage to total display memory capacity, which can reflect the occupancy degree of the display memory. The display memory bandwidth occupancy can be the ratio of display memory bandwidth occupancy to display memory bandwidth, which can reflect the occupancy degree of the display memory bandwidth. The display memory usage fluctuation coefficient is an index for measuring the fluctuation of the display memory usage over time, which can reflect the dynamic change of the display memory usage.

[0042] In step S103, the support vector machine (SVM) model is trained based on the fusion feature data to obtain a GPU state prediction model.

[0043] In this embodiment, after the fusion feature data is generated, the SVM model can be trained using the fusion feature data to obtain a GPU state prediction model. SVM is a supervised learning algorithm and is widely used in classification and regression analysis. By inputting the fusion feature data and the corresponding GPU display memory state into the SVM model for training, a GPU state prediction model can be obtained, which can predict the GPU display memory state according to the fusion feature data. The GPU state prediction model can predict the current state of the GPU display memory, such as normal, warning, or severe, according to the input fusion feature data.

[0044] In step S104, the real-time fusion feature of the GPU display memory is predicted based on the GPU state prediction model to obtain a GPU state prediction result.

[0045] In this embodiment, after the GPU state prediction model is trained, the real-time GPU display memory fusion feature data can be predicted using the GPU state prediction model. In actual operation, the system can obtain the state parameters of the GPU display memory in real time and generate corresponding fusion feature data. Then, the real-time fusion feature data is input into the GPU state prediction model, and the GPU state prediction model outputs the real-time display memory state of the GPU and the predicted confidence. The confidence is an index for measuring the certainty of the prediction result of the GPU state prediction model, which is usually a value between 0 and 1. By predicting the real-time display memory state of the GPU and the predicted confidence, the use of the GPU display memory can be understood in time, and the reliability of the prediction result can be evaluated according to the confidence, so that a more intelligent decision can be made.

[0046] In the present disclosure, by constructing a multi-feature fusion support vector machine model, the state of GPU memory can be more accurately predicted, reducing false positives and false negatives. At the same time, by introducing a confidence evaluation mechanism, the reliability of the prediction results can be more intuitively understood, thereby improving the operation and maintenance efficiency and system stability. In addition, it can also adapt to different GPU hardware environments and application scenarios, with high universality and flexibility.

[0047] In another embodiment, the step S101 of "obtaining the state parameters of the GPU memory" includes at least one of the following:

[0048] Based on the monitoring interface of the GPU, the state parameters of the GPU memory are obtained.

[0049] In this embodiment, the monitoring interface is an interface provided by the GPU hardware or its driver, which is used to obtain various running state information of the GPU in real time. The monitoring interface usually exists in the form of an application programming interface (API), which allows software programs to access the hardware state parameters of the GPU. For example, NVIDIA's GPU provides a monitoring interface such as NVIDIA Management Library (NVML), through which the memory usage, temperature, power consumption, and other information of the GPU can be obtained. If a system uses NVIDIA's GPU and installs the NVML library, the state parameters of the GPU memory, such as memory usage and memory bandwidth occupancy, can be obtained by calling the API functions provided by NVML.

[0050] Based on the baseboard management controller (BMC), the state parameters of the GPU memory are obtained.

[0051] In this embodiment, the baseboard management controller (BMC) is a management chip integrated in the server hardware, which is used to monitor and manage the hardware state of the server. BMC can collect state information of various hardware components including GPU. For example, in some servers in data centers, BMC can monitor the memory temperature and memory usage of GPU. Through BMC, the state parameters of GPU memory can be obtained even when the operating system is not started, which is very useful for hardware fault diagnosis and system maintenance.

[0052] Based on the kernel log of the GPU, the state parameters of the GPU memory are obtained.

[0053] In this embodiment, the kernel logs of the GPU refer to the log information related to the GPU recorded by the operating system kernel. These kernel logs usually contain various events and state information during the operation of the GPU. For example, when the GPU memory has an error or reaches a certain threshold, the kernel may record relevant log information. By analyzing these kernel logs, the state parameters of the GPU memory can be extracted. For example, in a Linux system, by viewing the kernel logs output by the dmesg command, the log entries related to the GPU memory can be found, and parameters such as memory usage can be obtained.

[0054] Based on the registers corresponding to the GPU, the state parameters of the GPU memory are obtained.

[0055] In this embodiment, the registers of the GPU are storage units inside the GPU hardware for storing control and state information. By directly reading the values of the GPU registers, detailed state parameters of the GPU memory can be obtained. For example, some GPU registers may store information such as the current usage of the memory, the occupancy of the memory bandwidth, etc. By directly accessing these registers through a specific driver or tool, the state parameters of the GPU memory can be obtained.

[0056] In this disclosure, by providing multiple ways to obtain the state parameters of the GPU memory, the flexibility and adaptability of the system are enhanced. This means that no matter on different hardware platforms or in different system running states, the required state parameters of the GPU memory can be effectively obtained. This diversified acquisition method ensures that even in the case of some interfaces being unavailable or some hardware failures, the system can still obtain the state parameters of the GPU memory through other ways, thereby improving the reliability and stability of the entire monitoring system.

[0057] Figure 2 A flowchart of a graphics processing unit monitoring method according to an embodiment of the disclosure is shown Figure 2 As shown in Figure 2 A graphics processing unit monitoring method includes:

[0058] Step S201, obtaining the state parameters of the GPU memory.

[0059] The specific implementation details of step S201 are similar to those of step S101, which will not be repeated here.

[0060] If it is determined that the fusion feature data does not need to be simulated, steps S202-S204 are performed:

[0061] Step S202, based on the total memory capacity and the memory usage, generating the memory usage rate.

[0062] In this embodiment, the GPU memory usage rate is obtained by calculating the ratio of the GPU memory usage to the total GPU memory capacity. The GPU memory usage rate reflects the degree of occupation of the current GPU memory. For example, if the total GPU memory capacity of a GPU is 8 GB and the current GPU memory usage is 4 GB, then the GPU memory usage rate is 4 GB divided by 8 GB, which is 50%.

[0063] Step S203, based on the GPU memory bandwidth and the GPU memory bandwidth occupancy, generating a GPU memory bandwidth occupancy rate.

[0064] In this embodiment, the GPU memory bandwidth occupancy rate is obtained by calculating the ratio of the GPU memory bandwidth occupancy to the GPU memory bandwidth. The GPU memory bandwidth occupancy rate reflects the usage of the current GPU memory bandwidth. For example, if the GPU memory bandwidth of a GPU is 320 GB / s and the current GPU memory bandwidth occupancy is 160 GB / s, then the GPU memory bandwidth occupancy rate is 160 GB / s divided by 320 GB / s, which is 50%.

[0065] Step S204, based on a random coefficient and the GPU memory usage rate, generating a GPU memory usage rate fluctuation coefficient.

[0066] In this embodiment, the GPU memory usage rate fluctuation coefficient is obtained by multiplying the GPU memory usage rate by a random coefficient. The random coefficient is used to simulate the fluctuation of the GPU memory usage rate over time. For example, assuming that the GPU memory usage rate is 50% and the random coefficient randomly varies between 0.9 and 1.1, then the GPU memory usage rate fluctuation coefficient may vary between 45% and 55%. The GPU memory usage rate fluctuation coefficient can help the model better capture the dynamic changes of the GPU memory usage rate, thereby improving the accuracy of the prediction. By introducing the random coefficient, the fluctuation of the GPU memory usage rate at different time points can be simulated, so that the model can better adapt to the dynamic changes in actual operation. In an example, the fusion feature data can be a three-dimensional feature vector [u, b, u·δ], where u is the GPU memory usage rate, b is the GPU memory bandwidth occupancy rate, δ is the random coefficient, and u·δ is the GPU memory usage rate fluctuation coefficient.

[0067] If it is determined that the fusion feature data needs to be simulated and generated, steps S205-S206 are performed:

[0068] Step S205, based on the state parameters, determining the mean vector and the covariance matrix of the fusion feature data under different GPU memory states.

[0069] In this embodiment, in the process of generating fusion feature data, first, the existing state parameters are needed to determine the existing GPU memory usage, GPU memory bandwidth occupancy, and GPU memory usage fluctuation coefficient, and then the statistical characteristics of these feature data under different GPU memory states are determined, specifically including the mean vector and the covariance matrix. These statistical characteristics can reflect the distribution law of the GPU memory features under different states. The GPU memory state represents the risk level of the GPU memory.

[0070] In an example, the mean vector is a multi-dimensional vector, where each dimension represents the average value of a feature. For example, suppose we have three features: GPU memory usage, GPU memory bandwidth occupancy, and GPU memory usage fluctuation coefficient. In the normal state, the mean values of these features may be [0.4, 0.3, 0.45] (i.e., GPU memory usage 40%, GPU memory bandwidth occupancy 30%, and GPU memory usage fluctuation coefficient 45%). In the warning state, the mean values may be [0.7, 0.6, 0.75], and in the severe state, the mean values may be [0.9, 0.8, 0.95]. These mean vectors can represent the average level of features under different GPU memory states.

[0071] In an example, the covariance matrix is a square matrix that describes the correlation between features. It reflects the joint distribution of features. For example, suppose that in the normal state, the covariance between GPU memory usage and GPU memory bandwidth occupancy is 0.01, indicating that there is a certain positive correlation between the two features. The covariance matrix can help us understand the relationship between features, so as to better simulate and predict the GPU memory state.

[0072] Step S206, based on the mean vector and the covariance matrix, generating fusion feature data under each GPU memory state through normal distribution simulation.

[0073] In this embodiment, after determining the mean vector and the covariance matrix under different GPU memory states, these statistical characteristics can be used to generate fusion feature data under each state through normal distribution simulation. Specifically, normal distribution is a common probability distribution, whose shape is determined by the mean and the covariance matrix. By setting different mean vectors and covariance matrices, feature data that conforms to the distribution of a specific state can be generated.

[0074] For example, suppose we have determined the mean vector and the covariance matrix under the normal, warning, and severe states, where:

[0075] Normal state: mean vector [40, 30, 40×1.0], covariance matrix [[5², 0, 0], [0, 10², 0], [0, 0, (40×0.1)²]];

[0076] Warning state: mean vector [75, 60, 75x1.0], covariance matrix [[5², 0, 0], [0, 10²,0], [0, 0, (75x0.1)²]].

[0077] Critical state: mean vector [90, 80, 90x1.0], covariance matrix [[5², 0, 0], [0, 10²,0], [0, 0, (90x0.1)²]].

[0078] Then generate synthetic data, that is, generate synthetic data in each state by normal distribution using the above mean vector and covariance matrix.

[0079] In an example, the fusion feature data in each GPU memory state can be simulated based on the following code:

[0080] def generate_synthetic_data(self, n_samples=100, states=3):

[0081] np.random.seed(42)

[0082] means = {

[0083] 0: [40, 30, 40 1.0],# normal

[0084] 1: [75, 60, 75 1.0],# warning

[0085] 2: [90, 80, 90 1.0]# critical

[0086] }

[0087] cov = np.diag([5 2, 10 2, (0.1 means[0][0]) 2])# share covariance structure

[0088] X, y = [], []

[0089] for state in range(states):

[0090] X_state = np.random.multivariate_normal(means[state], cov, n_samples)

[0091] X.append(X_state)

[0092] y.append(np.full(n_samples, state, dtype=int))

[0093] X = np.vstack(X)

[0094] y = np.hstack(y)

[0095] idx = np.random.permutation(len(X))

[0096] return X[idx], y[idx]。

[0097] Step S207, based on the fusion feature data, the support vector machine SVM model is trained to obtain a GPU state prediction model.

[0098] Step S208, based on the GPU state prediction model, the real-time fusion features of the GPU memory are predicted to obtain a GPU state prediction result.

[0099] The specific implementation details of steps S207-S208 are similar to steps S103-S104, and will not be repeated here.

[0100] In the present disclosure, actual fusion feature data can be generated based on actual state parameters, or fusion feature data can be simulated based on actual state parameters, which can effectively solve the problem of insufficient data, provide more samples for model training, and thus improve the robustness and accuracy of the model.

[0101] In another embodiment, step S205 "determining the mean vector and covariance matrix of the fusion feature data under different GPU memory states based on the state parameters" comprises:

[0102] Based on the state parameters, the fusion feature data under different GPU memory states is determined;

[0103] Based on the fusion feature data under different GPU memory states, the mean vector and covariance matrix are determined.

[0104] In this embodiment, in determining the fusion feature data under different GPU memory states, first, different memory states need to be identified and classified according to the existing state parameters. These memory states usually include normal state, warning state and serious state. For example, based on these state parameters, fusion feature data can be generated, and then based on historical data or experimental data, the GPU memory state corresponding to each fusion feature data is determined, so as to determine the fusion feature data under different GPU memory states.

[0105] In this embodiment, after the fusion feature data under different memory states is determined, the next step is to calculate the distribution characteristics of the fusion feature data under each memory state, that is, the mean vector and covariance matrix of the fusion feature data under each memory state.

[0106] In an example, for the fusion feature data under each state, the mean of each feature is calculated respectively. For example, the fusion feature data under the normal state is {( , , ),( , , ),…}, and the mean vector is , where N is the total number of fusion feature data under the normal state. The calculation method of the mean vector under the warning state and the serious state is similar to that under the normal state.

[0107] In an example, for the fusion feature data under each state, the covariance between each feature is calculated. The covariance matrix is a symmetric matrix, and the diagonal elements represent the variance of each feature, and the non-diagonal elements represent the covariance between features. For example, the covariance matrix under the normal state is , and the element represents the covariance between the i-th feature and the j-th feature. The calculation method of the covariance matrix under the warning state and the serious state is similar to that under the normal state.

[0108] In this disclosure, by determining the mean vector and covariance matrix based on the fusion feature data under different GPU memory states, the rationality of the fusion feature data generation and the accuracy of the model training are significantly improved. The simulation data generated by this method can accurately reflect the feature distribution law under different memory states, making the model training more close to the actual situation.

[0109] In another embodiment, after the step S201 of "obtaining the state parameters of the GPU memory", the GPU monitoring method further includes at least one of:

[0110] In response to the simulation environment variable being set to 1, it is determined that fusion feature data of GPU memory needs to be simulated and generated based on state parameters; the simulation environment variable is used to indicate whether fusion feature data is obtained through simulation generation.

[0111] In response to the detection that the GPU is not present, it is determined that fused feature data of the GPU memory needs to be generated based on state parameters.

[0112] In this embodiment, it is also necessary to determine whether it is necessary to simulate the generation of fused feature data, that is... σ The judgment process following step S201. There are two ways to determine whether to simulate the generation of fusion feature data. One method is to set a simulation environment variable (e.g., GPU_SIMULATION). When this variable is set to 1, the system automatically switches to simulation mode and generates simulated GPU memory fusion feature data. This simulated fusion feature data is generated based on existing state parameters and can simulate the memory state in a real environment. For example, if the user believes that the amount or quality of actual fusion feature data generated using existing state parameters is insufficient, the simulation environment variable can be set to 1 to trigger simulated data generation.

[0113] In this embodiment, in addition to triggering simulated data generation through simulated environment variables, the system can also automatically detect the presence of GPU hardware. If no GPU is detected in the system (e.g., by calling a GPU detection tool or API and finding that the returned GPU list is empty), the system will automatically switch to simulation mode and generate simulated GPU memory fusion feature data. For example, suppose the system calls GPUtil.getGPUs() during initialization, but the returned result is empty, indicating that there is no available GPU in the current environment. In this case, the system will automatically enable the simulated data generation mechanism to generate simulated fusion feature data according to preset state parameters to ensure the normal operation of the system and the smooth progress of development and testing.

[0114] This disclosure significantly enhances the system's flexibility and adaptability by introducing simulated environment variables and an automatic detection mechanism. It allows the generation of simulated data even without real GPU hardware, thereby reducing hardware dependence and lowering development and testing costs. Simultaneously, the system can automatically detect the hardware environment and switch to simulation mode, reducing manual intervention and improving system robustness and user experience. This mechanism not only accelerates the development cycle but also ensures smooth development and testing even in resource-constrained environments, enhancing the overall performance and reliability of the system.

[0115] In another embodiment, the GPU memory state includes a normal state, a warning state and a serious state; the mean vector includes a mean of memory usage, a mean of memory bandwidth occupancy and a mean of memory usage fluctuation coefficient; the covariance matrix includes a variance of memory usage, a variance of memory bandwidth occupancy and a variance of memory usage fluctuation coefficient.

[0116] In another embodiment, after the step S102 of "generating the fusion feature data of the GPU memory based on the state parameters", it further includes:

[0117] performing outlier filtering on the fusion feature data;

[0118] converting the filtered fusion feature data into a target format supported by the SVM model; the target format is a two-dimensional array, the first dimension is the number of fusion feature data, and the second dimension is the number of features in the fusion feature data.

[0119] In this embodiment, after generating the fusion feature data, in order to ensure the quality and reliability of the data, it is necessary to perform outlier filtering on these data. Outliers refer to those data points that deviate significantly from the normal range, which may be due to measurement errors, data collection errors or sensor failures, etc. If these outliers are directly used for model training, it may cause the model to learn the wrong pattern, thereby affecting the accuracy of prediction. Outlier filtering can usually be achieved by statistical methods. For example, the Z-score method can be used to identify and filter outliers. Z-score is the number of standard deviations of a data point from the mean. If the Z-score of a data point exceeds a certain threshold, it can be considered as an outlier and filtered out. For example, assuming we have a set of fusion feature data, we calculate the mean and standard deviation of each feature σ , we can calculate the Z-score of each data point, and if the Z-score of a certain data point is greater than 3 σ or less than -3 Figure 3 , it is marked as an outlier and filtered out.

[0120] In this embodiment, after completing the outlier filtering, the next step is to convert the filtered fusion feature data into a format that can be accepted by the support vector machine (SVM) model. The SVM model usually requires the input data to be in the form of a two-dimensional array, where the first dimension represents the number of data samples, and the second dimension represents the number of features in each sample. For example, assuming we have 100 samples, each containing 3 features (such as memory usage, memory bandwidth occupancy and memory usage fluctuation coefficient), the converted data format should be a 100x3 two-dimensional array.

[0121] In an example, the fusion feature data can be filtered for outliers and converted to the format based on the following code:

[0122] def preprocess_features(self, gpu_data):

[0123] features = []

[0124] for data in gpu_data:

[0125] feat = [

[0126] data['memory_percent'],

[0127] data['memory_bandwidth'],

[0128] data['memory_percent'] np.random.uniform(0.9, 1.1) ]

[0130] features.append(feat)

[0131] features = np.array(features)

[0132] # Outlier filtering (Z-score 3σ)

[0133] if len(features)>0:

[0134] mean = np.mean(features, axis=0)

[0135] std = np.std(features, axis=0)

[0136] # Calculate the Z-score and remove samples with an absolute value > 3.

[0137] z_scores = np.abs((features - mean) / std)

[0138] valid_mask = np.all(z_scores<3, axis=1)

[0139] features = features[valid_mask]

[0140] # Dimension Validation

[0141] return features.reshape(-1, 3).

[0142] In the present disclosure, by introducing the steps of outlier filtering and data format conversion, the data quality and input compatibility of the model are significantly improved. Outlier filtering can effectively remove erroneous or outlier data points, ensuring the reliability and accuracy of the input data. Data format conversion ensures that the filtered data can be correctly processed by the SVM model, improving the training effect and prediction performance of the model.

[0143] In another embodiment, step S103 "training a support vector machine (SVM) model based on the fused feature data to obtain a GPU state prediction model" includes:

[0144] Labeling the corresponding GPU memory state for the fused feature data;

[0145] Inputting the labeled fused feature data into the SVM model for training to obtain the GPU state prediction model.

[0146] In this embodiment, before training the SVM model, the fused feature data needs to be labeled, that is, each data sample is assigned a corresponding GPU memory state label. These labels usually include "normal", "warning" and "serious" categories, indicating the memory state represented by each data sample. For example, suppose we have a set of fused feature data, each data sample contains memory usage rate, memory bandwidth occupancy rate and memory usage rate fluctuation coefficient. In the labeling process, we will judge which memory state each sample belongs to according to the actual values of these features and assign the corresponding label.

[0147] For example: if a data sample has a memory usage rate of 40%, a memory bandwidth occupancy rate of 30%, and a memory usage rate fluctuation coefficient of 40%, it can be labeled as "normal" state. If a data sample has a memory usage rate of 75%, a memory bandwidth occupancy rate of 60%, and a memory usage rate fluctuation coefficient of 75%, it can be labeled as "warning" state. If a data sample has a memory usage rate of 90%, a memory bandwidth occupancy rate of 80%, and a memory usage rate fluctuation coefficient of 90%, it can be labeled as "serious" state.

[0148] After completing the data labeling, the next step is to input the labeled fused feature data into the SVM model for training, thereby obtaining the GPU state prediction model. Thus, by labeling the corresponding GPU memory state for the fused feature data and inputting the labeled data into the SVM model for training, the model can learn the feature differences between different memory states, thereby improving the classification accuracy and prediction ability of the model.

[0149] Figure 3 A flowchart of a graphics processing unit monitoring method is shown Figure 3 As shown in Figure 4 A graphics processing unit monitoring method includes:

[0150] Step S301, obtaining a state parameter of a graphics processing unit (GPU) memory.

[0151] Step S302, generating fusion feature data of the GPU memory based on the state parameter.

[0152] Step S303, training a support vector machine (SVM) model based on the fusion feature data to obtain a GPU state prediction model.

[0153] The specific implementation details of steps S301-S303 are similar to steps S101-S103, which will not be repeated here.

[0154] Step S304, obtaining real-time fusion features of the GPU memory.

[0155] In this embodiment, before real-time prediction, the real-time fusion features of the GPU memory need to be obtained first. The real-time fusion features reflect the current usage of the GPU memory, including memory usage, memory bandwidth occupancy, and memory usage fluctuation coefficient, etc.

[0156] Step S305, inputting the real-time fusion features into the GPU state prediction model, and the GPU state prediction model outputs prediction probabilities of different GPU memory states.

[0157] In this embodiment, after obtaining the real-time fusion features, the next step is to input the real-time fusion features into the trained GPU state prediction model. The model will calculate the prediction probabilities of different memory states according to the input real-time fusion features. For example, assuming we use an SVM-based classification model, the model will output the prediction probability of each state (such as normal, warning, and severe). For example, assuming the input real-time fusion feature data is [75, 50, 75], the model output prediction probability may be: normal state: 10%; warning state: 80%; severe state: 10%.

[0158] Step S306, determining the GPU memory state corresponding to the maximum value of the prediction probability as the real-time memory state of the GPU, and determining the maximum value of the prediction probability as the confidence.

[0159] In this embodiment, the real-time memory state of the GPU also needs to be determined according to the maximum value of the prediction probability, and the maximum value is taken as the confidence. In the above example, the maximum value of the prediction probability is 80%, and the corresponding real-time memory state of the GPU is "warning", and the confidence is 80%.

[0160] In this disclosure, real-time prediction of GPU memory status is achieved by inputting real-time fused feature data into a trained GPU state prediction model, and the confidence level of the prediction is provided. This method can not only quickly and accurately identify the current memory status, but also evaluate the reliability of the prediction results through the confidence level. This enables the system to promptly detect potential problems and take corresponding measures, improving the system's stability and operational efficiency.

[0161] In another embodiment, after step S306 "determining the GPU memory state corresponding to the maximum predicted probability as the real-time memory state of the GPU, and determining the maximum predicted probability as the confidence level", a graphics processing unit monitoring method further includes:

[0162] If the confidence level is less than the confidence threshold, the real-time memory status is determined to be unknown.

[0163] In this embodiment, the GPU state prediction model may sometimes be uncertain about the classification of certain real-time fused features. This uncertainty can be measured by confidence level. If the confidence level is lower than a preset threshold (e.g., 70%), it indicates that the GPU state prediction model's prediction of the current memory state is not reliable enough. In this case, to ensure the reliability of the prediction results, the system will mark the real-time memory state as "unknown," thereby avoiding erroneous decisions caused by inaccurate model predictions.

[0164] Figure 4 A schematic diagram of the structure of a graphics processing unit monitoring device according to an embodiment of the present disclosure is shown, such as... Figure 5 As shown, a graphics processing unit monitoring device includes:

[0165] The acquisition module 10 is used to acquire the status parameters of the GPU memory; the status parameters include the total memory capacity, memory usage, memory bandwidth, and memory bandwidth utilization of the GPU; the generation module 11 is used to generate fused feature data of the GPU memory based on the status parameters; the fused feature data includes the GPU memory utilization rate, memory bandwidth utilization rate, and memory utilization rate fluctuation coefficient; the training module 12 is used to train the support vector machine (SVM) model based on the fused feature data to obtain a GPU state prediction model; the GPU state prediction model can predict the GPU memory state; the prediction module 13 is used to predict the real-time fused features of the GPU memory based on the GPU state prediction model to obtain the GPU state prediction result; the state prediction result includes the real-time memory state of the GPU and the confidence level.

[0166] In another embodiment, the obtaining module 10 is further configured to: obtain the state parameter of the GPU memory based on a monitoring interface of the GPU; obtain the state parameter of the GPU memory based on a baseboard management controller (BMC); obtain the state parameter of the GPU memory based on a kernel log of the GPU; and obtain the state parameter of the GPU memory based on a register corresponding to the GPU.

[0167] In another embodiment, the generating module 11 is further configured to: generate a memory usage rate based on the total memory capacity and the memory usage amount; generate a memory bandwidth occupancy rate based on the memory bandwidth and the memory bandwidth occupancy amount; and generate a memory usage rate fluctuation coefficient based on a random coefficient and the memory usage rate, wherein the random coefficient represents a fluctuation of the memory usage rate over time.

[0168] In another embodiment, the generating module 11 is further configured to: determine a mean vector and a covariance matrix of the fusion feature data in different GPU memory states based on the state parameter, wherein the GPU memory state represents a risk level of the GPU memory; and generate the fusion feature data in each GPU memory state through normal distribution simulation based on the mean vector and the covariance matrix.

[0169] In another embodiment, the generating module 11 is further configured to: determine the fusion feature data in different GPU memory states based on the state parameter; and determine the mean vector and the covariance matrix based on the fusion feature data in different GPU memory states.

[0170] In another embodiment, the GPU monitoring device further includes a determining module configured to: determine that the fusion feature data of the GPU memory needs to be generated based on the state parameter in response to an simulation environment variable being set to 1, wherein the simulation environment variable is used to indicate whether the fusion feature data is obtained through simulation; and determine that the fusion feature data of the GPU memory needs to be generated based on the state parameter in response to detecting that there is no GPU.

[0171] In another embodiment, the GPU memory state includes a normal state, a warning state, and a serious state; the mean vector includes a mean of the memory usage rate, a mean of the memory bandwidth occupancy rate, and a mean of the memory usage rate fluctuation coefficient; and the covariance matrix includes a variance of the memory usage rate, a variance of the memory bandwidth occupancy rate, and a variance of the memory usage rate fluctuation coefficient.

[0172] In another embodiment, the GPU monitoring device further includes a data processing module configured to: perform outlier filtering on the fusion feature data; and convert the filtered fusion feature data into a target format supported by an SVM model, wherein the target format is a two-dimensional array, a first dimension of the two-dimensional array is a number of the fusion feature data, and a second dimension of the two-dimensional array is a number of features in the fusion feature data.

[0173] In another embodiment, the training module 12 is further configured to: label the corresponding GPU memory state for the fusion feature data; input the labeled fusion feature data into the SVM model for training, and obtain a GPU state prediction model.

[0174] In another embodiment, the prediction module 13 is further configured to: obtain real-time fusion features of the GPU memory; input the real-time fusion features into the GPU state prediction model, and the GPU state prediction model outputs prediction probabilities of different GPU memory states; determine the GPU memory state corresponding to the maximum value of the prediction probabilities as the real-time memory state of the GPU, and determine the maximum value of the prediction probabilities as the confidence.

[0175] In another embodiment, the prediction module 13 is further configured to: in response to the confidence being less than the confidence threshold, determine that the real-time memory state is unknown.

[0176] According to embodiments of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0177] Figure 5 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0178] As shown in ​ The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0179] A plurality of components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0180] The computing unit 801 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as a graphics processing unit monitoring method. For example, in some embodiments, a graphics processing unit monitoring method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of a graphics processing unit monitoring method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform a graphics processing unit monitoring method by other any appropriate means, such as by means of firmware.

[0181] The various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0182] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0184] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0185] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] The computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0187] It should be understood that the steps shown in the above-mentioned various forms of flowcharts can be reordered, added, or deleted. For example, each step described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0188] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0189] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method of graphics processing unit monitoring, the method comprising: The method comprises: obtaining state parameters of GPU memory; the state parameters comprise total GPU memory capacity, GPU memory usage, GPU memory bandwidth and GPU memory bandwidth occupancy; in response to an analog environment variable being set to 1 or in response to detecting that there is no GPU, determining mean vectors and covariance matrices of fusion feature data in different GPU memory states based on the state parameters; the GPU memory state represents the risk degree of GPU memory; the analog environment variable is used to indicate whether the fusion feature data is obtained by analog generation; based on the mean vectors and covariance matrices, generating fusion feature data in each GPU memory state by normal distribution analog generation; the fusion feature data comprises GPU memory usage rate, GPU memory bandwidth occupancy rate and GPU memory usage rate fluctuation coefficient; the GPU memory usage rate fluctuation coefficient is used to measure the fluctuation of GPU memory usage rate over time; based on the fusion feature data, training a support vector machine (SVM) model to obtain a GPU state prediction model; the GPU state prediction model can predict the GPU memory state; based on the GPU state prediction model, predicting real-time fusion features of the GPU memory to obtain a GPU state prediction result; the state prediction result comprises real-time GPU memory state and confidence level of the GPU; wherein, based on the GPU state prediction model, the real-time fusion features of the GPU memory are predicted, comprising: obtaining real-time fusion features of the GPU memory; inputting the real-time fusion features into the GPU state prediction model, and the GPU state prediction model outputs prediction probabilities of different GPU memory states; determining the GPU memory state corresponding to the maximum value of the prediction probability as the real-time GPU memory state, and determining the maximum value of the prediction probability as the confidence level; wherein, the method further comprises: in response to the confidence level being less than a confidence threshold, determining that the real-time GPU memory state is unknown.

2. The method of claim 1, wherein, The method further comprises: obtaining state parameters of GPU memory based on a monitoring interface of the GPU; obtaining state parameters of GPU memory based on a baseboard management controller (BMC); obtaining state parameters of GPU memory based on kernel logs of the GPU; obtaining state parameters of GPU memory based on registers corresponding to the GPU.

3. The method of claim 1, wherein, The method further comprises: based on the total GPU memory capacity and GPU memory usage, generating the GPU memory usage rate; based on the GPU memory bandwidth and GPU memory bandwidth occupancy, generating the GPU memory bandwidth occupancy rate; based on a random coefficient and the GPU memory usage rate, generating the GPU memory usage rate fluctuation coefficient; the random coefficient represents the fluctuation of the GPU memory usage rate over time.

4. The method of claim 1, wherein, The method further comprises: based on the state parameters, determining fusion feature data in different GPU memory states; Determine the mean vector and the covariance matrix based on the fusion feature data in different GPU memory states.

5. The method of claim 1, wherein, The GPU memory states include a normal state, a warning state, and a serious state; the mean vector includes a mean of memory usage, a mean of memory bandwidth occupancy, and a mean of memory usage fluctuation coefficient; and the covariance matrix includes a variance of memory usage, a variance of memory bandwidth occupancy, and a variance of memory usage fluctuation coefficient.

6. The method of claim 1, wherein, After the fusion feature data of the GPU memory is generated based on the state parameters, the method further includes: performing outlier filtering on the fusion feature data; converting the filtered fusion feature data into a target format supported by the SVM model; the target format is a two-dimensional array, the first dimension is the number of fusion feature data, and the second dimension is the number of features in the fusion feature data.

7. The method of claim 1, wherein, Training the support vector machine (SVM) model based on the fusion feature data to obtain a GPU state prediction model includes: labeling the fusion feature data with corresponding GPU memory states; inputting the labeled fusion feature data into the SVM model for training to obtain the GPU state prediction model.

8. A graphics processing unit monitoring device, comprising: The apparatus includes: an acquisition module configured to acquire state parameters of GPU memory; the state parameters include total GPU memory capacity, memory usage, memory bandwidth, and memory bandwidth occupancy; a generation module configured to, in response to a simulation environment variable being set to 1 or in response to detecting that no GPU exists, determine a mean vector and a covariance matrix of fusion feature data in different GPU memory states based on the state parameters; the GPU memory states represent the risk level of GPU memory; the simulation environment variable is used to indicate whether the fusion feature data is acquired by simulation generation; generate fusion feature data in each GPU memory state by normal distribution simulation based on the mean vector and the covariance matrix; the fusion feature data includes GPU memory usage, memory bandwidth occupancy, and memory usage fluctuation coefficient; the memory usage fluctuation coefficient is used to measure the fluctuation of memory usage over time; a training module configured to train a support vector machine (SVM) model based on the fusion feature data to obtain a GPU state prediction model; the GPU state prediction model can predict the state of GPU memory; a prediction module configured to predict real-time fusion features of the GPU memory based on the GPU state prediction model to obtain a state prediction result of the GPU; the state prediction result includes the real-time memory state of the GPU and a confidence level; The prediction module is further configured to: acquire real-time fusion features of the GPU memory; input the real-time fusion features into the GPU state prediction model, and the GPU state prediction model outputs prediction probabilities of different GPU memory states; determine the GPU memory state corresponding to the maximum prediction probability as the real-time memory state of the GPU, and determine the maximum prediction probability as the confidence level. In response to the confidence being less than a confidence threshold, determining that the real-time GPU memory status is unknown.

9. The apparatus of claim 8, wherein, The obtaining module is further configured to: obtain the state parameter of the GPU memory based on a monitoring interface of the GPU; obtain the state parameter of the GPU memory based on a baseboard management controller (BMC); obtain the state parameter of the GPU memory based on a kernel log of the GPU; obtain the state parameter of the GPU memory based on a register corresponding to the GPU.

10. The apparatus of claim 8, wherein, The generating module is further configured to: generate the GPU memory usage rate based on the total GPU memory capacity and the GPU memory usage amount; generate the GPU memory bandwidth occupancy rate based on the GPU memory bandwidth and the GPU memory bandwidth occupancy amount; generate the GPU memory usage rate fluctuation coefficient based on a random coefficient and the GPU memory usage rate; the random coefficient represents fluctuation of the GPU memory usage rate over time.

11. The apparatus of claim 8, wherein, The generating module is further configured to: determine fusion feature data in different GPU memory states based on the state parameter; determine the mean vector and the covariance matrix based on the fusion feature data in different GPU memory states.

12. The apparatus of claim 8, wherein, The GPU memory state includes a normal state, a warning state, and a serious state; the mean vector includes a mean of the GPU memory usage rate, a mean of the GPU memory bandwidth occupancy rate, and a mean of the GPU memory usage rate fluctuation coefficient; and the covariance matrix includes a variance of the GPU memory usage rate, a variance of the GPU memory bandwidth occupancy rate, and a variance of the GPU memory usage rate fluctuation coefficient.

13. An electronic device, comprising: comprise: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

14. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-7.

Images