A GPU fault alarm method, device, equipment and medium

Abstract

The application provides a GPU fault alarm method, device, equipment and medium, the method comprises: obtaining the historical data of multiple groups of GPU monitoring items, and the health alarm state data corresponding to the historical data of each group of GPU monitoring items; analyzing the historical data of the multiple groups of the GPU monitoring items and the corresponding health alarm state data, determining the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU; generating a fault alarm message of the GPU according to the influence proportion of each of the multiple GPU monitoring items and real-time data. In the application, the fault alarm is based on the influence proportion of each GPU monitoring item on the health alarm state, and then the overall alarm of the GPU is realized based on each monitoring item data, so that the method can truly reflect the fault condition of the GPU.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fault prediction, and particularly relates to a GPU fault alarm method and device, equipment and a medium. BACKGROUND

[0002] In recent years, the significant improvement of server computing power has broken the barriers to the development of deep learning. People have widely applied deep learning to various fields of production and life. As an important part of the server, a graphics processing unit (GPU) plays an important role in deep learning. Therefore, it is crucial to monitor the health status of the GPU.

[0003] In the prior art, the health alarm state of the GPU is usually an independent alarm, but each alarm is independent and cannot truly reflect the fault condition of the GPU. Therefore, how to truly and accurately alarm the fault of the GPU is a problem to be solved. SUMMARY

[0004] In view of the above problems, the embodiments of the present application provide a GPU fault alarm method, device, equipment and medium in order to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of the embodiments of the present application discloses a GPU fault alarm method, which comprises:

[0006] obtaining historical data of multiple groups of GPU monitoring items and health alarm state data corresponding to the historical data of each group of GPU monitoring items, wherein one group of GPU monitoring items comprises multiple GPU monitoring items;

[0007] analyzing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to determine the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU;

[0008] generating a fault alarm message of the GPU according to the influence proportion of each of the multiple GPU monitoring items and real-time data.

[0009] Optionally, the analyzing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to determine the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU comprises:

[0010] processing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain a feature matrix;

[0011] Deep learning is performed based on the plurality of feature matrices, to obtain an influence proportion of each of the plurality of GPU monitoring items on the health alarm state of the GPU.

[0012] Optionally, the historical data of the plurality of groups of GPU monitoring items and the corresponding health alarm state data are processed to obtain a feature matrix, including:

[0013] The historical data of the plurality of groups of GPU monitoring items and the corresponding health alarm state data are linearly regressed to obtain a plurality of groups of linear regression results;

[0014] According to the plurality of groups of linear regression results, a feature matrix is created, and elements in the feature matrix include health alarm state data and linear regression coefficients of GPU monitoring items.

[0015] Optionally, the method further includes:

[0016] According to the influence proportion from large to small, the first N GPU monitoring items are determined from the plurality of GPU monitoring items;

[0017] According to the influence proportion of each of the first N GPU monitoring items and real-time data, a failure alarm message of the GPU is generated.

[0018] Optionally, the method further includes:

[0019] For each of the plurality of GPU monitoring items, according to real-time data of the GPU monitoring item, an alarm message of the GPU monitoring item is output, and the alarm message carries the influence proportion of the GPU monitoring item.

[0020] Optionally, the real-time data of each of the first N GPU monitoring items and / or the historical data of the plurality of groups of GPU monitoring items are obtained through BMC monitoring.

[0021] A second aspect of an embodiment of the application discloses a GPU failure alarm device, and the device includes:

[0022] A data acquisition module is configured to acquire historical data of a plurality of groups of GPU monitoring items and health alarm state data corresponding to the historical data of each group of GPU monitoring items, and each group of GPU monitoring items includes a plurality of GPU monitoring items.

[0023] A data analysis module is configured to analyze the historical data of the plurality of groups of GPU monitoring items and the corresponding health alarm state data, and determine an influence proportion of each of the plurality of GPU monitoring items on the health alarm state of the GPU.

[0024] The fault alarm module is configured to generate a fault alarm message of the GPU according to the respective influence proportion of each of the plurality of GPU monitoring items and real-time data.

[0025] Optionally, the fault analysis module comprises:

[0026] The data processing module is configured to process the historical data of the plurality of sets of GPU monitoring items and the corresponding health alarm state data to obtain a feature matrix.

[0027] The data learning module is configured to perform deep learning based on the plurality of feature matrices to obtain the respective influence proportion of each of the plurality of GPU monitoring items on the health alarm state of the GPU.

[0028] In a third aspect, an electronic device is disclosed, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the GPU fault alarm method according to the first aspect of the present application.

[0029] In a fourth aspect, a computer readable storage medium is disclosed, which stores a computer program / instruction, and the computer program / instruction is executable on a processor to implement the GPU fault alarm method according to the first aspect of the present application.

[0030] The embodiments of the present application have the following advantages:

[0031] In the embodiments of the present application, the influence proportion of each GPU monitoring item on the health alarm state of the GPU is obtained by analyzing the historical data of the plurality of sets of GPU monitoring items and the corresponding health alarm state data of each set of historical data, and then the fault alarm is performed according to the real-time data of each GPU monitoring item and the influence proportion of each GPU monitoring item on the health alarm state of the GPU. Since the fault alarm is performed based on the influence proportion of each GPU monitoring item on the health alarm state, and the overall alarm of the GPU is performed based on the monitoring item data, the method can truly reflect the fault condition of the GPU. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 is a GPU fault alarm method step flowchart provided by the embodiments of the present application;

[0034] Figure 2 is a GPU monitoring item health alarm state influence proportion analysis method step flowchart provided by an embodiment of the present application;

[0035] Figure 3 is a GPU fault alarm method application schematic diagram provided by an embodiment of the present application;

[0036] Figure 4 is a GPU fault alarm device structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more apparent, clear and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0038] In related fault alarm technologies, the health state alarm of a GPU usually includes: Get Thermal Alert State (acquire thermal alarm state), Check External Power (check external power), Get Board Power Supply Status (acquire board power supply status), etc. However, each alarm is independent and cannot truly reflect the fault condition of the GPU. Therefore, in order to overcome the limitations of the prior art, the applicant proposes the following technical concept: according to the historical GPU monitoring item data and the corresponding GPU health alarm state data, the influence proportion of each GPU monitoring item on the health alarm state is analyzed, and the GPU is fault alarmed based on the influence proportion of each GPU monitoring item on the health alarm state, thereby realizing the overall alarm of the GPU based on the GPU monitoring item data and ensuring the accuracy of the alarm.

[0039] Based on the above technical concept, an embodiment of the present application provides a GPU fault alarm method, as shown in Figure 1 , which is a GPU fault alarm method step flowchart provided by an embodiment of the present application. The method comprises the following steps. Figure 1

[0040] Step S101: acquiring historical data of multiple groups of GPU monitoring items and health alarm state data corresponding to the historical data of each group of GPU monitoring items, wherein one group of GPU monitoring items includes multiple GPU monitoring items.

[0041] ​In the embodiment, the historical data of the multiple groups of GPU monitoring items refers to the historical data of the GPU monitoring items at different time points in the past, and the historical data of each group of GPU monitoring items includes historical data of multiple GPU monitoring items. The GPU monitoring items refer to various performance data of the GPU during operation, such as GPU utilization, video memory utilization, video memory occupancy, GPU operating power consumption, GPU temperature, and the like. The health alarm state data corresponding to the historical data of each group of GPU monitoring items refers to the health alarm state data of the GPU at the same time point corresponding to the GPU monitoring item data. In addition, the health alarm state data of the GPU can be represented by a percentage, for example, the health alarm state of the GPU is represented by a percentage between 0 and 100%. The smaller the percentage, the better the running state of the GPU, and the smaller the probability of failure. The larger the percentage, the worse the running state of the GPU, and the larger the probability of failure.

[0042] In the embodiment, the health alarm state data corresponding to the historical data of the GPU monitoring items at different time points is continuously obtained through BMC (Baseboard Management Controller) monitoring. By obtaining sufficient GPU monitoring item data, the accuracy of the GPU failure alarm is ensured based on the collected large amount of GPU monitoring item data and the analysis of the influence of each monitoring item data on the health alarm state of the GPU in the subsequent steps.

[0043] Step S102: Analyzing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data, and determining the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU.

[0044] In the embodiment, the influence proportion of the GPU monitoring item on the GPU health alarm state refers to the influence degree of different monitoring items on the GPU health alarm state (i.e. GPU failure). The greater the influence proportion, the greater the influence degree of the monitoring item on the GPU health alarm state. The smaller the influence proportion, the smaller the influence degree of the monitoring item on the GPU health alarm state, or even no influence. The health alarm state data of the GPU is a comprehensive reflection of the data of each monitoring item of the GPU, that is, the health alarm state data is influenced by multiple GPU monitoring items. For example, as the GPU runs, the temperature of the GPU rises or the running power consumption of the GPU increases, which may influence the health alarm state of the GPU. However, the existing single failure alarm technology has the same influence of each GPU monitoring item data on the health alarm state data, but in fact, each monitoring item has different influence on the health alarm state of the GPU. Therefore, the alarm result of the existing single failure alarm technology cannot truly reflect the health alarm state of the GPU as a whole, and thus it is necessary to analyze the historical data of the GPU monitoring item and the corresponding health alarm state data to determine the influence proportion of each GPU monitoring item data on the health alarm state data.

[0045] In the embodiment, the relationship between the health alarm state data and the data of each GPU monitoring item is found, the relationship between each GPU monitoring item and the health alarm state (i.e. failure of the GPU) is considered from the overall perspective, and thus compared with the previous single alarm failure prediction method, the scheme can more truly reflect the failure condition of the GPU as a whole.

[0046] In an optional embodiment, as shown in Figure 2 the method further includes the following steps:

[0047] Step S201: processing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain a feature matrix.

[0048] In the embodiment, the historical data of the multiple groups of GPU monitoring items at different time points and the corresponding health alarm state data are processed to obtain a feature matrix, and thus different health alarm state data and the coefficients related to each GPU monitoring item corresponding to the health alarm state are included in each feature matrix. In the subsequent steps, learning is performed based on multiple different feature matrices, and thus the relationship between the health alarm state data and the data of each GPU monitoring item, that is, the influence proportion of each monitoring item on the health alarm state, is obtained.

[0049] Specifically, the historical data of the plurality of groups of GPU monitoring items and the corresponding health alarm state data are processed to obtain a feature matrix, including: performing linear regression processing on the historical data of the plurality of groups of GPU monitoring items and the corresponding health alarm state data to obtain a plurality of groups of linear regression results; and creating a feature matrix according to the plurality of groups of linear regression results, wherein elements in the feature matrix include health alarm state data and linear regression coefficients of GPU monitoring items.

[0050] For example, taking the historical data of two groups of GPU monitoring items as an example, the historical data of a GPU monitoring item 1 can be represented as: x 11 , x 12 , x 13 , x 14 , …, and the health alarm state of the GPU at this time is represented as y1; the historical data of a GPU monitoring item 2 can be represented as: x 21 , x 22 , x 23 , x 24 , …, and the health alarm state of the GPU at this time is represented as y2.

[0051] The historical data of each group of GPU monitoring items is subjected to linear regression, and the regression result is represented as:

[0052] y1=k 11 *x 11 +k 12 *x 12 +k 13 *x 13 +k 14 *x 14 +……

[0053] y2=k 21 *x 21 +k 22 *x 22 +k 23 *x 23 +k 24 *x 24 +……

[0054] wherein k 11 , k 12 , k 13 , k 14 , … are linear regression coefficients of the GPU monitoring item data x 11 , x 12 , x 13 , x 14 , …, k 21 , k 22 , k 23 , k 24, …, are linear regression coefficients of GPU monitoring item data x 21 , x 22 , x 23 , x 24 , ….

[0055] Based on the linear regression results of the two groups above, the feature matrix is created as follows:

[0056]

[0057] In this embodiment, by continuously collecting the GPU monitoring item data and the corresponding health alarm state data, and performing linear regression processing on the historical data of each group of GPU monitoring items and the corresponding health alarm state data, a plurality of linear regression results are obtained. The n*m feature matrix is created according to the linear regression results of the historical data of a plurality of groups of GPU monitoring items and the corresponding health alarm state data. n represents the historical data of n groups of GPU monitoring items, and m represents m-1 GPU monitoring items (one column of elements in the matrix is health alarm state data). That is, in the n*m feature matrix, the elements in the same row include the linear regression coefficients of the GPU monitoring data at the same time and the corresponding health alarm state data at the same time. One column of elements represents the health alarm state data at different times, and each of the remaining columns represents the linear regression coefficients of the same monitoring item at different times.

[0058] In this embodiment, the historical data of a plurality of groups of GPU monitoring items and the corresponding health alarm state data are processed into a feature matrix, and then subsequent deep learning is performed. Once, the learning of a plurality of groups of GPU monitoring item data is performed, and the learning efficiency is improved.

[0059] Step S201: performing deep learning based on a plurality of feature matrices to obtain the influence proportion of each of the plurality of GPU monitoring items on the health alarm state of the GPU.

[0060] In this embodiment, the feature matrix includes the health alarm state data and the linear feature coefficients of each GPU monitoring item corresponding to the health alarm state. Therefore, based on a large number of feature matrices, deep learning can obtain linear feature coefficients (i.e., k1, k2, k3, k4, …) of each monitoring item that can more accurately reflect the health alarm state of the GPU. The size of the linear feature coefficients of each GPU monitoring item can accurately reflect which part of the GPU monitoring item has the largest influence proportion on the health alarm state of the entire GPU.

[0061] In an optional embodiment, in order to ensure the accuracy of the proportion of each GPU monitoring item to the health alarm state of the GPU, during the operation of the GPU, the historical data of the GPU monitoring items are continuously obtained and the corresponding feature matrix is generated, learning is performed in real time, and the linear feature coefficients (i.e., k1, k2, k3, k4, …) of each GPU monitoring item are continuously updated according to the result of deep learning, that is, the proportion of each GPU monitoring item to the health alarm state is updated, and then the proportion of each monitoring item with high enough accuracy and reflecting the health alarm state is obtained. Subsequently, when the fault alarm is performed based on the proportion of each monitoring item, more accurate GPU fault alarm can be achieved.

[0062] Step S103: generating a fault alarm message of the GPU according to the real-time data and the proportion of each GPU monitoring item.

[0063] The proportion of each GPU monitoring item to the health alarm state (i.e., GPU fault) of the GPU is determined through the above steps. In this step, the real-time data of each GPU monitoring item is obtained through BMC monitoring, and the alarm message reflecting the health alarm state as a whole is generated according to the real-time data of each GPU monitoring item and the proportion of the monitoring item to the health alarm state of the GPU. Specifically, the real-time health alarm state data is generated according to the real-time data of each GPU monitoring item and the corresponding proportion, and the fault alarm is performed according to the health alarm state data, that is, when the value of the real-time health alarm state data reaches the preset threshold, the alarm message is generated to perform the alarm. For example, the threshold of the GPU fault alarm is 70%, and when the real-time health alarm state data obtained according to the real-time monitoring item data and the corresponding proportion is greater than 70% during the operation of the GPU, the alarm message is generated to perform the alarm. Since the proportion of each GPU monitoring item to the health alarm state is obtained based on the analysis (i.e., deep learning) of the historical data, the health alarm state data obtained based on the real-time GPU monitoring item data and the respective proportion is more accurate, and thus the fault alarm message generated when the fault alarm is performed can better reflect the real GPU fault condition.

[0064] The method provided in the embodiment overcomes the problem of inaccuracy of the previous single alarm. For example, for the previous single alarm, it is considered that each monitoring item has the same influence on the fault of the GPU. When it is found that the data of multiple monitoring items of the GPU changes or is abnormal, an alarm is given based on all the monitoring item data. However, the monitoring item data may not have an influence on the health alarm state of the GPU or the influence is very small, which may result in an incorrect fault level of the alarm or false alarm. In the embodiment, the influence of the GPU monitoring item on the health alarm state is considered, and the alarm is given according to the influence of each GPU monitoring item on the health alarm state, so that the alarm result can more truly reflect the fault of the GPU as a whole.

[0065] In addition, the fault of the GPU can also be pre-alarmed according to the influence of each GPU monitoring item on the health alarm state. Specifically, real-time health alarm state data is generated according to the real-time data of each GPU monitoring item and the corresponding influence, and the GPU fault is pre-alarmed according to the change trend of the health alarm state data. For example, the threshold of the GPU fault alarm is 70%, and during the working process of the GPU, the real-time health alarm state data obtained according to the real-time monitoring item data and the corresponding influence is 60%, and it is detected that the real-time health alarm state data increases all the time along with the working of the GPU. At this time, a corresponding fault pre-alarm message is generated to pre-alarm the GPU fault, so that the staff can check the GPU according to the fault pre-alarm message, and the occurrence of the GPU fault is avoided.

[0066] In the embodiment, the fault is alarmed by the monitoring item data of the GPU and the influence on the health alarm state, so that the influence of each monitoring item of the GPU on the health alarm state as a whole is comprehensively considered, and the GPU is alarmed as a whole based on the monitoring item data. The scheme can more truly reflect the fault of the GPU as a whole.

[0067] In an optional embodiment, the generating of the fault alarm message of the GPU according to the influence and real-time data of each of the plurality of GPU monitoring items comprises:

[0068] According to the influence from large to small, the first N GPU monitoring items are determined from the plurality of GPU monitoring items.

[0069] The fault alarm message of the GPU is generated according to the influence and real-time data of each of the first N GPU monitoring items.

[0070] In the embodiment, the health alarm state is affected by multiple GPU monitoring item data, each GPU monitoring item has different proportion of influence on the health alarm state (i.e. GPU failure), and there are some GPU monitoring item data that have little or even no influence on the GPU failure. For the part of GPU monitoring item data that have little or even no influence, it is not the focus of attention for failure alarm. Therefore, in order to reduce the waste of alarm resources and improve the efficiency of failure alarm, the proportion of influence of each GPU monitoring item on the health alarm state is sorted from large to small, and then the first N GPU monitoring items with large proportion of influence are selected for attention, and an alarm message is generated according to the real-time data of the GPU monitoring items with large proportion of influence to realize GPU failure alarm.

[0071] For example, the proportion of influence of each GPU monitoring item on the health alarm state is obtained through the above steps, and each monitoring is sorted from large to small according to the size of the proportion of influence, and the sorting result is: 1-GPU memory occupancy, 2-GPU running power consumption, 3-GPU temperature, 4-GPU utilization, 5-memory utilization. If only GPU memory occupancy, GPU running power consumption and GPU temperature have large proportion of influence on the health alarm state, only the real-time running data of GPU memory occupancy, GPU running power consumption and GPU temperature need to be viewed in real time, and a corresponding failure alarm message is generated according to the change of the three monitoring item data to perform alarm.

[0072] In an optional embodiment, a threshold of the proportion of influence of each GPU monitoring item on the health alarm state can also be set, and N GPU monitoring items with proportion of influence on the health alarm state exceeding the threshold are selected as the key monitoring item data, that is, the real-time data of the GPU monitoring items with proportion of influence on the health alarm exceeding the threshold is viewed to perform failure alarm. For example, the threshold is set to 0.2 (assuming that the values of the proportion of influence of all GPU monitoring items on the health alarm state are between 0 and 1), and the GPU monitoring items with proportion of influence on the health alarm state greater than or equal to 0.2 are selected as the key monitoring items, and an alarm message is generated according to the change of the real-time data of the key monitoring items to realize GPU failure alarm.

[0073] In the embodiment, according to the proportion of influence of each GPU monitoring item on the failure, the GPU monitoring items with large proportion of influence are selected to alarm the failure of the GPU. Compared with the previous method of monitoring all data for failure alarm, the method reduces the viewing of unnecessary GPU monitoring item data, saves the waste of data monitoring resources, and also ensures the accuracy of GPU failure alarm.

[0074] In an optional embodiment, for each GPU monitoring item in the plurality of GPU monitoring items, an alarm message of the GPU monitoring item is output according to real-time data of the GPU monitoring item, and the alarm message carries an influence proportion of the GPU monitoring item.

[0075] In the embodiment, real-time and health alarm state data are generated according to real-time data of each GPU and the influence proportion on the health alarm state, real-time health alarm state data are generated, a corresponding alarm message is generated when it is found that the real-time health alarm state data reaches a threshold, and fault alarm is realized. The influence proportion of each GPU monitoring item generating the fault alarm message is contained in the generated fault alarm message, and each GPU monitoring item is displayed in order according to the size of the influence proportion. After the alarm message is generated, the alarm message can be input to an external device (such as a display screen) for display, or reminded through LED light, or reminded through voice broadcast and the like. After the alarm message is output, an operator can further process the alarm message by checking the type of the alarm message and the influence proportion of each monitoring item in the alarm message.

[0076] When the alarm message is processed, each GPU monitoring item data in the alarm message is processed according to the influence proportion on the health alarm state data, that is, each monitoring item is processed in order according to the influence proportion from large to small. For example, when the alarm message is received, each GPU monitoring item in the alarm message is processed according to the influence proportion ordering result (the ordering result is: 1-GPU memory occupation, 2-GPU running power consumption, 3-GPU temperature), that is, the GPU memory occupation is processed first, then the GPU running power consumption is processed, and finally the GPU temperature is processed. Specifically, the problems of each monitoring item corresponding hardware or software are investigated in order according to the size of the influence proportion on the health alarm state, and then the possible fault causes are determined and the fault problems are processed in advance. Since the fault alarm is processed based on the influence proportion of each GPU monitoring item, the cause of the fault can be found more accurately and quickly, and the fault can be processed in time.

[0077] Furthermore, in this embodiment, the real-time data and historical data of each GPU monitoring item are obtained through BMC monitoring. Since BMC can provide system status monitoring for remote system management, backplane control such as restart, power restoration, and power failure, FRU information, and a small amount of SEL information entry, and provides IPMI v1.5 messages for local system manageability, ensures the security of remote connections, and can spontaneously perform system health monitoring and corrective actions for serious events, as well as provide LAN warnings, this embodiment utilizes BMC to obtain GPU monitoring item data, ensuring the reliability and security of the collected GPU monitoring item data, thereby guaranteeing accurate alarms for GPU faults.

[0078] Figure 3 This paper illustrates a schematic diagram of an application system for the GPU fault alarm method provided in an embodiment of this application. Figure 3 As shown, the application system includes a GPU module, BMC monitoring, a data processing system, a deep learning system, a fault alarm system, and a fault handling system. The GPU module is part of the server. In practical applications, the BMC monitoring continuously acquires historical data of GPU monitoring items at different times and the corresponding GPU health alarm status data for each historical GPU monitoring item. This acquired historical data and corresponding health alarm status data are then input into the data processing system for processing. The data processing system processes n sets of historical data for GPU monitoring items and their corresponding health alarm status data into a feature matrix. The choice of n can be based on factors such as the computational power and optimal learning efficiency of the deep learning system. The specific processing steps of the data processing system include: performing linear regression on each set of historical data for GPU monitoring items and their corresponding health alarm status data; creating a feature matrix based on the n linear regression results; and continuously processing the input historical data for GPU monitoring items and their corresponding health alarm status data to obtain multiple feature matrices. Each feature matrix contains n GPU health alarm status data and the weight of each GPU monitoring item on each GPU health alarm status data.

[0079] The deep learning system learns based on the plurality of feature matrices to obtain the influence proportion of each GPU monitoring item on the health alarm state. The deep learning process includes: the deep learning system receives the plurality of feature matrices from the data processing system, and performs deep learning according to the feature matrices. After each learning is completed, the influence proportion of each GPU monitoring item on the health alarm state is recorded, and the next learning is started based on the current learning. When the difference between the learning results of the previous and next learning (i.e. the influence proportion of each GPU monitoring item on the health alarm state obtained by the previous and next learning) is less than a preset value, it is considered that the learning is completed. The learning result of the last learning is taken as the final influence proportion of each GPU monitoring item on the health alarm state, and the influence proportion of each GPU monitoring item on the health alarm state is sent to the fault alarm system, so that the fault alarm system performs fault prediction based on the influence proportion of each GPU monitoring item on the health alarm state.

[0080] After receiving the influence proportion of each GPU monitoring item on the health alarm state, the fault alarm system determines the GPU monitoring item with a large influence proportion on the GPU health alarm state as a key monitoring item. The way of determining the key monitoring item includes: 1) sorting according to the influence proportion on the GPU health alarm state from large to small, and selecting the first N GPU monitoring items with large influence proportion as the key monitoring items; 2) setting an influence threshold on the GPU health alarm state, and taking the GPU monitoring item with an influence proportion exceeding the threshold as the key monitoring item. After determining the key monitoring item, the fault alarm system receives real-time data of the key monitoring item collected from the BMC, and generates real-time health alarm state data according to the real-time data of the key monitoring item and the influence proportion of each key monitoring item on the GPU health alarm state. When the health alarm state data reaches a threshold, a fault alarm message is generated, and the fault alarm message is output to the fault processing system, so that the fault processing system processes the fault message. In each alarm message, the real-time data of the monitoring item generating the alarm message and the influence proportion of each monitoring item on the health alarm state are included, and each monitoring item is displayed in order of the influence proportion.

[0081] In addition, in order to ensure the accuracy of the influence proportion of each GPU monitoring item on the health alarm state, the real-time data of the GPU monitoring item collected and the generated health alarm state data are input into the data processing system for processing during the fault alarm process, that is, the real-time data of the GPU monitoring item at n different time points and the health alarm state data corresponding to each time point are processed to obtain a feature matrix. The deep learning system continuously learns based on the new feature matrix. If it is found that the difference between the influence proportion of the new each GPU monitoring item on the health alarm state and the influence proportion of each GPU monitoring item currently used for fault alarm on the health alarm state is greater than a preset value, the influence proportion of the new each GPU monitoring item on the health alarm state is sent to the fault alarm system. Then, the fault alarm system determines the key monitoring item according to the new influence proportion, and generates an alarm message according to the real-time data and the influence proportion of the key monitoring item, so as to realize GPU fault alarm.

[0082] Finally, after receiving the fault message, the fault processing system can remind by displaying the alarm message on the display screen or voice broadcasting. At the same time, the fault processing system processes according to the influence proportion of each key monitoring item on the health alarm state, that is, the monitoring items are processed in order from large to small according to the influence proportion. Specifically, the problems of the hardware or software corresponding to each key monitoring item are investigated according to the influence proportion on the health alarm state, and then the possible fault causes are determined and the fault problem is processed in advance.

[0083] In the embodiment, the influence proportion of each GPU monitoring item on the health alarm state (GPU fault) of the GPU is obtained by analyzing the historical data of the multiple groups of GPU monitoring items and the health alarm state data corresponding to the historical data of each group of monitoring items. Then, the real-time data of each monitoring item and the influence proportion of each monitoring item on the health alarm state of the GPU are used for fault alarm. Since the fault alarm is based on the influence proportion of each monitoring item on the health alarm state of the GPU, the overall alarm of the GPU is realized by the monitoring item data, so that the fault condition of the GPU can be truly and accurately reflected. In addition, a large amount of GPU monitoring item data is collected to obtain sufficient GPU monitoring item data, and the influence proportion of each GPU monitoring item data on the health alarm state is continuously obtained through deep learning, so that more accurate GPU fault alarm is realized. The influence proportion of each GPU monitoring item is used to focus on the monitoring item data with larger influence proportion for fault alarm, which improves the efficiency of fault alarm.

[0084] The embodiment of the application also provides a GPU fault alarm device, which refers to Figure 4 , Figure 4is a structural schematic diagram of a GPU fault alarm device proposed by an embodiment of the present application, as shown in the figure, the device comprises: Figure 4

[0085] a data acquisition module 41, configured to acquire historical data of multiple groups of GPU monitoring items and health alarm state data corresponding to the historical data of each group of GPU monitoring items, one group of GPU monitoring items comprising multiple GPU monitoring items;

[0086] a data analysis module 42, configured to analyze the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data, and determine an influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU;

[0087] a fault alarm module 43, configured to generate a fault alarm message of the GPU according to the influence proportion of each of the multiple GPU monitoring items and real-time data.

[0088] In an optional embodiment, the fault analysis module comprises:

[0089] a data processing module, configured to process the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain a feature matrix;

[0090] a data learning module, configured to perform deep learning based on multiple feature matrices to obtain the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU.

[0091] In an optional embodiment, the data processing module comprises:

[0092] a first data processing unit, configured to perform linear regression processing on the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain multiple groups of linear regression results;

[0093] a second data processing unit, configured to create a feature matrix according to the multiple groups of linear regression results, wherein elements in the feature matrix comprise health alarm state data and linear regression coefficients of GPU monitoring items.

[0094] In an optional embodiment, the fault alarm module comprises:

[0095] a monitoring item confirmation module, configured to determine the first N GPU monitoring items from the multiple GPU monitoring items in descending order of the influence proportion;

[0096] a monitoring item alarm module, configured to generate a fault alarm message of the GPU according to the influence proportion of each of the first N GPU monitoring items and real-time data.

[0097] ​In an alternative embodiment, the apparatus further comprises:

[0098] a fault output module, configured to output, for each GPU monitoring item in the plurality of GPU monitoring items, an alarm message of the GPU monitoring item according to real-time data of the GPU monitoring item, the alarm message carrying an influence proportion of the GPU monitoring item.

[0099] The embodiments of the present application further provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the GPU fault alarm method when executed.

[0100] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction implements the GPU fault alarm method when executed by a processor.

[0101] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0102] The embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method, device, electronic device and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal equipment to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal equipment produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one or more flows and / or blocks. Figure 1 The device for implementing the functions specified in one or more flows and / or blocks.

[0103] These computer program instructions can also be stored in a computer readable memory which can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one or more flows and / or blocks. Figure 1 The device for implementing the functions specified in one or more flows and / or blocks.

[0104] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, thus the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one block or multiple blocks.

[0105] Although the preferred embodiments of the present application have been described, those skilled in the art who understand the basic inventive concept can make additional changes and modifications to the embodiments once they get the basic inventive concept. Therefore, the appended claims are intended to cover all the preferred embodiments and all the changes and modifications falling within the scope of the embodiments of the present application.

[0106] Finally, it should also be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or terminal device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or terminal device including the element.

[0107] The above provides a detailed introduction to the GPU fault alarm method, device, equipment and medium provided by the present application. In this document, specific examples are applied to explain the principles and implementation modes of the present application. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation mode and application range; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A GPU failure alarm method, characterized by, The method comprises: obtaining historical data of multiple groups of GPU monitoring items and health alarm state data corresponding to the historical data of each group of GPU monitoring items, wherein each group of GPU monitoring items comprises multiple GPU monitoring items; analyzing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to determine the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU, comprising: processing the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain a feature matrix, and performing deep learning based on multiple feature matrices to obtain the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU; the feature matrix comprises at least one linear feature coefficient corresponding to a GPU monitoring item; the linear feature coefficient is used to represent the influence proportion of the alarm of the GPU monitoring item on the health alarm state of the entire GPU; generating a fault alarm message of the GPU according to the influence proportion of each of the multiple GPU monitoring items and real-time data.

2. The method of claim 1, wherein, The processing of the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain a feature matrix comprises: performing linear regression processing on the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to obtain multiple groups of linear regression results; creating a feature matrix according to the multiple groups of linear regression results, wherein the elements in the feature matrix comprise health alarm state data and linear regression coefficients of GPU monitoring items.

3. The method of claim 1, wherein, The generating of the fault alarm message of the GPU according to the influence proportion of each of the multiple GPU monitoring items and real-time data comprises: determining the first N GPU monitoring items from the multiple GPU monitoring items in descending order of the influence proportion; generating a fault alarm message of the GPU according to the influence proportion of each of the first N GPU monitoring items and real-time data.

4. The method of claim 1, wherein, The method further comprises: for each of the multiple GPU monitoring items, outputting an alarm message of the GPU monitoring item according to real-time data of the GPU monitoring item, wherein the alarm message carries the influence proportion of the GPU monitoring item.

5. The method according to any of claims 1 to 4, characterized in that, The real-time data of the first N GPU monitoring items and / or the historical data of the multiple groups of GPU monitoring items are obtained through BMC monitoring.

6. A GPU failure alarm device characterized by comprising: The device comprises: a data acquisition module configured to obtain historical data of multiple groups of GPU monitoring items and health alarm state data corresponding to the historical data of each group of GPU monitoring items, wherein each group of GPU monitoring items comprises multiple GPU monitoring items; a data analysis module configured to analyze the historical data of the multiple groups of GPU monitoring items and the corresponding health alarm state data to determine the influence proportion of each of the multiple GPU monitoring items on the health alarm state of the GPU; a fault alarm module configured to generate a fault alarm message of the GPU according to the influence proportion of each of the multiple GPU monitoring items and real-time data; wherein the data analysis module comprises: a data processing module, configured to process historical data of the plurality of groups of GPU monitoring items and corresponding health alarm state data to obtain a feature matrix; a data learning module, configured to perform deep learning based on a plurality of the feature matrices to obtain an influence proportion of each of the plurality of GPU monitoring items on the health alarm state of the GPU; the feature matrix comprises a linear feature coefficient corresponding to at least one GPU monitoring item; and the linear feature coefficient is used to represent an influence proportion of an alarm of a GPU monitoring item on the health alarm state of the entire GPU.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the GPU fault alarm method according to any one of claims 1 to 5 when executed.

8. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction implements the GPU fault alarm method according to any one of claims 1 to 5 when executed by the processor.

Images