Fault determination method, model training method, and electronic device

Abstract

The application discloses a fault determination method, a model training method and an electronic device, and relates to the technical field of computers, and comprises the following steps: performing feature extraction processing on multi-dimensional running data generated in the running process of an image processor by a first model to obtain target features; performing processing on the target features by a target model to obtain a target fault probability distribution corresponding to at least one running data, so as to determine the fault existing in the GPU. Since the target features obtained from the multi-dimensional running data can more comprehensively reflect the information of the GPU, the expression ability and the distinguishing degree of the GPU fault features are improved. The technical problem that the accuracy of determining the GPU fault is low is solved, and the technical effect that the GPU fault is more accurately determined is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a fault determination method, a model training method, and an electronic device. BACKGROUND

[0002] A graphics processing unit (GPU) is a hardware component in a computing device that is specifically used to accelerate graphics rendering, video processing, scientific computing, and the like. If the GPU fails during operation, it can cause data leakage, system crashes, and the like of the computing device.

[0003] In order to discover the fault of the GPU in time, a diagnostic tool can be used to monitor performance data, and generate a prompt message when an exception is discovered. A user can determine the fault of the GPU according to the prompt message and artificial experience. In this way, the fault root cannot be accurately determined, which leads to low accuracy of determining the fault of the GPU. SUMMARY

[0004] The present application provides a fault determination method, a model training method, and an electronic device, to at least solve the problem of low accuracy of determining the fault of the GPU in the related art.

[0005] The present application provides a fault determination method, comprising:

[0006] obtaining at least one running data corresponding to the image processing unit, the at least one running data being multi-dimensional data generated during operation of the image processing unit;

[0007] performing feature extraction processing on the at least one running data by using a first model to obtain target features corresponding to the at least one running data, the target features being used to indicate fault features of the image processing unit;

[0008] performing processing on the target features by using a target model to obtain a target fault probability distribution corresponding to the at least one running data;

[0009] determining a target fault type corresponding to the image processing unit according to the target fault probability distribution.

[0010] The present application provides a model training method, comprising:

[0011] obtaining at least one sample data corresponding to the image processing unit, the at least one sample data being multi-dimensional running data generated during operation of the image processing unit;

[0012] performing feature extraction processing on the at least one sample data by using a first model to obtain sample features corresponding to the at least one sample data;

[0013] According to the sample features corresponding to the at least one sample data, the preset second model is trained to obtain a target model, and the target model is used to determine a target fault type corresponding to the image processor.

[0014] The application further provides a fault determination apparatus, which comprises:

[0015] The acquisition module is configured to acquire at least one running data corresponding to the image processor, the at least one running data being multi-dimensional data generated in a running process of the image processor.

[0016] The feature extraction module is configured to perform feature extraction processing on the at least one running data by using a first model to obtain target features corresponding to the at least one running data, the target features being used to indicate fault features of the image processor.

[0017] The feature processing module is configured to perform processing on the target features by using a target model to obtain a target fault probability distribution corresponding to the at least one running data.

[0018] The determination module is configured to determine a target fault type corresponding to the image processor according to the target fault probability distribution.

[0019] The application further provides a model training apparatus, which comprises:

[0020] The acquisition module is configured to acquire at least one sample data corresponding to the image processor, the at least one sample data being multi-dimensional running data generated in a running process of the image processor.

[0021] The feature extraction module is configured to perform feature extraction processing on the at least one sample data by using a first model to obtain sample features corresponding to the at least one sample data.

[0022] The training module is configured to train a preset second model according to the sample features corresponding to the at least one sample data to obtain a target model, and the target model is used to determine a target fault type corresponding to the image processor.

[0023] The application further provides an electronic device, which comprises a memory configured to store a computer program and a processor configured to execute the computer program to implement the steps of any one of the above fault determination methods or model training methods.

[0024] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of any one of the above fault determination methods or model training methods.

[0025] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the above fault determination methods or model training methods.

[0026] According to the application, the target features obtained from multi-dimensional running data can more comprehensively reflect the information of the GPU, improving the expression ability and distinguishability of the GPU fault features. The fault type determined according to the target features and the target model can solve the technical problem of low accuracy in determining the GPU fault, and achieve the technical effect of more accurately determining the GPU fault. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0028] Figure 1A An application scenario schematic diagram is provided for the embodiments of the application.

[0029] Figure 1B Another application scenario schematic diagram is provided for the embodiments of the application.

[0030] Figure 2 A flowchart of a fault determination method is provided for the embodiments of the application.

[0031] Figure 3 A process schematic diagram of displaying a fault determination result is provided for the embodiments of the application.

[0032] Figure 4 A flowchart of a feature extraction method is provided for the embodiments of the application.

[0033] Figure 5 A flowchart of a model training method is provided for the embodiments of the application.

[0034] Figure 6 A structural schematic diagram of a fault determination device is provided for the embodiments of the application.

[0035] Figure 7 A structural schematic diagram of a model training device is provided for the embodiments of the application.

[0036] Figure 8 A structural schematic diagram of an electronic device is provided for the application. DETAILED DESCRIPTION

[0037] With reference to the drawings and specific embodiments described below, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0038] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0039] In the determination of the fault of the CPU, the multi-dimensional running data generated in the running process of the image processor is subjected to feature extraction processing by the first model to obtain target features. The target features are processed by the target model to obtain a target fault probability distribution corresponding to at least one running data, so as to determine the fault existing in the GPU. Since the target features obtained from the multi-dimensional running data can more comprehensively reflect the information of the GPU, the expression ability and the discrimination degree of the GPU fault features are improved. In this way, the fault type of the GPU determined according to the target features is more accurate. Avoiding the determination of the fault of the GPU according to artificial experience, the case that the fault root cannot be accurately determined is avoided, and the accuracy of the determination of the fault of the GPU is improved.

[0040] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0041] In combination with the specific application environment architecture or the specific hardware architecture on which the execution of the model training method depends, the specific application environment architecture or the specific hardware architecture is described here. For reference Figures 1A-1B .

[0042] Figure 1A An application scenario diagram is provided for the embodiments of the present application. As shown in Figure 1A The diagnostic device of the diagnostic equipment 101 can obtain the performance data of the GPU in the running process of the first equipment 102. And according to the performance data, it is determined whether the performance index of the GPU is greater than or equal to the performance threshold. If the performance index of the GPU is greater than or equal to the performance threshold, the diagnostic device of the diagnostic equipment 101 generates a prompt information. The user can determine the fault of the GPU in the first equipment 102 according to the prompt information.

[0043] Figure 1B This is another application scenario diagram provided by the embodiment of the present application. Figure 1B As shown, the second device 103 is provided with a diagnostic device. The diagnostic device of the second device 103 can obtain performance data of the GPU of the second device 103 during operation. The diagnostic device of the second device 103 can determine whether the performance index of the GPU is greater than or equal to a performance threshold based on the performance data. If the performance index of the GPU is greater than or equal to the performance threshold, the diagnostic device of the second device 103 generates a prompt message. Based on the prompt message, the user can determine that the GPU of the second device 103 is faulty.

[0044] Figure 2 A flowchart of a fault determination method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, an embodiment of the present application provides a fault determination method, which is described in detail as follows:

[0045] S201: Acquire at least one operating data corresponding to an image processor.

[0046] The execution subject of the embodiments of the present application may be an electronic device, or a fault determination device provided in the electronic device. The fault determination device may be implemented via software, or a combination of software and hardware. The electronic device may be a terminal device, a server, or the like. The terminal device may be a mobile phone, a tablet computer, or a computer, or the like.

[0047] The at least one operating data is multi-dimensional data generated during the operation of the image processor.

[0048] The at least one operating data may include performance data, log data, and status data.

[0049] At least one operating data corresponding to the image processor can be obtained in the following manner: collecting at least one first performance data, at least one first log, and at least one first status data during the operation of the image processor; preprocessing the at least one first performance data, at least one first log, and at least one first status data to obtain at least one intermediate data, where the preprocessing includes data cleaning and data normalization; and fusing the at least one intermediate data according to the time when the at least one first performance data, at least one first log data, and at least one first status data are generated to obtain at least one operating data.

[0050] The first performance data may be hardware performance data of the GPU, which may be obtained through a preset interface. The first log may be obtained by parsing a driver log. The first status data may be status data of the GPU system operation.

[0051] For example, the first performance data can be a memory usage rate, a computing core utilization rate, a fan rotation speed, a temperature, a power consumption, etc. The first log can be an error code, a warning information, a driver loading log, etc. The first state data can be a process occupation of GPU resources, a task queue length, etc.

[0052] The data cleaning processing can remove invalid data and outliers. The data normalization processing can make data of different dimensions have a unified dimension and value range. The at least one sample data is sequence data related to a moment.

[0053] S202: performing feature extraction processing on the at least one running data through the first model to obtain a target feature corresponding to the at least one sample data.

[0054] The first model includes a first processing module and a second processing module. The first processing module can be a convolutional neural network, and the second processing module can be a long short-term memory network.

[0055] The target feature is used to indicate a fault feature of the image processor.

[0056] The at least one running data can be processed through the first model to obtain the target feature corresponding to the at least one sample data in the following manner: determining performance data, log data and state data in the at least one running data; performing feature extraction processing on the performance data and the state data through the first processing module to obtain first features; performing semantic analysis processing and feature extraction processing on the log data through the second processing module to obtain second features; and determining the target feature corresponding to the at least one running data according to the first features and the second features.

[0057] Since the sample log has fields such as error codes and warning information, the sample log data can be processed through the second processing module to obtain semantic features. Then, the second processing module is used to perform feature extraction processing on the semantic features corresponding to the sample log data to obtain the second features. In this way, the second features can accurately reflect the fault information in the sample log data, thereby improving the accuracy of determining the GPU fault through the target model.

[0058] For example, the at least one running data includes running data A1 to running data A20, the running performance data in the at least one running data includes running data A1 to running data A10, the running log data includes running data A11 to running data A15, and the running state data includes running data A16 to running data A20. The electronic device performs feature extraction processing on the running data A1 to running data A10 and the running data A16 to running data A20 through the first processing module to obtain a first feature as running feature A1. The electronic device performs semantic analysis processing and feature extraction processing on the running data A11 to running data A15 through the second processing module to obtain a second feature as running feature A2. According to the running feature A1 and the running feature A2, the target feature corresponding to the at least one running data is determined as feature A.

[0059] S203: Processing the target feature through the target model to obtain a target fault probability distribution corresponding to the at least one running data.

[0060] The target model can be a deep learning model.

[0061] For example, processing the target feature through the target model to obtain a target fault probability distribution corresponding to the at least one running data can be as shown in Table 1:

[0062] Table 1

[0063]

[0064] S204: Determining a target fault type corresponding to the image processor according to the target fault probability distribution.

[0065] The target fault type corresponding to the image processor is a fault type corresponding to a fault most likely to occur in the image processor. According to the target probability distribution, the fault type that the target feature is likely to exist can be determined as the target probability of each fault type. The fault type corresponding to the maximum target probability is determined as the fault type existing in the image processor.

[0066] For example, according to Table 1 shown above, it is determined that the maximum target probability is 0.5, and the fault type corresponding to the maximum target probability is fault type A2. Therefore, it can be determined that the target fault type corresponding to the image processor in the electronic device is fault type A2.

[0067] Optionally, the determination result of the fault type of the GPU can be displayed in the form of a chart, a table, etc. through an interface provided by the electronic device.

[0068] For example, the target fault probability distribution is displayed through a column chart, the trend of the health state evaluation value of the GPU over time is displayed through a line chart, or detailed information such as the fault type, the fault position, and the target probability corresponding to the fault type currently detected is listed through a table.

[0069] The following describes the process of displaying the fault determination result in combination with Figure 3 The process of displaying the fault determination result is described. Figure 3 The process of displaying the fault determination result provided by the embodiments of the present application is shown in the schematic diagram. As shown in the schematic diagram, the interface 301 is provided. The electronic device generates a column chart corresponding to the target fault probability distribution and displays the column chart corresponding to the target fault probability distribution through the interface 301 after processing the target feature through the target model to obtain the target fault probability distribution corresponding to at least one running data. The user can click the operation page to obtain the displayed column chart corresponding to the target fault probability distribution. Figure 3

[0070] Optionally, the interface 301 can also be a page provided by a preset device. After the electronic device processes the target feature through the target model to obtain the target fault probability distribution corresponding to at least one running data, the electronic device sends the target fault probability distribution corresponding to at least one running data to the preset device. The preset device generates and displays a column chart according to the target fault probability distribution corresponding to at least one running data. The preset device can be a terminal device used by the user.

[0071] Optionally, a report can also be generated according to the fault type determination result. The report content includes fault diagnosis summary, fault cause analysis, fault treatment suggestion, historical fault comparative analysis, etc., to help the user quickly understand the fault condition of the GPU and take corresponding repair measures.

[0072] The intuitive result display and detailed report generation can provide the user with a convenient fault diagnosis tool and decision support, help him quickly locate the fault cause and develop a reasonable repair scheme, thereby shortening the fault repair time and reducing the operation and maintenance cost.

[0073] It should be noted that the electronic device can determine the fault type of the GPU in the electronic device through the first model and the target model. The fault type of the GPU in other devices can also be determined through the first model and the target model. The present application does not make any limitation.

[0074] The fault determination method provided by the embodiments of the present application can more comprehensively reflect the information of the GPU through the target feature obtained by the multi-dimensional running data, improve the expression ability and distinguishability of the GPU fault feature. In this way, the target model can more accurately determine the fault of the GPU by processing the target feature. The situation that the fault of the GPU cannot be accurately determined according to the artificial experience is avoided, and the accuracy of determining the fault of the GPU is improved.

[0075] On the basis of any one of the above embodiments, the following describes the process of displaying the fault determination result in combination with Figure 4 ​The process of obtaining the target feature through the feature extraction processing of the first model (S202) is described.

[0076] Figure 4 A flowchart of the feature extraction method provided by the embodiments of the present application is shown in Figure 4 The embodiments of the present application provide a feature extraction method, which is described in detail as follows.

[0077] S401: determining performance data, log data and state data in at least one running data.

[0078] The performance data, log data and state data can be determined according to the data type and / or data source of the running data.

[0079] For example, the data type of the running data can be performance type, log type or state type. The data source of the running data can be obtained from an interface, analyzed log or state running from a GPU system.

[0080] S402: performing feature extraction processing on the performance data and the state data through a first processing module to obtain first features.

[0081] Suppose the first processing module is a convolutional neural network, the first features can be obtained by performing feature extraction processing on the performance data and the sample state data through the convolutional neural network. .

[0082] S403: performing semantic analysis processing and feature extraction processing on the log data through a second processing module to obtain second features.

[0083] Suppose the second processing module is a long short-term memory network, the second features can be obtained by performing semantic analysis processing and feature extraction processing on the log data through the long short-term memory network. .

[0084] S404: determining the target feature corresponding to the at least one running data according to the first features and the second features.

[0085] The target feature corresponding to the at least one running data can be determined according to the first features and the second features in the following manner: determining a feature matrix according to the first features and the second features; obtaining a first parameter and a first preset function; determining a first feature value corresponding to the first features and a second feature value corresponding to the second features for any one matrix feature value in the feature matrix; and performing fusion processing on the feature matrix, the first features and the second features according to the first parameter and the first preset function to obtain the sample feature corresponding to the at least one sample data.

[0086] The characteristic matrix can be determined based on the first feature and the second feature in the following manner: obtaining an attention function; determining at least one first eigenvalue in the first feature and at least one second eigenvalue in the second feature; for any first eigenvalue, processing at least one second eigenvalue in the second feature according to the attention function and the first eigenvalue, respectively, to obtain at least one matrix eigenvalue; and determining that the characteristic matrix includes at least one matrix eigenvalue.

[0087] The feature matrix can be an attention score matrix between the first feature and the second feature. for Matrix of order.

[0088] The matrix eigenvalue can be determined by the following formula 1:

[0089]

[0090] in, is the matrix eigenvalue; is the attention function; For the first feature The first eigenvalue; For the second feature The second eigenvalue; is the normalized probability distribution function; is the dimension of the feature vector.

[0091] For example, suppose the first feature is , the second feature is , we can determine that the sample feature matrix is ​​a 3×3 matrix. According to the above formula 1, the matrix feature value is determined to be , determine the matrix eigenvalues , and so on, the sample feature matrix is ​​as follows:

[0092]

[0093] The feature matrix, the first feature, and the second feature can be fused according to the first parameter and the first preset function to obtain a target feature corresponding to at least one operating data in the following manner: the feature matrix and the first feature are processed according to the first parameter and the first preset function to obtain a first intermediate feature; the feature matrix and the second feature are processed according to the first parameter and the first preset function to obtain a second intermediate feature; the first intermediate feature and the second intermediate feature are fused to obtain a target feature corresponding to at least one operating data.

[0094] The first preset function may be an attention pooling function. The first parameter may be a learnable parameter, and the first parameter may be adjusted based on the results of the model training process to obtain the target model or the results of using the first model and the target model, so that the sample features more accurately reflect the GPU fault-related information.

[0095] The target feature corresponding to at least one operating data can be determined by the following formula 2:

[0096]

[0097] in, is a target feature corresponding to at least one operation data; is the first parameter; is the first preset function; is the sample feature matrix. The explanations of other parameters are as above.

[0098] Since the operating data is time-dependent, the first parameter can be adjusted based on the time corresponding to the operating data to adjust the weights of the first and second intermediate features. This allows the sample features to more accurately reflect the expressiveness and discrimination of GPU fault characteristics, thereby improving the accuracy of GPU fault identification based on the target model and target features.

[0099] The feature extraction method provided in the embodiments of this application processes multi-dimensional operational data to obtain target features that can more comprehensively reflect GPU fault information, improving the expressiveness and discriminability of GPU fault features. This improves the accuracy of GPU fault identification based on the target model and target features.

[0100] Optionally, when determining the target fault type corresponding to the GPU using the target model, the electronic device may also determine possible faults of the GPU using the first preset model and the second preset model, and generate prompt information to prompt the user. The first preset model and the second preset model may be set in the electronic device.

[0101] Based on any of the above embodiments, the process of using the first preset model to determine a possible fault of the GPU is described below.

[0102] At least one historical performance data corresponding to the image processor in a historical period and at least one running performance data corresponding to the image processor are acquired; the at least one historical performance data and the at least one running performance data are processed through a first preset model to obtain at least one target performance data of the image processor; for any one target performance data, if a performance value corresponding to the target performance data is greater than or equal to a preset value, a prompt information is generated, the prompt information being used for indicating that a target performance corresponding to the target performance data is abnormal; the prompt information is displayed, or the prompt information is sent to a preset device.

[0103] The first preset model is an autoregressive integrated moving average (ARIMA) model.

[0104] The ARIMA model can be determined through the following formula 3:

[0105]

[0106] wherein, is the order of the autoregressive term; is the order of the differencing term; is the order of the moving average term; is the autoregressive coefficient; is the moving average coefficient; is the lag operator; is the performance data value of the time series at time ; is the white noise error term.

[0107] The autoregressive term represents the relationship between the current observation value and the observation values at the past time points. Herein, represents the influence degree of the observation value at the past time point on the current observation value. The lag operator moves the time series backward, for example, . Through the autoregressive term, the model can capture the trend and periodic changes in the time series, and use the past information to predict the value at the current time.

[0108] The differencing term is used for the stationary processing of the time series. When the time series has a trend or seasonal change, the differencing can eliminate these non-stationary factors, so that the time series becomes stationary. The order of differencing represents the number of times of differencing. For example, when , That is, the first difference of the time series is taken to obtain a new sequence, which represents the difference between the observation values of adjacent two time points.

[0109] Moving average term represents the relationship between the observation value at the current time point and the white noise error term at the past time point. Among them, represents the influence degree of the white noise error term at the past time point on the observation value at the current time point.

[0110] White noise error term is a random variable with mean 0 and constant variance, representing random fluctuations that the model cannot explain. Through the moving average term, the model can capture the random fluctuation characteristics in the time series, and use the past error information to correct the prediction value at the current time point.

[0111] The at least one model parameter of the first preset model includes These parameters are obtained by analyzing and modeling the historical data. Usually, statistical methods such as autocorrelation function and partial autocorrelation function analysis are used to determine the values of and , and the value of is determined by unit root test. The autoregressive coefficient and the moving average coefficient are estimated from the data by maximum likelihood estimation or least squares method, etc. The determination of these parameters makes the first preset model better fit the time series data and be used for future prediction.

[0112] Suppose we have a temperature time series data of a GPU, and after analysis, it is determined that the at least one model parameter of the first preset model is , , , the autoregressive coefficient , , and the moving average coefficient . Then, the first preset model is determined by the above formula 3 as follows:

[0113]

[0114] After expansion, we get:

[0115]

[0116] This model represents the temperature value at the current time point and the temperature values , , at the past three time points, as well as the white noise error terms at the past two time points 、 There is a linear relationship between the temperature data and the error term. Through this model, the temperature value at the current time can be predicted using the temperature data and the error term at the past time, and the target performance value is obtained. If the target performance value is greater than or equal to a preset value, it may indicate that the temperature of the GPU is abnormal, and a prompt information can be generated to prompt the user.

[0117] On the basis of any one of the above embodiments, the process of determining the possible fault of the GPU by the second preset model will be described below.

[0118] At least one running data corresponding to the image processor is obtained; the at least one running data is processed for feature extraction by the first model to obtain a target feature corresponding to the at least one running data; the target feature is processed by the second preset model to obtain a target probability of a fault occurring in the image processor; if the target probability is greater than or equal to a preset probability, a prompt information is generated to indicate that the image processor may have a fault; the prompt information is displayed, or the prompt information is sent to a preset device.

[0119] The second preset model can be a logistic regression model.

[0120] It should be noted that the at least one running data corresponding to the image processor can be obtained by referring to the process of obtaining the at least one sample data described above, which will not be repeated here.

[0121] It should be noted that the process of obtaining the target feature corresponding to the at least one running data can be obtained by referring to the process of obtaining the sample feature corresponding to the at least one sample data, which will not be repeated here.

[0122] Suppose there are target features, respectively , the target probability can be determined by the second preset model through the following formula 4:

[0123]

[0124] Wherein, is the target probability; is a logistic function; is at least one model parameter of the second preset model. Other parameter explanations are described above.

[0125] The at least one model parameter of the second preset model can be obtained by training through the maximum likelihood estimation method. The goal of maximum likelihood estimation is to find a set of parameter values, so that under these parameters, the probability of known fault sample data is maximum. Through the parameters obtained by training, the second preset model can learn the relationship between different target features and the target probability of fault occurrence, so as to predict the fault probability of new data in actual application.

[0126] The logic function can be determined by the following formula 5:

[0127]

[0128] The parameters of formula 5 are explained above.

[0129] The output value of the logic function is between 0 and 1, which can map the feature value of the linear combination to a probability value. When tends to positive infinity, tends to 1; when tends to negative infinity, tends to 0. In the failure probability prediction, the role of the logic function is to convert the feature value of the linear combination into the probability of failure occurrence, so that the probability value is between 0 and 1, which is convenient for probability interpretation and decision-making.

[0130] The linear combination is the core of the logistic regression model, where is the intercept term, which represents the target probability when all feature values are 0, and the log-odds. is the weight coefficient of the feature , which represents the degree of influence of the feature on the probability of failure occurrence. The larger the weight coefficient, the greater the influence of the feature on the probability of failure occurrence.

[0131] Suppose we have two features, GPU memory usage and computing core utilization , and we have obtained the model parameters , , through training. Now there is real-time running data of a GPU, with GPU memory usage of 80% and computing core utilization of 70%, i.e. , . Substituting the above data into the linear combination, we get the following expression:

[0132]

[0133] Through the above formula 4 and formula 5, the target probability is determined as follows:

[0134]

[0135] According to the second preset model, the target probability corresponding to the CPU in the real-time running state of the GPU is 85.8%. Users can judge whether further inspection or maintenance measures need to be taken according to this probability value to prevent potential failure occurrence.

[0136] When the target model is used to determine the target fault type corresponding to the GPU, the electronic device can further determine the possible faults of the GPU through the first preset model and the second preset model, improving the flexibility of determining the faults of the GPU. The prompt information can be generated in real time, so that the user can know the potential faults of the GPU in advance, take preventive maintenance measures in time, avoid further expansion of the faults, reduce the system downtime and data loss risk caused by the faults, and significantly improve the reliability and usability of the GPU system.

[0137] The target model can be obtained by training the preset second model. Based on any one of the above embodiments, the process of training the model will be described below in combination with Figure 5 The process of training the model will be described below.

[0138] Figure 5 The flowchart of the model training method provided by the embodiments of the present application is shown in Figure 5 The embodiments of the present application provide a model training method, and the method will be described in detail as follows.

[0139] S501: Obtain at least one sample data corresponding to an image processor.

[0140] The execution subject of the embodiments of the present application can be an electronic device or a model training device arranged in the electronic device. The model training device can be implemented by software or by a combination of software and hardware. The electronic device can be a terminal device, a server, etc. The terminal device can be a mobile phone, a tablet computer, a computer, etc.

[0141] The at least one sample data can include sample performance data, sample log data, and sample state data.

[0142] It should be noted that the execution process of S501 can refer to S201, which will not be described herein again.

[0143] S502: Perform feature extraction processing on the at least one sample data through the first model to obtain sample features corresponding to the at least one sample data.

[0144] It should be noted that the execution process of S501 can refer to the feature extraction processing process provided in any one of the above embodiments, which will not be described herein again.

[0145] S503: Obtain a sample fault probability distribution corresponding to the at least one sample feature.

[0146] The sample fault probability distribution can be a sample fault probability distribution obtained in an actual use process.

[0147] The sample fault probability distribution includes at least one sample fault type and a sample probability that a fault type possibly existing in the sample feature is the sample fault type.

[0148] For example, the at least one sample fault type includes fault type A1~fault type A5. The sample fault probability distribution can be specifically shown in Table 2:

[0149] Table 2

[0150]

[0151] S504: Determine that the sample data set includes at least one sample feature and a sample fault probability distribution corresponding to the at least one sample feature.

[0152] For example, the sample data set can be specifically shown in Table 3:

[0153] Table 3

[0154]

[0155] S505: Perform first training processing on the second model according to the at least one sample feature to obtain a first intermediate model.

[0156] The first intermediate model can be obtained by: performing first iteration processing on the at least one sample feature by the second model to obtain a predicted fault probability distribution corresponding to each sample feature; determining a target loss value according to the predicted fault probability distribution corresponding to each sample feature and the sample fault probability distribution; and updating at least one model parameter of the second model according to the target loss value to obtain the first intermediate model.

[0157] The target loss value can be determined by a loss function.

[0158] S506: Perform i-th training processing on the i-1-th intermediate model according to the at least one sample feature to obtain an i-th intermediate model, until the i-th intermediate model converges, and determine the i-th intermediate model as a target model.

[0159] i is 2, 3, …, N.

[0160] The i-th intermediate model can be obtained in the following manner until the i-th intermediate model converges and the i-th intermediate model is determined as the target model: performing the i-th iterative processing on at least one sample feature through the second model to obtain the predicted fault probability distribution corresponding to each sample feature; obtaining the sample weight and regularization coefficient corresponding to the sample feature; determining the first loss value according to the sample weight corresponding to each sample feature, the sample fault probability distribution corresponding to each sample feature, and the predicted fault probability distribution corresponding to each sample feature; determining the second loss value according to the sample weight corresponding to each sample feature and the regularization coefficient; determining the sum of the first loss value and the second loss value as the target loss value; if the target loss value is greater than the preset loss value or N is less than the preset number of times, performing the i+1-th training processing on the i-th intermediate model to obtain the i+1-th intermediate model; if the target loss value is less than or equal to the preset loss value, and N is greater than or equal to the preset number of times, determining that the i-th intermediate model converges, and determining the i-th intermediate model as the target model.

[0161] During the training process, in order to avoid the problem of unbalanced sample features, the weighted cross entropy loss function is used to determine the target loss value.

[0162] The target loss value can be determined by the following formula 6:

[0163]

[0164] in, The target loss value corresponding to the i-th iteration processing; is the first loss value corresponding to the i-th iteration; The second loss value corresponding to the i-th iteration process.

[0165] The following formula 7 can be used to determine the The first loss value corresponding to the iterative processing:

[0166]

[0167] in, For the The sample weight corresponding to each sample feature; For the The sample failure probability distribution corresponding to the sample features; For the The predicted failure probability distribution corresponding to the sample features; For Take the logarithm. The other parameters are explained above.

[0168] The following formula 8 can be used to determine the The second loss value corresponding to the iterative processing:

[0169]

[0170] wherein, is a regularization coefficient. The explanations of other parameters are as described above. is used to control the strength of regularization, and improve the generalization ability of the target model.

[0171] The sample weight corresponding to the sample feature can be obtained by the following manner: obtaining at least one sample fault type; determining a first number of sample features corresponding to each sample fault type according to a sample fault probability distribution corresponding to each sample feature; and determining a sample weight corresponding to each sample feature according to the first number and a second number of at least one sample fault type.

[0172] The sample weight of the xth sample feature can be determined by the following formula 9:

[0173]

[0174] wherein, is the first number; is the second number. The explanations of other parameters are as described above.

[0175] If the target loss value is greater than the preset loss value or N is less than the preset number of times, at least one model parameter of the ith intermediate model is updated according to the target loss value, and the i+1th intermediate model is obtained.

[0176] The target loss value is less than or equal to the preset loss value, and N is greater than or equal to the preset number of times, indicating that the target loss value tends to be stable in the case of being less than or equal to the preset loss value. By minimizing the target loss value, each intermediate model can effectively learn the mapping relationship between the fault feature and the fault type, and avoid overfitting in the training process, thereby improving the accuracy and robustness of the target model.

[0177] The model training method provided by the embodiments of the present application can more comprehensively reflect the information of the GPU by using the sample features obtained from the multi-dimensional running data, thereby improving the expression ability and distinguishability of the GPU fault features. In this way, the target model obtained by training can more accurately determine the fault of the GPU. The determination of the fault of the GPU according to the artificial experience can avoid the case that the fault root cannot be accurately determined, thereby improving the accuracy of determining the fault of the GPU.

[0178] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0179] Figure 6This is a schematic diagram of the structure of the fault determination device provided in the embodiment of the present application. Figure 6 As shown, an embodiment of the present application further provides a fault determination device 600, comprising:

[0180] An acquisition module 601 is configured to acquire at least one piece of operating data corresponding to an image processor, where the at least one piece of operating data is multi-dimensional data generated during the operation of the image processor;

[0181] A feature extraction module 602 is configured to perform feature extraction processing on at least one operating data using a first model to obtain a target feature corresponding to the at least one operating data, where the target feature is used to indicate a fault feature of the image processor;

[0182] A feature processing module 603 is configured to process target features using a target model to obtain a target fault probability distribution corresponding to at least one operating data;

[0183] The determination module 604 is configured to determine the target fault type corresponding to the image processor according to the target fault probability distribution.

[0184] In a possible implementation, the feature extraction module 602 is specifically configured to:

[0185] determining performance data, log data, and status data in the at least one operational data;

[0186] Performing feature extraction processing on the performance data and the status data by a first processing module to obtain a first feature;

[0187] Performing semantic analysis and feature extraction on the log data by a second processing module to obtain a second feature;

[0188] A target feature corresponding to at least one operating data is determined based on the first feature and the second feature.

[0189] In a possible implementation, the feature extraction module 602 is specifically configured to:

[0190] Determine a feature matrix based on the first feature and the second feature;

[0191] Obtaining a first parameter and a first preset function;

[0192] According to the first parameter and the first preset function, the feature matrix, the first feature and the second feature are fused to obtain a target feature corresponding to at least one operating data.

[0193] In a possible implementation, the feature extraction module 602 is specifically configured to:

[0194] Get the attention function;

[0195] determining at least one first feature value in the first feature, and at least one second feature value in the second feature;

[0196] for any one first feature value, processing the at least one second feature value in the second feature according to the attention function and the first feature value, to obtain at least one matrix feature value;

[0197] determining that the feature matrix includes the at least one matrix feature value.

[0198] In a possible implementation, the feature extraction module 602 is specifically configured to:

[0199] processing the feature matrix and the first feature according to the first parameter and the first preset function, to obtain a first intermediate feature;

[0200] processing the feature matrix and the second feature according to the first parameter and the first preset function, to obtain a second intermediate feature;

[0201] fusing the first intermediate feature and the second intermediate feature, to obtain the target feature corresponding to the at least one running data.

[0202] In a possible implementation, the determining module 604 is further configured to:

[0203] obtaining at least one historical performance data corresponding to the image processor in a historical period, and at least one running performance data corresponding to the image processor;

[0204] processing the at least one historical performance data and the at least one running performance data through a first preset model, to obtain at least one target performance data of the image processor;

[0205] for any one target performance data, if a performance value corresponding to the target performance data is greater than or equal to a preset value, generating a prompt information, the prompt information being used to indicate that a target performance corresponding to the target performance data is abnormal;

[0206] displaying the prompt information, or sending the prompt information to a preset device.

[0207] In a possible implementation, the determining module 604 is further configured to:

[0208] obtaining at least one running data corresponding to the image processor;

[0209] performing feature extraction processing on the at least one running data through a first model, to obtain a target feature corresponding to the at least one running data;

[0210] processing the target feature through a second preset model, to obtain a target probability of failure of the image processor;

[0211] If the target probability is greater than or equal to the preset probability, a prompt information is generated, and the prompt information is used to indicate that the image processor may have a fault;

[0212] The prompt information is displayed, or the prompt information is sent to a preset device.

[0213] The features of the embodiments of the fault determination apparatus can be referred to the related descriptions of the embodiments of the fault determination method, which will not be repeated here.

[0214] Figure 7 A structural schematic diagram of a model training apparatus provided by the embodiments of the present application is shown in FIG. 7. Figure 7 As shown in FIG. 7, the embodiments of the present application further provide a model training apparatus 700, which comprises:

[0215] An acquisition module 701 is configured to acquire at least one sample data corresponding to the image processor, wherein the at least one sample data is multi-dimensional running data generated in a running process of the image processor.

[0216] A feature extraction module 702 is configured to perform feature extraction processing on the at least one sample data by using a first model to obtain sample features corresponding to the at least one sample data.

[0217] A training module 703 is configured to train a preset second model according to the sample features corresponding to the at least one sample data to obtain a target model, wherein the target model is used to determine a target fault type corresponding to the image processor.

[0218] In a possible implementation, the training module 703 is specifically configured to:

[0219] acquire a sample fault probability distribution corresponding to the at least one sample feature;

[0220] determine that a sample data set comprises the at least one sample feature and the sample fault probability distribution corresponding to the at least one sample feature;

[0221] train the second model according to the sample data set to obtain the target model.

[0222] In a possible implementation, the training module 703 is specifically configured to:

[0223] perform a first training processing on the second model according to the at least one sample feature to obtain a first intermediate model;

[0224] perform an i-th training processing on an (i-1)-th intermediate model according to the at least one sample feature to obtain an i-th intermediate model, until the i-th intermediate model converges, and the i-th intermediate model is determined as the target model, wherein i is 2, 3, …, or N.

[0225] In a possible implementation, the training module 703 is specifically configured to:

[0226] perform i-th iteration processing on the at least one sample feature by using the second model, to obtain a predicted failure probability distribution corresponding to each sample feature;

[0227] obtain a sample weight corresponding to the sample feature and a regularization coefficient;

[0228] determine a first loss value according to the sample weight corresponding to each sample feature, the sample failure probability distribution corresponding to each sample feature, and the predicted failure probability distribution corresponding to each sample feature;

[0229] determine a second loss value according to the sample weight corresponding to each sample feature and the regularization coefficient;

[0230] determine a target loss value as a sum of the first loss value and the second loss value;

[0231] if the target loss value is greater than a preset loss value or N is less than a preset number of times, perform i+1-th training processing on the i-th intermediate model to obtain an i+1-th intermediate model;

[0232] if the target loss value is less than or equal to the preset loss value and N is greater than or equal to the preset number of times, it is determined that the i-th intermediate model converges, and the i-th intermediate model is determined as the target model.

[0233] In a possible implementation, the training module 703 is specifically configured to:

[0234] obtain at least one sample failure type;

[0235] determine a first number of sample features corresponding to each sample failure type according to the sample failure probability distribution corresponding to each sample feature;

[0236] determine a sample weight corresponding to each sample feature according to the first number and a second number of the at least one sample failure type.

[0237] The description of the features in the embodiment of the model training apparatus can refer to the related description of the embodiment of the model training method, which will not be repeated here.

[0238] Figure 8 The structural schematic diagram of the electronic device provided in the present application is shown in FIG. 8. As shown in FIG. 8, the electronic device 800 provided in the present embodiment includes at least one processor 801 and a memory 802. Optionally, the electronic device 800 further includes a communication component 803. The processor 801, the memory 802, and the communication component 803 are connected through a bus. Figure 8

[0239] ​In the implementation process, the at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that the at least one processor 801 performs the model training method embodiments described above.

[0240] The specific implementation process of the processor 801 can refer to the method embodiments described above, which have similar implementation principles and technical effects, and details are not described here.

[0241] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0242] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0243] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0244] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above model training method embodiments when running.

[0245] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0246] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the steps in any of the model training method embodiments described above.

[0247] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in any of the model training method embodiments described above.

[0248] The skilled person can further realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0249] The above provides a fault determination method, a model training method and an electronic device. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only to help understand the method of the present application and its core idea. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A fault determination method, characterized in that: include: Acquire at least one piece of operating data corresponding to the image processor, where the at least one piece of operating data is multi-dimensional data generated during the operation of the image processor, and the at least one piece of operating data includes performance data, log data, and status data; performing feature extraction processing on the at least one operating data using a first model to obtain a target feature corresponding to the at least one operating data, wherein the target feature is used to indicate a fault feature of the image processor, wherein the first model includes a first processing module and a second processing module, the first processing module is a convolutional neural network, and the second processing module is a long short-term memory network; Obtaining a target feature corresponding to the at least one operating data includes: determining a feature matrix based on the first feature and the second feature; obtaining a first parameter and a first preset function; and fusing the feature matrix, the first feature, and the second feature based on the first parameter and the first preset function to obtain the target feature corresponding to the at least one operating data; The target feature is processed by a target model to obtain a target fault probability distribution corresponding to the at least one operating data, the target model is obtained by training a preset second model, and the model training method includes: obtaining at least one sample data corresponding to the image processor, the at least one sample data being multi-dimensional operating data generated during the operation of the image processor; performing feature extraction processing on the at least one sample data by a first model to obtain a sample feature corresponding to the at least one sample data; training the preset second model based on the sample feature corresponding to the at least one sample data to obtain a target model, the target model being used to determine a target fault type corresponding to the image processor; A target fault type corresponding to the image processor is determined according to the target fault probability distribution.

2. The fault determination method according to claim 1, characterized in that: Performing feature extraction processing on the at least one operating data using the first model to obtain a target feature corresponding to the at least one operating data includes: determining performance data, log data, and status data in the at least one operational data; Performing feature extraction processing on the performance data and the status data by the first processing module to obtain a first feature; Performing semantic analysis and feature extraction on the log data by the second processing module to obtain a second feature; A target feature corresponding to the at least one operating data is determined according to the first feature and the second feature.

3. The fault determination method according to claim 1, wherein: Determining a feature matrix according to the first feature and the second feature includes: Get the attention function; determining at least one first feature value of the first feature and at least one second feature value of the second feature; For any first eigenvalue, processing at least one second eigenvalue in the second features according to the attention function and the first eigenvalue to obtain at least one matrix eigenvalue; Determining the characteristic matrix includes determining the at least one matrix eigenvalue.

4. The fault determination method according to claim 1, characterized in that: The method further comprises: performing a fusion process on the feature matrix, the first feature, and the second feature according to the first parameter and the first preset function to obtain a target feature corresponding to the at least one operating data. Processing the feature matrix and the first feature according to the first parameter and the first preset function to obtain a first intermediate feature; Processing the feature matrix and the second feature according to the first parameter and the first preset function to obtain a second intermediate feature; The first intermediate feature and the second intermediate feature are fused to obtain a target feature corresponding to the at least one operating data.

5. The fault determination method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire at least one historical performance data corresponding to the image processor within a historical period, and at least one operating performance data corresponding to the image processor; Processing the at least one historical performance data and the at least one operating performance data using a first preset model to obtain at least one target performance data of the image processor; For any target performance data, if the performance value corresponding to the target performance data is greater than or equal to a preset value, a prompt message is generated, where the prompt message is used to indicate that the target performance corresponding to the target performance data is abnormal; Display the prompt information, or send the prompt information to a preset device.

6. The fault determination method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquiring at least one operating data corresponding to the image processor; performing feature extraction processing on the at least one operating data using the first model to obtain a target feature corresponding to the at least one operating data; Processing the target feature by a second preset model to obtain a target probability of the image processor failing; If the target probability is greater than or equal to a preset probability, generating a prompt message, wherein the prompt message is used to indicate that the image processor may fail; Display the prompt information, or send the prompt information to a preset device.

7. The fault determination method according to claim 1, characterized in that: Training the second model according to the sample feature corresponding to the at least one sample data to obtain a target model includes: Obtaining a sample failure probability distribution corresponding to at least one sample feature; Determining that the sample data set includes the at least one sample feature and a sample failure probability distribution corresponding to the at least one sample feature; The second model is trained according to the sample data set to obtain the target model.

8. The fault determination method according to claim 7, characterized in that: Training the second model according to the sample data set to obtain the target model includes: Performing a first training process on the second model according to the at least one sample feature to obtain a first intermediate model; According to the at least one sample feature, the i-1th intermediate model is trained for the i-th time to obtain the i-th intermediate model, until the i-th intermediate model converges, and the i-th intermediate model is determined as the target model, where i is 2, 3, ..., N.

9. The fault determination method according to claim 8, characterized in that: According to the at least one sample feature, performing an i-th training process on the i-1th intermediate model to obtain an i-th intermediate model, until the i-th intermediate model converges, and determining the i-th intermediate model as the target model, comprising: Performing an i-th iterative process on the at least one sample feature using the second model to obtain a predicted fault probability distribution corresponding to each sample feature; Obtaining sample weights and regularization coefficients corresponding to the sample features; Determine a first loss value according to a sample weight corresponding to each sample feature, a sample failure probability distribution corresponding to each sample feature, and a predicted failure probability distribution corresponding to each sample feature; Determining a second loss value according to the sample weight corresponding to each sample feature and the regularization coefficient; determining the sum of the first loss value and the second loss value as a target loss value; If the target loss value is greater than the preset loss value or N is less than the preset number of times, the i+1th training process is performed on the i-th intermediate model to obtain the i+1th intermediate model; If the target loss value is less than or equal to the preset loss value, and N is greater than or equal to the preset number of times, it is determined that the i-th intermediate model has converged, and the i-th intermediate model is determined as the target model.

10. The fault determination method according to claim 9, characterized in that: Obtaining the sample weight corresponding to the sample feature includes: Obtain at least one sample fault type; Determining a first number of sample features corresponding to each sample fault type according to a sample fault probability distribution corresponding to each sample feature; A sample weight corresponding to each sample feature is determined according to the first number and the second number of the at least one sample fault type.

11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 10 when executing the computer program.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images