Methods, apparatus, and media of diagnosing and optimizing data analytics systems

Abstract

Embodiments of the present disclosure relate to a method, device and computer storage medium for diagnosing and optimizing a data analysis model. The diagnosis method comprises: obtaining information about an analysis engine in a data analysis system to be diagnosed and data to be diagnosed associated with the analysis engine; determining a diagnosis engine corresponding to the analysis engine based on the received information; and providing an uncertainty measure associated with at least one of the analysis engine and the data to be diagnosed by processing the data to be diagnosed using the diagnosis engine. Based on such a method, the diagnosis engine corresponding to the analysis engine can be used to determine the impact of the model or the data to be diagnosed on the accuracy of the prediction of the analysis engine of the data analysis system, thereby providing support for users to select a suitable data analysis model or determine possible defects of the data to be diagnosed.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of artificial intelligence, and more particularly, to methods, devices and computer storage media for diagnosing and optimizing a data analysis system. BACKGROUND

[0002] With the continuous development of computer technology, data analysis systems have been widely applied in various aspects of people's life, and analysis engines such as deep neural networks are increasingly used in various types of tasks such as image recognition, speech recognition and target detection. Generally, training data needs to be used to train the analysis engine, but the analysis engine often cannot achieve the performance of the training stage in the prediction process. There can be many reasons for the performance decline of the analysis engine in the prediction stage, and it is often difficult for developers to find defects in the analysis engine and make targeted improvements, which is undesirable. SUMMARY

[0003] Embodiments of the present disclosure provide a scheme for diagnosing and optimizing a data analysis system.

[0004] According to a first aspect of the present disclosure, a method for diagnosing a data analysis system is provided. The method comprises: obtaining information about an analysis engine in the data analysis system to be diagnosed and diagnosis data associated with the analysis engine, the diagnosis data comprising at least one of training data and prediction data for the analysis engine; determining a diagnosis engine corresponding to the analysis engine based on the received information; and providing an uncertainty measure associated with at least one of the analysis engine and the diagnosis data by processing the diagnosis data using the diagnosis engine, the uncertainty measure indicating a degree of certainty of a prediction result obtained by the analysis engine for processing the diagnosis data.

[0005] According to a second aspect of the present disclosure, a method for optimizing a data analysis system is provided. The method comprises: providing information about an analysis engine in the data analysis system and diagnosis data associated with the analysis engine to a diagnosis system, the diagnosis data comprising at least one of training data and prediction data for the analysis engine; receiving an uncertainty measure associated with at least one of the analysis engine and the diagnosis data from the diagnosis system, the uncertainty measure indicating a degree of certainty of a prediction result obtained by the analysis engine for processing the diagnosis data; and updating the analysis engine based on the uncertainty measure.

[0006] According to a third aspect of the present disclosure, a device for diagnosing a data analytics system is provided. The device comprises: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which when executed by the at least one processing unit, cause the device to perform actions comprising: obtaining information about an analytics engine in the data analytics system to be diagnosed and to-be-diagnosed data associated with the analytics engine, the to-be-diagnosed data comprising at least one of training data and prediction data for the analytics engine; determining, based on the received information, a diagnosis engine corresponding to the analytics engine; and providing, by processing the to-be-diagnosed data with the diagnosis engine, an uncertainty measure associated with at least one of the analytics engine and the to-be-diagnosed data, the uncertainty measure indicating a degree of certainty of a prediction result obtained by the analytics engine for processing the to-be-diagnosed data.

[0007] According to a fourth aspect of the present disclosure, a device for optimizing a model analytics system is provided. The device comprises: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which when executed by the at least one processing unit, cause the device to perform actions comprising: sending, to a diagnosis system, information about an analytics engine in the data analytics system and to-be-diagnosed data associated with the analytics engine, the to-be-diagnosed data comprising at least one of training data and prediction data for the analytics engine; receiving, from the diagnosis system, an uncertainty measure associated with at least one of the analytics engine and the to-be-diagnosed data, the uncertainty measure indicating a degree of certainty of a prediction result obtained by the analytics engine for processing the to-be-diagnosed data; and updating the analytics engine based on the uncertainty measure.

[0008] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer readable program instructions for carrying out the method according to the first aspect.

[0009] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer readable program instructions for carrying out the method according to the second aspect.

[0010] The summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. The summary is not intended to identify key or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like elements throughout the figures, and in which:

[0012] Figure 1 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented;

[0013] Figure 2 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented;

[0014] Figure 3 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented;

[0015] Figure 4 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented;

[0016] Figure 5 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented;

[0017] Figure 6 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented;

[0018] Figures 7A-7F FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented; and

[0019] Figure 8 FIG. 1 illustrates a schematic diagram of an environment in which embodiments of the present disclosure can be implemented; DETAILED DESCRIPTION

[0020] Preferred embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0021] The term "comprising" and variations thereof as used herein are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal sense unless expressly so defined herein.

[0022] As described above, with the continuous development of computer technology, data analysis systems have been widely applied in various aspects of people's life, and analysis engines such as deep neural networks are increasingly used in various types of tasks such as image recognition, speech recognition and target detection. People increasingly rely on the prediction results of data analysis systems, especially in critical fields such as medical diagnosis, financial investment, business and public decision-making, the reliability and stability of the prediction results are particularly important. However, the analysis engine often cannot achieve the performance in the training phase during the prediction process. Therefore, people are increasingly concerned about how to determine the defects of the analysis engine or the training data / prediction data in the data analysis system and make targeted improvements.

[0023] According to some embodiments of the present disclosure, a scheme for diagnosing a data analysis system is provided. In the scheme, first, information about an analysis engine in the data analysis system to be diagnosed and to-be-diagnosed data associated with the analysis engine can be obtained, wherein the to-be-diagnosed data includes at least one of training data and prediction data for the analysis engine. Subsequently, a diagnosis engine corresponding to the analysis engine can be determined based on the received information, and the to-be-diagnosed data is processed by utilizing the diagnosis engine to provide an uncertainty measure associated with at least one of the analysis engine and the to-be-diagnosed data, wherein the uncertainty measure indicates a degree of certainty of a prediction result obtained by the analysis engine for processing the to-be-diagnosed data. In this way, the diagnosis engine corresponding to the analysis engine can be used to determine the impact of the model or the to-be-diagnosed data (including training data and prediction data) on the accuracy of the prediction of the analysis engine of the data analysis system, thereby providing support for the user to determine possible defects of the model or the to-be-diagnosed data. In addition, by utilizing a separate diagnosis engine to diagnose the data analysis system, the developer can be provided with diagnosis suggestions for the data analysis system in the case of affecting the normal operation of the data analysis system.

[0024] According to some embodiments of the present disclosure, a scheme for optimizing a data analysis system is also provided. In the scheme, first, information about an analysis engine in the data analysis system and to-be-diagnosed data associated with the analysis engine can be provided to a diagnosis system, wherein the to-be-diagnosed data includes at least one of training data and prediction data for the analysis engine. Subsequently, an uncertainty measure associated with at least one of the analysis engine and the to-be-diagnosed data is received from the diagnosis system, wherein the uncertainty measure indicates a degree of certainty of a prediction result obtained by the analysis engine for processing the to-be-diagnosed data. Then, the analysis engine can be updated based on the uncertainty measure. In this way, the analysis engine can be updated based on the uncertainty measure determined by the analysis engine, thereby improving the performance of the data analysis system.

[0025] Example Environment

[0026] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, the example environment 100 includes a data analysis system 110 and a diagnostic system 140. The data analysis system 110 includes an analysis engine 125, which can learn certain knowledge and capabilities from existing data to process new data. The analysis engine 125 can be designed to perform various tasks, such as image classification, object detection, speech recognition, machine translation, content filtering, etc. Examples of analysis engines 125 include, but are not limited to, various deep neural networks (DNNs), convolutional neural networks (CNNs), support vector machines (SVMs), decision trees, random forest models, etc.

[0027] like Figure 1 As shown, the data analysis system 110 also includes training data 115 for training the analysis engine 125 and prediction data 120 for prediction by the analysis engine 125. In some embodiments, the training data 115 and prediction data 120 applied to the analysis engine 125 are data such as images, medical data, business data, etc., and the analysis engine 125 can provide the results obtained from data analysis (also referred to as prediction results). Specifically, the analysis engine 125 can perform at least one of the following operations on the input data (including training data 115 and prediction data 120): prediction, recognition, classification, pattern analysis, feature detection, and extraction. In some embodiments, the analysis engine 125 can provide recognition predictions for the input data; for example, when the input data is an image, the prediction can be a recognition result for the image (e.g., the object included in the image is a cat or a dog). In some embodiments, the analysis engine 125 can provide predictions for the input data; for example, when the input data is business data, the prediction can be a predictive analysis by the analysis engine 125 of sales volume, customer churn rate, ad click-through rate, etc.

[0028] The data analysis system 110 can provide the diagnostic system 140 with information 130 related to the analysis engine 125 and diagnostic data 135 via an interface with the diagnostic system 140, wherein the diagnostic data includes at least one of training data 115 and prediction data 120. In some embodiments, the interface between the data analysis system 110 and the diagnostic system 140 can be a physical communication interface. Alternatively, the interface between the data analysis system 110 and the diagnostic system 140 can also be a software-implemented data communication interface. In some embodiments, the information 130 may include information indicating the type and network structure of the analysis engine 125. For example, when the analysis engine 125 includes a deep neural network, the information 130 may include information indicating the specific structure of the deep neural network, such as information indicating that the neural network includes a 3-layer MLP (Multilayer Perceptron), and the number of hidden units in each layer is 45, 135, and 405, respectively. It should be understood that the information 130 may not include information about the specific weights in the analysis engine 125.

[0029] like Figure 1 As shown, the diagnostic system 140 includes a diagnostic engine 145. Similar to the analysis engine 125, the diagnostic engine 145 is also capable of learning certain knowledge and abilities from existing data to process new data. In some embodiments, the diagnostic engine 145 can provide a quantitative measure of the degree of uncertainty regarding the prediction outcome (hereinafter referred to as the "uncertainty measure"). In some embodiments, in response to the analysis engine 125 including a neural network module, the diagnostic system 140 can construct the neural network module based on received information 130 related to the analysis engine 125 and update the neural network module using probabilistic inference as part of the diagnostic engine 145. In the diagnostic engine 145, weights are represented by a specific distribution rather than fixed values. In some embodiments, for example, the diagnostic engine 145 may include a Bayesian neural network (BNN). The diagnostic system 140 can provide the uncertainty measure 150 by processing the received data 135 to be diagnosed using the diagnostic engine 145. The data analysis system 110 can update the analysis engine accordingly upon receiving the uncertainty measure 150. The following details the process by which the diagnostic system 140 generates the uncertainty measure 150 and the process by which the data analysis system 110 optimizes the analysis engine 125 based on the uncertainty measure 150 determined by the diagnostic system 140.

[0030] Diagnosis of a data analytics system

[0031] The following will refer to Figures 2-4 To describe in more detail the process used in diagnostic data analysis systems. Figure 2 A flowchart of a process 200 for a diagnostic data analysis system according to some embodiments of the present disclosure is shown. Process 200 can be performed by...Figure 1 The diagnostic system 140 is used to implement this. For ease of discussion, it will be combined with... Figure 1 To describe process 200.

[0032] like Figure 2 As shown in box 202, the diagnostic system 140 acquires information 130 about the analysis engine 125 in the data analysis system 110 to be diagnosed and diagnostic data 135 associated with the analysis engine 125, wherein the diagnostic data 135 includes at least one of training data 115 and prediction data 120 used for the analysis engine 125.

[0033] In some embodiments, the data analysis system 110 may be deployed on a cloud platform, where information 130 about the analysis engine 125, as well as training data 115 and prediction data 120, are also stored accordingly on the cloud platform. Users may access the cloud platform to perform corresponding data analysis tasks using the data analysis system 110.

[0034] In some embodiments, the diagnostic system 140 may be deployed on the same cloud platform as the data analysis system 110, so that the diagnostic system 140 can access the cloud platform's storage to obtain information about the analysis engine 125 and the data 135 to be diagnosed.

[0035] In some embodiments, the diagnostic system 140 may also be deployed to communicatively couple with the data analysis system 110, so that information about the analysis engine 125 and data 135 to be diagnosed can be received from the data analysis system 110 via a wired or wireless network.

[0036] In block 204, the diagnostic system 140 determines a diagnostic engine 145 corresponding to the analysis engine 125 based on the received information. In some embodiments, the diagnostic engine 145 may be constructed, for example, to include a pre-configured Bayesian neural network module.

[0037] The following will be referenced Figure 3 To describe the process of box 204. Figure 3 A flowchart illustrating a process 300 for determining a diagnostic engine 145 according to an embodiment of this disclosure is shown. Figure 3 As shown, in block 302, the diagnostic system 140 can determine whether the analysis engine 125 includes a neural network module based on the received information 130. In response to determining in block 302 that the analysis engine 125 includes a neural network module, process 300 proceeds to block 304, where the diagnostic system 140 can construct the neural network module based on the received information. For example, information 130 can indicate the number of layers, nodes, etc., in the neural network module, and the diagnostic system 140 can construct an untrained neural network module based on information 130.

[0038] At block 306, the diagnostic system 140 can update the neural network module with the probabilistic inference to obtain a probabilistic neural network module. Unlike a conventional deterministic neural network that represents certainty of a numerical value with a parameter, the probabilistic neural network represents the parameter in the network with a distribution, such that the probabilistic neural network module is able to better determine a measure of uncertainty for a prediction result. In some embodiments, the diagnostic system 140 can update the neural network module with Bayesian inference, such that all the probabilistic data network modules comprise a Bayesian neural network. Alternatively, the diagnostic system 140 can also update the neural network module with frequentist inference. At block 308, the diagnostic system 140 constructs the diagnostic engine 145 to include the obtained probabilistic neural network module.

[0039] Alternatively, if it is determined at block 302 that the analysis engine 125 does not include a neural network module, the process 300 proceeds to block 310, where the diagnostic system 140 constructs the diagnostic engine 145 to include a preconfigured probabilistic neural network module.

[0040] With continued reference to Figure 2 At block 206, the diagnostic system 140 provides an uncertainty measure 150 associated with at least one of the analysis engine 125 and the data to be diagnosed 135 by processing the data to be diagnosed 135 with the diagnostic engine 145, where the uncertainty measure 150 indicates a degree of certainty of a prediction result obtained by the analysis engine 125 for processing the data to be diagnosed 135.

[0041] In some embodiments, the data to be diagnosed 135 includes both the training data 115 and the prediction data 120. The uncertainty measure 150 determined by the diagnostic system 140 can include at least one of a model uncertainty measure, a data uncertainty measure, and a distribution uncertainty measure.

[0042] The model uncertainty measure indicates at least an influence of a degree of fit of a complexity of the analysis engine 125 and a size of the data to be diagnosed 135 on a performance of the analysis engine 125. For example, the model uncertainty measure can indicate whether a size of the training data 115 fits with a complexity of the analysis engine 125. For example, when the training data 115 is large in size, but the model included in the analysis engine 125 is simple, the degree of fit is poor.

[0043] The data uncertainty measure indicates at least an influence of a quality of the data to be diagnosed 135 on a performance of the analysis engine. For example, the quality of the data to be diagnosed 135 can indicate at least one aspect of whether the data to be diagnosed 135 is easy to predict, whether it includes more noise, whether there are more data missing, and the like.

[0044] The distribution uncertainty measure indicates at least an influence of how well the distribution of the prediction data 120 fits the distribution of the training data 115 on the performance of the analysis engine. For example, since the analysis engine 125 is trained with the training data 115, when the prediction data 120 has a large difference from the distribution of the training data 115

[0045] The following will introduce the process of determining the model uncertainty measure, the data uncertainty measure, and the distribution uncertainty measure by taking a Bayesian neural network as an example. For the Bayesian neural network module included in the diagnosis engine 145, for example, the following can be used to identify the Bayesian neural network, where θ represents the parameters of the network, x represents the input to the network, and represents the output of the network.

[0046] In addition, in order to separate the three uncertainty measures, the likelihood function of the output y can be decomposed according to the following formula:

[0047]

[0048] where x* represents a data item in the prediction data 120, D represents the training data 115, and the third term p(θ|D) in formula (1) represents the posterior distribution of the model parameters. In addition, φ represents whether x* fits the distribution of D, where the second term p(φ|x*, θ) in formula (1) assumes a Bernoulli distribution, φ = 1 represents that x* fits the distribution of D, and φ = 1 represents that x* does not fit the distribution of D. ε represents the data uncertainty, where the fourth term p(ε|D) in formula (1) represents the posterior distribution of ε. The first term p(y|φ, ε) in formula (1) represents the distribution of the output y.

[0049] In some embodiments, for the Bayesian neural network module included in the diagnosis engine 145, in order to enable the Bayesian neural network to distinguish whether the data is located within a specific distribution, the Bayesian neural network can be trained to enable:

[0050] (1) enable the Bayesian neural network to perform data prediction as well as the analysis engine 125. In some embodiments, this training target can be achieved by introducing a regular objective function of the Bayesian neural network;

[0051] (2) enable the Bayesian neural network to also distinguish whether the input data fits the distribution of the training data D. In some embodiments, this training target can be achieved by introducing random noise to the training data that fits the distribution to construct negative samples, and training the Bayesian neural network through the combination of positive samples and negative samples, so that the Bayesian neural network can distinguish whether certain input data fits a specific distribution.

[0052] ​In some embodiments, the uncertainty measure can be represented by one of the following: mutual information (MI), entropy, or variance. Compared to representing individual uncertainty by absolute value, mutual information can better reflect the correlation that can exist among the model uncertainty measure, the data uncertainty measure, and the distribution uncertainty measure.

[0053] In some embodiments, the diagnostic system 140 can determine a total uncertainty measure for the diagnostic engine 145 by processing the data to be diagnosed 135 using the diagnostic engine 145; and provide at least one of the following based on the total uncertainty measure: a model uncertainty measure, a data uncertainty measure, and a distribution uncertainty measure. Specifically, the model uncertainty measure can be determined by the following formula:

[0054]

[0055] The data uncertainty measure can be determined by the following formula:

[0056]

[0057] The distribution uncertainty measure can be determined by the following formula:

[0058]

[0059] In the formulas (2) to (4), where -∫logp(Y|D)dp(Y|D) represents the total uncertainty measure of the Bayesian neural network, the in the formula (2) represents the uncertainty measure left after knowing the network parameters θ, the in the formula (3) represents the uncertainty measure left after having the data uncertainty information, and the in the formula (4) represents the uncertainty measure left after knowing the data distribution information.

[0060] For each of the formulas (2) to (4), after the Bayesian neural network is trained using the training data 115, the q(θ|D) and p(ε|D) can be determined, and further based on the decomposition of the likelihood function in the formula (1), p(y|D, θ), and p(y|D, ε) can be further determined, and then the values of the model uncertainty measure, the data uncertainty measure, and the distribution uncertainty measure can be determined according to the formulas (2) to (4), respectively.

[0061] In some embodiments, the diagnostic system 140 can send the determined uncertainty metrics 150 to the data analysis system 110 to enable the data analysis system 110 to adjust at least one of the analysis engine 125 and the data to be diagnosed 135 based on the uncertainty metrics 150.

[0062] In some embodiments, the diagnostic system 140 can also provide an indication on adjusting at least one of the analysis engine 125 and the data to be diagnosed 135. As discussed above, different uncertainty metrics respectively indicate the influence of different factors on the model prediction performance. For example, when the model uncertainty metric is high, the diagnostic system 140 can provide an indication on increasing the training data. When the data uncertainty metric is high, the diagnostic system 140 can provide an indication, for example, on discarding the data to be diagnosed. When the distribution uncertainty metric is high, the diagnostic system 140 can provide an indication on retraining the analysis engine 125 with the prediction data 120. In some embodiments, the diagnostic system 140 can compare the determined uncertainty metrics with predetermined thresholds, and provide the above-mentioned indications when the uncertainty metrics are greater than the predetermined thresholds.

[0063] In some embodiments, the diagnostic system 140 can also select the uncertainty metric with the maximum value from the comparison of the model uncertainty metric, the data uncertainty metric and the distribution uncertainty metric, and provide an indication corresponding to the type of uncertainty metric. For example, when the data uncertainty metric is the maximum, the diagnostic system 140 can provide an indication on increasing the training data.

[0064] In some embodiments, the diagnostic system 140 can also provide an indication on adjusting at least one of the analysis engine 125 and the data to be diagnosed 135 by comparing based on the training data 115 and based on the prediction data 120. Specifically, Figure 4 A flowchart of providing an adjustment indication process 400 according to an embodiment of the present disclosure is shown.

[0065] As Figure 4 shown, at block 402, the diagnostic system 140 can obtain a first uncertainty metric by processing the training data 115 with the diagnostic engine 145. For example, the training data 115 can be, for example, data of the first quarter, the diagnostic system 140 can determine that the model uncertainty metric related to the training data of the first quarter is 0.6, the data uncertainty metric is 1.2, and the distribution uncertainty metric is 0.3 based on the training data 115.

[0066] At block 404, the diagnostic system 140 can obtain a second uncertainty measure by processing the prediction data 120 with the diagnostic engine 145. For example, the prediction data 120 can be data for a second quarter to be predicted, and the diagnostic system 140 can determine that the model uncertainty measure is 0.5, the data uncertainty measure is 1.0, and the distribution uncertainty measure is 0.6 based on the prediction data 120 related to the second quarter.

[0067] At block 406, the diagnostic system 140 can provide an indication to adjust at least one of the analysis engine 125 and the data 135 to be diagnosed based on comparing the first uncertainty measure and the second uncertainty measure. For example, based on comparing the first uncertainty measure related to the training data of the first quarter and the second uncertainty measure related to the prediction data of the second quarter, it can be found that the change of the distribution uncertainty measure is the largest (0.3), at which time the diagnostic system 140 can provide an indication to retrain the analysis engine 125 with the prediction data 120, for example.

[0068] In some embodiments, the input of the diagnostic engine 145 includes a plurality of data items respectively associated with different features. For example, in a scenario for predicting household electricity consumption, the data items for training and / or the data items for prediction can include a plurality of data items associated with different features, such as temperature of different rooms (e.g., bedroom, living room, kitchen, etc.), humidity of different rooms (e.g., bedroom, living room, kitchen, etc.), outdoor temperature, and outdoor humidity, etc. These different data items can be collected by sensors placed at different locations, for example.

[0069] In some embodiments, the diagnostic system 140 can also determine a contribution of at least one of the plurality of data items to the uncertainty measure based on the uncertainty measure. For example, for a particular uncertainty measure u, the contribution of a particular data item to the uncertainty measure (also referred to as feature uncertainty measure below) can be determined by taking derivative as:

[0070]

[0071] where u represents the uncertainty measure, represents a data item in the ith input associated with the jth feature. Considering that the contribution of a particular data item to the uncertainty measure can be different based on each input, the contribution of at least one of the plurality of data items to the uncertainty measure can be determined by taking average as:

[0072]

[0073] where N represents the number of inputs. For example, for the data uncertainty measure, N can represent the number of training samples.

[0074] For example, Figure 5 A diagram 500 showing the contribution of data items associated with different features to the distribution uncertainty measure is shown according to an embodiment of the disclosure. As Figure 5 shown, in this example, the input of the diagnostic engine 145 includes data items associated with 23 features, each of which can have different contribution 502 to the distribution uncertainty measure. For example, in this example, the data item corresponding to the 5th feature has the largest contribution.

[0075] In some embodiments, the diagnostic system 140 can also provide an indication to adjust at least one of the plurality of data items based on the determined contribution of the at least one data item to the uncertainty measure. For example, in the example of Figure 5 the diagnostic system 140 can provide an indication to adjust the data item corresponding to the 5th feature based on the largest contribution of the data item corresponding to the 5th feature, e.g., to cause the data analysis system 110 to remove the data item, replace the data item, or adjust the data item.

[0076] Based on the above-described method of diagnosing the data analysis system, embodiments of the disclosure are able to utilize a diagnostic engine corresponding to the analysis engine to determine the impact of the model or the data to be diagnosed (including training data and prediction data) on the accuracy of the prediction by the analysis engine of the data analysis system, thereby providing support for users to select a suitable data analysis model in the data analysis system or to determine possible defects of the data to be diagnosed. In addition, by utilizing a separate diagnostic engine to diagnose the data analysis system, the developer can be provided with diagnostic suggestions for the data analysis system in the case of affecting the normal operation of the data analysis system.

[0077] Optimization of a data analytics system

[0078] According to another aspect of the disclosure, embodiments of the disclosure also provide a method of optimizing a data analysis system. Figure 6 A flowchart of a process 600 for optimizing a data analysis system is shown according to some embodiments of the disclosure. The process 600 can be implemented by Figure 1 the data analysis system 110. For ease of discussion, the process 600 will be described in conjunction with Figure 1 the data analysis system 110.

[0079] At block 602, the data analytics system 110 provides the diagnostic system 140 with the information 130 about the analysis engine 125 in the data analytics system 110 and the to-be-diagnosed data 135 associated with the analysis engine 125, the to-be-diagnosed data 135 including at least one of the training data 115 and the prediction data 120 for the analysis engine 125. As described above with reference to block 202 in the process 200, in some examples, the data analytics system 110 can provide the information 130 and the to-be-diagnosed data 135 to the diagnostic system 140 through a wired or wireless network. Alternatively, the data analytics system 110 and the diagnostic system 140 can be configured in the same cloud platform, for example, and the diagnostic system 140 can obtain the information 130 and the to-be-diagnosed data 135 from the cloud platform.

[0080] At block 604, the data analytics system 110 receives, from the diagnostic system 140, an uncertainty measure 150 associated with at least one of the analysis engine 125 and the to-be-diagnosed data 135, where the uncertainty measure 150 indicates a degree of certainty of a prediction result obtained by the analysis engine 125 for processing the to-be-diagnosed data 135. The process of determining the uncertainty measure 150 is described above with reference to the process 300, and the data analytics system 110 can receive the uncertainty measure 150 from the diagnostic system 140. Figures 2-5 The process of determining the uncertainty measure 150 is described above with reference to the process 300, and the data analytics system 110 can receive the uncertainty measure 150 from the diagnostic system 140.

[0081] At block 606, the data analytics system 110 updates at least one of the analysis engine 125 and the to-be-diagnosed data 135 based on the uncertainty measure 150.

[0082] In some embodiments, the method 600 can further include block 608, where the data analytics system 110 can perform data analytics with the updated analysis engine 125. Specifically, the data analytics system 110 can receive new to-be-analyzed data, process the new to-be-analyzed data with the updated analysis engine 125 to provide a prediction for the new to-be-analyzed data.

[0083] In some embodiments, the received uncertainty measure 150 includes a model uncertainty measure, where the data uncertainty measure indicates at least an influence of a degree of fit of a complexity of the analysis engine and a size of the to-be-diagnosed data on a performance of the analysis engine. In some embodiments, the data analytics system 110 can compare the model uncertainty measure with a predetermined threshold. Upon determining that the model uncertainty measure is greater than the predetermined threshold, the data analytics system 110 updates the analysis engine 125 by training the analysis engine 125 with additional training data. In this way, the influence of the model uncertainty on the performance of the analysis engine 125 can be reduced.

[0084] In some embodiments, the received uncertainty measure includes a data uncertainty measure, where the data uncertainty measure indicates at least an impact of a quality of the to-be-diagnosed data 135 on the performance of the analysis engine 125. In some embodiments, the data analysis system 110 can compare the data uncertainty measure with a predetermined threshold. Upon determining that the data uncertainty measure is greater than the predetermined threshold, the data analysis system 110 may, for example, perform a data discard treatment on the to-be-diagnosed data 135 to ensure the quality of the training data.

[0085] In some embodiments, the received uncertainty measure includes a distribution uncertainty measure, where the distribution uncertainty measure indicates at least an impact of a degree of fit of the prediction data with a distribution of the training data on the performance of the analysis engine. In some embodiments, the data analysis system 110 can compare the distribution uncertainty measure with a predetermined threshold. Upon determining that the distribution uncertainty measure is greater than the predetermined threshold, the data analysis system 110 may, for example, retrain the analysis engine 125 with the prediction data 120 so that the analysis engine 125 is capable of accurately analyzing the prediction data 120.

[0086] In some embodiments, as discussed above, the data analysis system 110 can also determine a feature uncertainty measure from the received uncertainty measure, where the uncertainty measure indicates a contribution of at least one data item of a plurality of data items to the uncertainty measure, where the input of the analysis engine includes the plurality of data items respectively associated with different features. Determining the feature uncertainty measure can refer to the process described above with respect to equations (5) and (6), which is not repeated here. In some embodiments, the data analysis system 110 can compare the received feature uncertainty measure with a predetermined threshold, and when the feature uncertainty measure is greater than the predetermined threshold, the data analysis system 110 may, for example, perform at least one of the following: adding more data items to the input of the analysis engine; changing a property (e.g., granularity) of the data items corresponding to the feature; or discarding at least one data item. For example, in the example of FIG. 1, the data analysis system 110 can receive a feature uncertainty measure corresponding to the fifth feature from the diagnosis system 140, and upon determining that it is greater than a threshold, may, for example, discard or replace the data item corresponding to the fifth feature from the prediction data 120, thereby reducing the impact of the feature on the performance of the analysis engine 125. Figure 5

[0087] Based on the above-described methods of optimizing a data analysis system, embodiments of the present disclosure are capable of updating the analysis engine based on the uncertainty measure determined by the analysis engine, thereby improving the performance of the data analysis system.

[0088] Example Applications

[0089] ​Application 1: Credit card fraud analysis system

[0090] The method of diagnosing and optimizing a data analysis system discussed above can be applied in a data analysis system for predicting credit card fraud. For example, the credit card fraud analysis system can include an analysis engine for receiving input of data items associated with 25 features and providing a prediction on whether it is fraudulent. In one specific example, the analysis engine includes a neural network composed of 3 layers of MLP, and the number of hidden units in each layer is 45, 135, and 405, respectively.

[0091] By converting the neural network into a Bayesian neural network based on Bayesian inference, the analysis engine can determine the model uncertainty corresponding to different scales of training data. For example, Figure 7A A trend chart 700A showing the model uncertainty measure as a function of the scale of training data is shown, from which it can be seen that the model uncertainty measure is generally decreasing as the scale of training data increases, and thus the trend can assist the developer to determine whether it is necessary to increase the scale of training data or to reduce the complexity of the neural network.

[0092] In addition, Figure 7B A diagram 700B showing the contribution of different features to the distribution uncertainty measure (feature uncertainty measure) is also shown, from which it can be seen that the feature uncertainty measure associated with the third feature is the largest, and thus the data can indicate to the developer to check whether the collection of data items corresponding to the third feature has failed, and whether it is necessary to replace or modify the data items corresponding to the third feature, etc.

[0093] Based on the method of diagnosing and optimizing a data analysis system of the present disclosure, the current performance bottleneck of the credit card fraud analysis system can be determined, and the performance of the data analysis system can be improved accordingly.

[0094] Application 2: Channel subscription analysis system

[0095] The method of diagnosing and optimizing a data analysis system discussed above can be applied in a data analysis system for predicting channel subscription. For example, the channel subscription analysis system can include an analysis engine for receiving input of data items associated with 23 features and providing a prediction on whether a user will subscribe to the channel. In one specific example, the analysis engine includes a neural network composed of 2 layers of MLP, and the number of hidden units in each layer is 12 and 12, respectively.

[0096] By converting the neural network into a Bayesian neural network based on Bayesian inference, the analysis engine can determine the model uncertainty corresponding to different scales of training data. For example, Figure 7CA trend graph 700C showing how the model uncertainty measure varies with the size of the training data is shown. As can be seen from the graph, the model uncertainty measure is generally decreasing as the size of the training data increases, and thus this trend can assist the developer in determining whether the size of the training data needs to be increased, or whether the complexity of the neural network needs to be reduced.

[0097] In addition, Figure 7D A plot 700B showing the contribution of different features to the distribution uncertainty measure (feature uncertainty measure) is also shown. As can be seen from the plot 700B, the feature uncertainty measure associated with the fifth feature is the largest, and thus this data can indicate to the developer to check whether the collection of the data items corresponding to the fifth feature has failed, and whether the data items corresponding to the fifth feature need to be replaced or modified, etc.

[0098] The method of diagnosing and optimizing a data analysis system based on the present disclosure is able to determine the current performance bottleneck of the channel subscription analysis system, and accordingly improve the performance of the data analysis system.

[0099] Application three: shopping expenditure analysis system

[0100] The method of diagnosing and optimizing a data analysis system discussed above can be applied in a data analysis system for predicting shopping expenditure. For example, the channel subscription analysis system can include an analysis engine for receiving input of data items associated with 9 features, and providing a prediction on how much a user will spend, for example, during a shopping festival. In one specific example, the analysis engine includes a neural network composed of 2 layers of MLP, and the number of hidden units in each layer is 128 and 256 respectively.

[0101] By converting the neural network into a Bayesian neural network based on Bayesian inference, the analysis engine can determine the model uncertainty corresponding to different sizes of training data. For example,

[0102] Figure 7E A trend graph 700E showing how the model uncertainty measure varies with the size of the training data is shown. As can be seen from the graph, the model uncertainty measure is generally decreasing as the size of the training data increases, and thus this trend can assist the developer in determining whether the size of the training data needs to be increased, or whether the complexity of the neural network needs to be reduced.

[0103] In addition, Figure 7FA diagram 700F showing the contribution of different features to the distribution uncertainty measure (feature uncertainty measure) is also shown. As can be seen from the diagram 700F, the feature uncertainty measure associated with the first feature is the largest, and thus this data can indicate that the developer check whether the collection of the data item corresponding to the first feature has failed, and whether the data item corresponding to the first feature needs to be replaced or modified, etc.

[0104] The method of diagnosing and optimizing a data analysis system based on the present disclosure is able to determine the current performance bottleneck of the shopping expenditure analysis system, and accordingly improve the performance of the data analysis system.

[0105] It should be appreciated that the specific data analysis system listed above is only illustrative and is not intended to limit the present disclosure. The scheme of the diagnostic and optimization data analysis system involved in the present disclosure can be applied to any suitable data analysis system.

[0106] Example Device

[0107] Figure 8 A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. For example, the data analysis system 110, the diagnostic analysis system 140, the analysis engine 125, and / or the diagnostic engine 145 shown can be implemented by the device 800. As shown, the device 800 includes a central processing unit (CPU) 801 that can perform various appropriate actions and processes in accordance with computer program instructions stored in a read-only memory (ROM) 802 or loaded into a random access memory (RAM) 803 from a storage unit 808. Various programs and data required for operation of the device 800 can also be stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804. Figure 1

[0108] Various components in the device 800 are connected to the I / O interface 805, including an input unit 806 such as a keyboard, a mouse, etc., an output unit 807 such as various types of displays, speakers, etc., a storage unit 808 such as a magnetic disk, an optical disk, etc., and a communication unit 809 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0109] ​The various processes and processes described above, such as process 200 and / or process 600, can be performed by processing unit 801. For example, in some embodiments, process 200 and / or process 600 can be implemented as a computer software program tangibly embodied in a machine readable medium, such as storage unit 808. In some embodiments, portions of the computer program or all of the computer program can be loaded onto device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded onto RAM 803 and executed by CPU 801, one or more acts of process 200 and / or process 600 described above can be performed.

[0110] The present disclosure can be a method, apparatus, system, and / or computer program product. Computer program products can include computer readable storage media having computer readable program instructions embodied therewith.

[0111] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0112] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0113] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0114] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0115] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0116] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0117] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0118] The foregoing description of the exemplary embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the various embodiments to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the various embodiments be limited not with this detailed description, but rather by the claims appended hereto.

Claims

1. A method for a diagnostic data analysis system, the diagnostic data analysis system being configured to perform image processing, the method comprising: Obtain information about the analysis engine in the data analysis system to be diagnosed and the diagnostic data associated with the analysis engine, the diagnostic data including at least one of training data and prediction data used for the analysis engine; Based on the received information, determine the diagnostic engine corresponding to the analysis engine, including: i) In response to the information indicating that the analysis engine includes a neural network module, the neural network module is constructed based on the information. The neural network module is updated using probabilistic inference to obtain a probabilistic neural network module, and The diagnostic engine is constructed to include the obtained probabilistic neural network module; or ii) In response to the information indicating that the analysis engine does not include a neural network module, the diagnostic engine is constructed to include a pre-configured probabilistic neural network module; as well as By utilizing the diagnostic engine to process the data to be diagnosed, an uncertainty measure associated with at least one of the analysis engine and the data to be diagnosed is provided, the uncertainty measure indicating the degree of certainty of the prediction results obtained by the analysis engine for processing the data to be diagnosed.

2. The method of claim 1, wherein the data to be diagnosed includes both the training data and the prediction data, wherein providing the uncertainty measure includes: By processing the data to be diagnosed using the diagnostic engine, a total uncertainty metric for the diagnostic engine is determined; as well as Based on the total uncertainty measure, at least one of the following is provided: a model uncertainty measure, a data uncertainty measure, and a distribution uncertainty measure. The model uncertainty metric at least indicates the impact of the complexity of the analysis engine on the performance of the analysis engine and how well it fits the size of the data to be diagnosed. The data uncertainty metric at least indicates the impact of the quality of the data to be diagnosed on the performance of the analysis engine; and The distribution uncertainty metric at least indicates the impact of the degree of fit between the distribution of the predicted data and the distribution of the training data on the performance of the analysis engine.

3. The method of claim 1, wherein the input to the diagnostic engine comprises a plurality of data items associated with different features, wherein providing the uncertainty measure comprises: Based on the uncertainty measure, determine the contribution of at least one of the plurality of data items to the uncertainty measure.

4. The method according to claim 3, further comprising: Based on the contribution, instructions are provided regarding the adjustment of at least one of the plurality of data items.

5. The method according to claim 1, further comprising: Provide instructions on adjusting at least one of the analysis engine and the data to be diagnosed.

6. The method of claim 5, wherein providing instructions regarding adjusting the analysis engine and the at least one of the data to be diagnosed comprises: The indication is provided in response to the uncertainty metric being greater than a predetermined threshold.

7. The method of claim 5, wherein the data to be diagnosed comprises both the training data and the prediction data, wherein providing instructions regarding adjusting the analysis engine and the at least one of the data to be diagnosed comprises: A first uncertainty measure is obtained by processing the training data using the diagnostic engine. A second uncertainty measure is obtained by processing the predicted data using the diagnostic engine. as well as Based on a comparison of the first uncertainty measure and the second uncertainty measure, instructions are provided regarding adjusting the analysis engine and at least one of the data to be diagnosed.

8. A method for optimizing a data analysis system, the data analysis system being configured to perform image processing, the method comprising: Provide the diagnostic system with information about the analysis engine in the data analysis system and diagnostic data associated with the analysis engine, the diagnostic data including both training data and prediction data used for the analysis engine; The diagnostic system receives an uncertainty measure associated with at least one of the analysis engine and the data to be diagnosed, the uncertainty measure indicating the degree of certainty of the prediction obtained by the analysis engine for processing the data to be diagnosed, wherein the uncertainty measure associated with at least one of the analysis engine and the data to be diagnosed is provided by the diagnostic system through the following operations: A first uncertainty measure is obtained by processing the training data using a diagnostic engine. A second uncertainty measure is obtained by processing the predicted data using the diagnostic engine. as well as Based on a comparison of the first uncertainty measure and the second uncertainty measure, instructions are provided regarding adjusting the analysis engine and at least one of the data to be diagnosed; as well as Update at least one of the analysis engine and the data to be diagnosed based on the uncertainty measure.

9. The method of claim 8, wherein the uncertainty measure includes a model uncertainty measure, the model uncertainty measure indicating at least the impact of the complexity of the analysis engine on the performance of the analysis engine on the fit of the size of the data to be diagnosed, wherein updating at least one of the analysis engine and the data to be diagnosed based on the uncertainty measure comprises: In response to the model uncertainty metric exceeding a predetermined threshold, the analysis engine is updated by training the analysis engine using additional training data.

10. The method of claim 8, wherein the uncertainty measure includes a data uncertainty measure, the data uncertainty measure indicating at least the impact of the quality of the data to be diagnosed on the performance of the analysis engine, wherein updating at least one of the analysis engine and the data to be diagnosed based on the uncertainty measure comprises: In response to the data uncertainty metric exceeding a predetermined threshold, the data to be diagnosed is discarded.

11. The method of claim 8, wherein the uncertainty measure includes a distribution uncertainty measure, the distribution uncertainty measure indicating at least the impact of the degree of fit between the distribution of the predicted data and the training data on the performance of the analysis engine, wherein updating at least one of the analysis engine and the data to be diagnosed based on the uncertainty measure comprises: In response to the distribution uncertainty metric exceeding a predetermined threshold, the analysis engine is retrained using the predicted data.

12. The method of claim 8, wherein the input to the analysis engine comprises a plurality of data items respectively associated with different features, the method further comprising: A characteristic uncertainty measure is determined from the received uncertainty measure, the characteristic uncertainty measure indicating the contribution of at least one of the plurality of data items to the uncertainty measure.

13. The method of claim 12, wherein updating at least one of the analysis engine and the data to be diagnosed based on the uncertainty measure comprises: In response to the feature uncertainty measure being greater than a predetermined threshold, perform at least one of the following: Add more data items from the input of the analysis engine; Change the attributes of the data items corresponding to the feature; or Discard at least one of the data items.

14. An apparatus for a diagnostic data analysis system, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the method according to any one of claims 1-7.

15. An apparatus for optimizing a data analysis system, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the method according to any one of claims 8-13.

16. A computer-readable storage medium having computer-readable program instructions stored thereon, the computer-readable program instructions being used to perform the method according to any one of claims 1-7.

17. A computer-readable storage medium having computer-readable program instructions stored thereon, the computer-readable program instructions being used to perform the method according to any one of claims 8-13.

Images