Model evaluation method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides a model evaluation method, device, electronic device and storage medium, which relates to the field of artificial intelligence technology and can be used in scenarios such as language processing, image processing, and sentiment classification. The method includes: obtaining training data and test data; training the model using the training data; testing the trained model using the test data, and calculating the preset index value obtained by evaluating the model performance during the test; and determining to use at least one analysis module in the evaluation system to evaluate at least one of the training data, test data and model based on the preset index value. The implementation of the solution of the present application can effectively improve the effectiveness of algorithm evaluation in optimizing and improving the model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology. Specifically, the present application relates to a model evaluation method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, more and more products are integrating intelligent algorithms. The quality of an algorithm will affect the execution efficiency of the algorithm and even the program. Therefore, it is very important to integrate the appropriate algorithm into the product, otherwise the original performance of the product may be affected by the integration of the algorithm. Therefore, it is necessary to evaluate the quality of the algorithm.

[0003] In the prior art, a language interpretability tool (Language Interpretability Tool, LIT) is provided to address the problem of algorithm evaluation. The tool can be used to evaluate natural language processing (Natural Language Processing, NLP) algorithm models.

[0004] However, the LIT tool is only aimed at algorithm models and various natural language processing tasks in the field of natural language processing. It cannot be adapted to a wider range of algorithm models in other fields, resulting in poor adaptability. In addition, when performing white-box evaluation in the LIT tool, it is generally performed on the feature layer within the model, lacking evaluation and analysis of data samples and other levels. Its analysis results are relatively simple and cannot understand the quality issues of the algorithm model from multiple levels. Summary of the invention

[0005] The technical solution provided by this application is intended to solve at least one of the above technical defects, especially the technical defect of single analysis result of evaluation. Among them, the technical solution is as follows:

[0006] In a first aspect of the present application, a model evaluation method is provided, including: obtaining training data and test data; training a model using the training data; testing the trained model using the test data, and calculating a preset index value obtained by evaluating the model performance during the test; and determining, based on the preset index value, to use at least one analysis module in the evaluation system to evaluate at least one of the training data, test data, and model.

[0007] In one embodiment, determining to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data, and the model based on the preset indicator value includes:

[0008] If the preset indicator value is less than or equal to the preset threshold, at least one of the following is used for evaluation: using the coverage analysis module in the evaluation system for measuring the integrity of the test data to evaluate the model; using the uncertainty analysis module in the evaluation system for measuring the uncertainty of the test data to evaluate the test data;

[0009] Abnormal problems are evaluated by using at least one of the following: an attribution analysis module for analyzing the importance of different features of test data to model decisions, a visualization analysis module for analyzing the importance of different features of test data to model decisions, and an influence analysis module for analyzing the importance of training data to the model's decision-making based on test data.

[0010] In one embodiment, using the coverage analysis module to evaluate the model includes at least one of the following:

[0011] Determine the boundary range of each neuron in the model based on the training data, and evaluate the model based on the boundary range and test data;

[0012] Construct neuron pairs between model network layers and evaluate the model based on neuron pairs and test data.

[0013] In one embodiment, the boundary range of each neuron in the model is determined based on the training data, and the model is evaluated based on the boundary range and the test data, including:

[0014] Perform forward propagation processing based on the training data to determine the boundary range of each neuron in the model;

[0015] Perform forward propagation processing based on the test data to determine the number of neurons corresponding to each test sample in the test data that exceeds the boundary range;

[0016] Determine the confidence level of the model's decision on each test sample based on the number of neurons;

[0017] The results of the evaluation of the model are determined based on the confidence level.

[0018] In one embodiment, constructing a neuron pair and evaluating the model based on the neuron pair and test data includes:

[0019] Extract neurons from different layers of neurons in the model to construct multiple neuron pairs;

[0020] Determine the ratio of the number of neuron pairs in all activation states among the neuron pairs triggered by the test data to the number of all neuron pairs;

[0021] The results of the evaluation of the model are determined based on the ratio.

[0022] In one embodiment, determining the result of evaluating the model based on the ratio further includes:

[0023] Construct attack sample data based on the state information of neurons activated by untested data;

[0024] Based on the attack sample data, determine the boundary vulnerability information of the model.

[0025] In one embodiment, an uncertainty analysis module is used to evaluate test data, including:

[0026] Perform forward propagation processing based on the test data to determine the prediction results of the test data;

[0027] Determining an uncertainty value of the test data based on at least one of a variance, an entropy, and a mutual information of the prediction result;

[0028] The results of the evaluation of the test data are determined based on the uncertainty value.

[0029] In one embodiment, an attribution analysis module is used to evaluate abnormal problems, including:

[0030] The gradient integral algorithm is used to determine the distribution information of the contribution degree corresponding to each test sample in the test data in the sample characteristics;

[0031] The results of the evaluation for the abnormal problem are determined based on the distribution information.

[0032] In one embodiment, using the visual analysis module to evaluate abnormal problems includes at least one of the following:

[0033] A convolutional heat map algorithm is used to determine the heat map corresponding to each test sample image in the test data and the annotation information corresponding to the test sample image; the result of the evaluation on the abnormal problem is determined based on the heat map; the focus position in the heat map represents the focus area of ​​the classification task under the category corresponding to the annotation information;

[0034] In the process of model testing, a dimensionality reduction algorithm is used to monitor the distribution change information of the test data; based on the distribution change information of the test data, the results of the evaluation on abnormal problems are determined.

[0035] In one embodiment, using the influence analysis module to evaluate abnormal issues includes:

[0036] Determine the importance score of each training sample in the training data relative to the test data based on the training data and the test data;

[0037] The results of the evaluation of the abnormal issues are determined based on the importance scores.

[0038] In one embodiment, training the model using training data further includes:

[0039] Use dimensionality reduction algorithms to monitor distribution changes in training data during model training;

[0040] The results of evaluating the model are determined based on the distribution change information of the training data.

[0041] In a second aspect of the present application, a model evaluation device is provided, including: an acquisition module for acquiring training data and test data; a training module for training a model using the training data; a testing module for testing the trained model using the test data and calculating a preset indicator value on the test data; an evaluation module for determining, based on the preset indicator value, to use at least one analysis module in the evaluation system to evaluate at least one of the training data, test data and model.

[0042] In a third aspect of the present application, an electronic device is provided, the electronic device comprising:

[0043] one or more processors;

[0044] Memory;

[0045] One or more computer programs, wherein the one or more computer programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: execute the method provided in the first aspect.

[0046] In a fourth aspect of the present application, a computer-readable storage medium is provided, where the computer storage medium is used to store computer instructions. When the computer instructions are executed on a computer, the computer can execute the method provided in the first aspect.

[0047] The beneficial effects of the technical solution provided by this application are:

[0048] In the present application, an evaluation system is provided, and the evaluation system is used to evaluate the model based on the entire process of model training and testing; during implementation, training data and test data are obtained, the model is trained with the training data, and the trained model is tested with the test data, and the preset index value obtained by evaluating the model performance during the test is calculated, and then based on the preset index value, at least one analysis module in the evaluation system is determined to evaluate at least one of the training data, test data and model. The implementation of the present application can be adapted to various models, without limiting the field of the model, and improves the adaptability of model evaluation in various fields; on the other hand, the present application can also realize evaluation of various dimensions of training data, test data and models, provide multi-dimensional analysis results, and improve the efficiency of model optimization and improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0050] Figure 1 A flowchart of model evaluation in related technologies;

[0051] Figure 2 A flow chart of a model evaluation method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of a model evaluation method provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a model evaluation method provided in an embodiment of the present application using a gradient integral algorithm for evaluation;

[0054] Figure 5 A schematic diagram of a visualization result of a model evaluation method using a gradient integration algorithm in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of a model evaluation method using a convolutional heat map algorithm for evaluation in an embodiment of the present application;

[0056] Figure 7a A schematic diagram of a visualization result of a model evaluation method using a convolutional heat map algorithm in an embodiment of the present application;

[0057] Figure 7b A schematic diagram of a visualization result of a model evaluation method using a convolutional heat map algorithm in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of using an influence analysis module for evaluation in a model evaluation method in an embodiment of the present application;

[0059] Figure 9a A schematic diagram of the first period of model training in a model evaluation method in an embodiment of the present application;

[0060] Figure 9b A schematic diagram of the second period of model training in a model evaluation method in an embodiment of the present application;

[0061] Fig.10 A schematic diagram of the structure of a model evaluation device provided in an embodiment of the present application;

[0062] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0064] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0066] The following is an explanation of the technologies and terms involved in this application:

[0067] AI (Artificial Intelligence) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0068] In this application, natural language processing, machine learning / deep learning and other directions are mainly involved. Among them, NLP (Nature Language processing) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes technologies such as text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc. ML (Machine Learning) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Specializes in how computers simulate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0069] In related technologies, in order to effectively evaluate the quality of artificial intelligence algorithm models, select appropriate algorithm models to be integrated into products in the corresponding fields, or improve the performance of algorithm models that have been integrated into products, a language interpretability tool LIT is proposed. This tool can be used to evaluate natural language processing algorithm models. However, it is only applicable to algorithm models and various natural language processing tasks in the field of natural language processing, and cannot be adapted to a wider range of algorithm models in other fields. The adaptability of this tool is poor. Moreover, when performing white-box evaluation in the LIT tool, it is generally performed on the feature layer within the model (such as attention, salience maps, and embeddings), lacking evaluation and analysis of data samples and other levels. The analysis results are relatively simple and cannot understand the quality issues of the algorithm model from multiple levels.

[0070] In addition, the evaluation system provided in the related art is only accessed during the test phase (such as Figure 1 As shown in the figure, during the testing phase, the quality of the algorithm model is controlled by a single standard quality red line. It is impossible to give the possible reasons why the algorithm performance does not meet the standards and the general optimization direction), and the evaluation method is relatively single. The content that can be fed back to the algorithm model is very limited, resulting in low efficiency in optimizing and improving the algorithm model.

[0071] To solve at least one of the above problems, the present application provides a model evaluation method, device, electronic device and computer-readable storage medium; specifically, on the basis of conventional quality red line evaluation, full use is made of the white box evaluation capability in the AI-DL (Artificial Intelligence-Deep learning) toolbox to perform multi-dimensional evaluation. When the toolbox is applied to the automated link of algorithm evaluation, the depth and breadth of the content fed back to the algorithm are expanded.

[0072] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0073] The present application provides a model evaluation method, such as Figure 2 As shown, Figure 2 A flow chart of a model evaluation method provided in an embodiment of the present application is shown, wherein the method can be executed by any electronic device, such as a user terminal or a server. The user terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but the present application is not limited to this. As an optional implementation, the method can be executed by a user terminal. For the convenience of description, in the description of some optional embodiments below, the user terminal will be used as an example of the execution subject of the method. Specifically, the method includes the following steps S101-S104:

[0074] Step S101: Obtain training data and test data.

[0075] Specifically, training data refers to sample data used to build a model (which can be used to train the model or determine model parameters, such as determining the weights in ANN (Artificial Neural Network), and test data refers to sample data used to test the model, such as testing the generalization ability of a trained model.

[0076] Among them, training data and test data can be collected according to the field of current application of the model. For example, for image classification, the training data and test data can be sample data including image data and corresponding annotation information of the image data; for language processing, the training data and test data can be sample data including voice data or text data and corresponding annotation information.

[0077] Optionally, when obtaining training data and test data, a series of sample data can be obtained first, and then divided based on a preset ratio, such as 80% of the sample data as training data and 20% as sample data. The training data and test data are divided by the preset ratio, and the adjustment of model parameters in the training and testing stages is combined. This is conducive to improving the model's adaptability to fresh samples, improving the model's generalization ability, and improving the coverage in each sample during learning.

[0078] Step S102: Use training data to train the model.

[0079] Specifically, in the embodiment of the present application, when the training data is used to train the model, it can not only be used to train the model, but also can be used to synchronously adjust various parameters involved in building the model; among them, hyperparameters involved can be set through empirical values.

[0080] Among them, it can be understood that training a model in machine learning means learning the ideal values ​​of all weights and biases through labeled samples; in the training process of the machine learning algorithm, what is performed is learning the information of multiple samples and trying to find a model that can minimize the loss. The goal of training is to minimize the loss. In fact, the model training process is a process of continuous trial and error.

[0081] Step S103: using test data to test the trained model, and calculating a preset index value obtained by evaluating the model performance during the test.

[0082] Specifically, the process of model testing can be understood as considering whether the processing results of the model meet the expected accuracy or error through a series of test data with annotated information, and then evaluating the quality of the model. Among them, in step S103, after the model training is completed, the test data will be used to verify the model, and the performance of the model will be evaluated by the preset index value, for example: the higher the accuracy, the better the model performance. Optionally, in actual applications, the preset index value can be obtained by combining multiple index values, or it can be a single index value; for example, for a classification algorithm model, the evaluation index can be one of accuracy, recall, F1 score, ROC (Receiver Operating Characteristic) and AUC (Area Under Curve, the area under the ROC curve and the coordinate axis).

[0083] Optionally, in the embodiment of the present application, the preset index value can be determined by at least one of ROC and AUC. Among them, AUC is a model evaluation index in the field of machine learning, which is the area under the ROC curve.

[0084] The following describes the ROC curve and AUC index that may be used in the embodiments of the present application.

[0085] Taking the binary classification (0, 1) model as an example, the final output of the model is a probability value, which indicates the probability of the result being 1. Therefore, when determining whether the input X belongs to 0 or 1, it is generally represented by setting a threshold. If it exceeds the threshold, X is classified as 1, and if it is lower than the threshold, X is classified as 0. Therefore, different thresholds will lead to different classification results, that is, the corresponding confusion matrices are different. When the threshold gradually evolves from 0 to 1, many pairs of classification results will be formed. If the values ​​are expressed in a coordinate system, the corresponding ROC curve can be obtained. The area under the ROC curve corresponds to the AUC value, which can be used to evaluate the performance of the model. Among them, the AUC indicator can be used to indicate the possibility that the probability value of predicting a sample as positive is greater than the probability value of predicting it as negative; the AUC value can intuitively evaluate the quality of the model. Generally speaking, the larger the AUC value, the better the quality of the model.

[0086] Step S104: Determine based on the preset indicator value whether to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data, and the model.

[0087] Specifically, the evaluation system can be composed of at least one analysis module, each analysis module can be constructed by various neural networks, and the analysis modules can be connected through mathematical logic (such as performing comprehensive calculation and analysis on the outputs of multiple analysis modules to obtain optimization improvement information or evaluation information that the evaluation system feeds back to the model), forming the overall framework of the evaluation system.

[0088] Optionally, in step S104, at least one analysis module in the evaluation system is used to evaluate at least one of the training data, test data, and model; it is understandable that one analysis module can be used to evaluate the training data, test data, or model separately, or one analysis module can be used to evaluate at least two of the training data, test data, and model at the same time.

[0089] Among them, determining which analysis module in the evaluation system is used to evaluate which object based on the preset indicator value may include the following two cases, which are explained by taking the AUC value as the preset indicator value:

[0090] The first one: assuming that the threshold is set to 0.8 (the threshold can be adjusted according to actual conditions and is not limited in this application), when the AUC value is less than or equal to 0.8 (indicating that the quality of the model is poor), the analysis module in the evaluation system can be used to perform a global analysis of the model, training data, and test data.

[0091] The second type: assuming that the threshold is set to 0.8 (this threshold can be adjusted according to actual conditions and is not limited in this application), when the AUC value is greater than 0.8 (indicating that the quality of the model is basically up to standard, but some abnormal problem bad cases still need to be analyzed), the analysis module in the evaluation system can be used to analyze the model, training data and test data for abnormal problems.

[0092] In the embodiment of the present application, the provided AI-DL toolbox is referred to as an evaluation system in the provided model evaluation method, and the execution process of method steps S101-S104 reflects the content of the AI-DL toolbox performing automated evaluation of the algorithm model when applied.

[0093] When implementing this application, training data and test data are obtained, and after the model is trained using the training data, the trained model is tested using the test data, and the preset index value obtained by evaluating the model performance during the test is calculated, and then based on the preset index value, at least one analysis module in the evaluation system is determined to evaluate at least one of the training data, test data, and model. The implementation of this application can be adapted to various types of models, without limiting the field of the model, and improves the adaptability of model evaluation in various fields; on the other hand, this application can also implement evaluations of various dimensions of training data, test data, and models, provide multi-dimensional analysis results, and improve the efficiency of model optimization and improvement.

[0094] Next, combine Figure 3 , a process of how to evaluate at least one of the training data, the test data, and the model is described.

[0095] In one embodiment, step S104 determines based on the preset indicator value whether to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data, and the model, including the following steps S1041-S1042:

[0096] Step S1041: If the preset indicator value is less than or equal to the preset threshold, at least one of the following is used for evaluation: using the coverage analysis module in the evaluation system for measuring the integrity of the test data to evaluate the model; using the uncertainty analysis module in the evaluation system for measuring the uncertainty of the test data to evaluate the test data.

[0097] Among them, the coverage analysis module adopts a coverage algorithm, and coverage can be used to measure the integrity of sample data. In an embodiment of the present application, the model can be evaluated based on the coverage of measuring the integrity of training data and test data.

[0098] Among them, considering that for general deep learning models, the model generally directly outputs a high-confidence prediction result, but the model cannot determine the degree of confidence of the prediction result, but the degree of confidence of the model in its own prediction result is mutually constrained by the accuracy of the prediction. Therefore, an embodiment of the present application provides an algorithm for evaluating the uncertainty of the model for sample prediction based on uncertainty measurement, which can be called an uncertainty algorithm, which is specifically used in the uncertainty analysis module of the evaluation system. Specifically, the uncertainty algorithm can apply the MC-dropout (Monte-Carlo Dropout, Monte Carlo) algorithm, or the Deep-Ensemble algorithm.

[0099] Step S1042: Use at least one of the attribution analysis module for analyzing the importance of different features of test data to model decisions, the visualization analysis module for analyzing the importance of different features of test data to model decisions, and the influence analysis module for analyzing the importance of training data to the model's decision-making based on test data in the evaluation system to evaluate abnormal problems.

[0100] Optionally, step S1042 is implemented as follows: Figure 3 As shown, it can be understood that no matter whether the current indicator exceeds the quality red line, the embodiments of the present application analyze the abnormal problem bad case.

[0101] Among them, the attribution analysis module is used to analyze the importance of different features of samples (test data) for model decision-making in the field of machine learning involved in this application. In the evaluation system, a gradient scoring algorithm can be used for attribution analysis. Since the gradient scoring algorithm has a wide range of applicability, the use of the gradient scoring algorithm can enable the attribution analysis module to be applied to model evaluation in the fields of computer vision and natural language processing.

[0102] Compared with the visualization algorithms corresponding to the visualization analysis module and the attribution analysis module (observing whether the model can focus on the correct area from the perspective of a single sample), the influence function algorithm used in the influence analysis module used in this embodiment considers the dimension of the entire training set and mines the subsets in the training data that have an important influence on the model decision (training samples in the training data). The visualization algorithms corresponding to the visualization analysis module and the attribution analysis module can locate the problems of the model itself and help optimize and improve the model on the algorithm side; while the influence function algorithm helps to discover and improve problems in the training data on the algorithm side by scoring the training data used by the model.

[0103] Among them, the visualization analysis module in the embodiment of the present application can adopt the feature visualization algorithm of the convolution heat map grad-cam or grad-cam++, or can adopt UMAP (uniform manifold approximation and projection, manifold learning dimensionality reduction algorithm) or t-SNE (t-distributed Stochastic Neighbor Embedding, data dimensionality reduction) algorithm. The feature visualization algorithm using the convolution heat map can analyze the model behavior from the feature level of the sample data, and infer the influencing factors of the model's classification decision (or infer how the model makes the classification decision) from the model's focus area (focus position) for a certain type of annotation and a certain input X. The use of a dimensionality reduction algorithm can focus on the local structure of the data, and then understand the dynamic changes of the algorithm model in a visual way by understanding the changes in data distribution.

[0104] Among them, the influence analysis module specifically adopts the influence function algorithm, which specifically uses the back propagation of gradients to discover which training samples have a greater influence on the decision when the model makes a decision for each test sample.

[0105] The following describes the specific process of evaluating the model using the coverage analysis module.

[0106] Optionally, using the coverage analysis module to evaluate the model includes at least one of the following steps A1-A2:

[0107] Step A1: Determine the boundary range of each neuron in the model based on the training data, and evaluate the model based on the boundary range and the test data.

[0108] Specifically, step A1 corresponds to the edge coverage analysis method adopted in the coverage analysis module, wherein the definition of edge coverage can be understood as follows: for each neuron in the model, all training samples in the training data can obtain the output value range of the neuron (boundary range, composed of an upper boundary and a lower boundary) after passing through the forward network. If the output value of a test sample in the test data on the neuron falls within the boundary range, it can be determined that the test sample and the training sample belong to the same distribution. In this case, the trained model is likely to make a correct judgment, that is, the quality of the model is better; otherwise, the test sample and the training sample do not belong to a same distribution. In this case, the trained model may not be able to make a correct judgment, that is, the quality of the model is poor.

[0109] Optionally, in step A1, the boundary range of each neuron in the model is determined based on the training data, and the model is evaluated based on the boundary range and the test data, including the following steps A11-A14:

[0110] Step A11: Perform forward propagation processing based on the training data to determine the boundary range of each neuron in the model.

[0111] Specifically, by using the edge coverage algorithm to perform forward propagation operations on all training samples of the training data, the upper and lower boundaries corresponding to the activation of each neuron in the model can be obtained. Based on the upper and lower boundaries, the boundary range corresponding to each neuron can be determined.

[0112] Step A12: Perform forward propagation processing based on the test data to determine the number of neurons corresponding to each test sample in the test data that exceeds the boundary range.

[0113] Specifically, by performing a forward propagation operation on any test sample in the test data, it is possible to calculate how many neurons the test sample has out-of-bounds conditions in. That is, when performing the forward propagation operation, it is possible to determine whether the test sample has an output value in each neuron that exceeds the boundary range determined in step A11, and thus ultimately determine the number of neurons corresponding to each test sample that exceeds the boundary range.

[0114] For example, assuming that there are currently N neurons, and when performing a forward propagation operation based on test sample A, among the corresponding N output values, 3 output values ​​exceed the boundary range of the corresponding neurons determined in step A11, then step A12 can determine that the number of neurons corresponding to the test sample A that exceeds the boundary range is 3.

[0115] Step A13: Determine the confidence of the model's decision on each test sample based on the number of neurons.

[0116] Specifically, the larger the number of neurons that exceed the boundary range of the test sample determined in step A12, the lower the decision confidence of the model for the test sample, and the greater the possibility of a corresponding abnormal problem bad case.

[0117] Optionally, in order to reduce the computational complexity of the evaluation system, the confidence corresponding to the number of neurons determined in step A12 can be determined by setting a mapping relationship between the number of neurons and the confidence. For example, when the number of neurons is within the first interval (small value), the confidence is 1; when it is within the second interval (large value), the confidence is 0.5; optionally, more intervals and corresponding confidences can be set accordingly to improve the accuracy of the confidence determined in step A13.

[0118] Step A14: Determine the result of evaluating the model based on the confidence level.

[0119] Specifically, the confidence level determined by the coverage analysis module can be used to feedback corresponding problems with the model, training data or test data, and provide directions for model optimization and improvement based on the corresponding problematic content.

[0120] In the embodiment of the present application, the higher the coverage of the sample data, the higher the accuracy of the corresponding model training and testing. In addition, through the implementation of the above steps A11-A14, it can be expected that for the algorithm model that is fully trained with training data, when the edge coverage algorithm is used, the corresponding abnormal problem bad-case when the model is tested with test data can be determined without providing the annotation information ground-truth. The edge coverage algorithm provided in the embodiment of the present application is conducive to reducing the annotation cost of test data.

[0121] Step A2: Construct neuron pairs between model network layers, and evaluate the model based on the neuron pairs and test data.

[0122] Specifically, compared to the edge coverage algorithm provided in step A1, step A2 adopts a neuron pair coverage algorithm, which involves neuron pair coverage. Compared to the edge coverage algorithm used in step A1, which calculates coverage from the perspective of a single neuron, the neuron pair coverage algorithm provided in step A2 calculates coverage from the perspective of neuron pairs between network layers. In step A2, to meet the evaluation requirements, neuron pairs between different network layers are constructed based on the model currently being evaluated. At this time, the constructed neuron pairs can have multiple activation states, and then the neuron pair coverage is determined based on the conditions that the neuron pairs can be activated after the test data passes through the network, and then the neuron pair coverage is used as an indicator in the coverage algorithm to measure whether the number of test samples is sufficient to evaluate the algorithm model capability, as a means of measuring the adequacy of the test samples.

[0123] Optionally, the specific implementation process of the neuron pair coverage algorithm provided in step A2 is described by taking DNN (Deep Neural Networks) as an example, but the embodiment of the present application is not limited to DNN networks and can also be applied to other neural networks. Specifically, in step A2, neuron pairs are constructed, and the model is evaluated based on the neurons for test data, including the following steps A21-A23:

[0124] Step A21: Extract neurons from different layers of neurons in the model to construct multiple neuron pairs.

[0125] Specifically, the specific implementation process of step A21 is illustrated below based on the DNN network: randomly sample two different neurons in the same layer of neurons in the DNN network, and randomly sample a neuron in the next layer of the layer to construct a neuron triple, that is, a neuron pair; and so on, multiple neuron pairs can be constructed. Among them, the randomly sampled neurons are located between adjacent layers of the network, and the number of randomly sampled neurons between different layers can be adjusted according to actual conditions, and this application does not limit this.

[0126] Step A22: Determine the ratio of the number of neuron pairs in all activation states among the neuron pairs triggered by the test data to the number of all neuron pairs.

[0127] Specifically, we continue to use the example in step A21 as an example to explain step A22: Since each neuron corresponds to two states, a neuron triple can correspond to eight different activation states, and each test sample in the test data can trigger an activation state in a triple. Assuming that there are 9 neuron pairs, and after all test samples in the test data are processed by the DNN network, only 3 neuron pairs have all eight states triggered. At this time, the number of neuron pairs in all activation states triggered by the test data is 3, and the number of all neuron pairs is 9. The ratio of the two is 3 / 9=1 / 3; this ratio is the neuron pair coverage.

[0128] Step A23: Determine the result of evaluating the model based on the ratio.

[0129] Specifically, the ratio (neuron pair coverage) determined in step A22 can be used as an indicator in the coverage analysis module to measure whether the amount of test data is sufficient to evaluate the ability of the algorithm model, as a means of measuring the adequacy of the test samples.

[0130] In one embodiment, the step A23 determines the result of evaluating the model based on the ratio, and further includes the following steps A231-A232:

[0131] Step A231: Construct attack sample data based on the state information of neurons that are not activated by the test data.

[0132] Specifically, there may be multiple activation states corresponding to a neuron pair. When there is an inactivated state in a neuron pair, a test sample corresponding to the neuron that triggers the inactivated state is obtained as a basis for constructing attack sample data. Optionally, all test samples that trigger a neuron but fail to activate the neuron are used as attack sample data.

[0133] Step A232: Determine the boundary vulnerability information of the model based on the attack sample data.

[0134] Optionally, by testing the model with attack sample data, the weaknesses of the model performance can be effectively controlled, the boundary vulnerability information of the model can be determined, and the direction of model optimization and improvement can be determined. Among them, boundary vulnerability information can be understood as information that causes abnormal problems and bad cases when the model makes decisions.

[0135] The following describes the specific process of evaluating test data using the uncertainty analysis module.

[0136] In one embodiment, the uncertainty analysis module adopts the MC-dropout algorithm as an example to illustrate the specific process of using the uncertainty analysis module to evaluate the test data. Specifically, in step S1041, the uncertainty analysis module is used to evaluate the test data, including the following steps B1-B3:

[0137] Step B1: Perform forward propagation processing based on the test data to determine the prediction results of the test data.

[0138] Specifically, when applying the MC-dropout algorithm, a dropout layer needs to be set in the model and turned on in the test phase of the model; under certain conditions of dropout, it can be regarded as adding Gaussian noise to the model. Among them, the test data is forward propagated to obtain the result of the output layer as the prediction result of the test data; for example: assuming that the test data is a picture to be classified, and there are currently 3 types, the prediction result can be the probability value corresponding to the classification of the picture into categories 1, 2, and 3.

[0139] Step B2: Determine the uncertainty value of the test data based on at least one of the variance, entropy and mutual information of the prediction results.

[0140] Among them, for the variance, the method is to consider the average of the sum of the squares of the differences between each data and the mean from the mathematical operation logic. Therefore, the prediction result can be directly used for variance calculation to determine the uncertainty value of the test data. The larger the variance of the prediction result, the lower the prediction accuracy of the model for the test sample. Among them, the larger the variance of the prediction result, the greater the influence of the Gaussian noise added to the model on the prediction result. Correspondingly, it indicates that the input data X of the current model should be on a certain boundary of the model, and a little noise may cause the input X to cross the boundary.

[0141] Here, entropy generally refers to a measure of the state of certain material systems, the degree to which certain material system states may occur, and in the embodiments of the present application, can be used to characterize the uncertainty of the model in predicting test data.

[0142] Among them, mutual information is a useful information measure, which can be regarded as the amount of information contained in a random variable about another random variable, or the uncertainty of a random variable reduced by knowing another random variable.

[0143] Step B3: Determine the result of evaluating the test data based on the uncertainty value.

[0144] Specifically, by using the uncertainty value determined by the uncertainty algorithm, the test data can be pre-screened to select test samples with higher model uncertainty (such as those with uncertainty values ​​in the top N in descending order in the test data) as the "difficult" evaluation set (i.e., determine the hard-target) so as to determine the customized evaluation content and boundary samples of the subsequent model based on the "difficult" evaluation set.

[0145] In one embodiment, taking the Deep-Ensemble algorithm adopted by the uncertainty analysis module as an example, the specific process of using the uncertainty analysis module to evaluate the test data is described. Specifically, the use of the Deep-Ensemble algorithm can be understood as taking the mean of the prediction results of the model for the test data as the uncertainty value when testing the model. Similar to the above-mentioned use of the MC-dropout algorithm, the uncertainty value can also be used to pre-screen the test data, and determine the "difficult" evaluation set (hard-target), and then determine the customized evaluation content and boundary samples of the subsequent model.

[0146] The following describes the specific process of using the attribution analysis module to evaluate abnormal problems.

[0147] Optionally, using the attribution analysis module to evaluate abnormal problems includes the following steps C1-C2:

[0148] Step C1: Use the gradient integration algorithm to determine the distribution information of the contribution degree corresponding to each test sample in the test data in the sample characteristics.

[0149] Among them, the principle of the integrated gradient algorithm is to integrate the gradient along different paths, hoping to obtain the contribution (contribution degree) of the non-zero gradient in the unsaturated area to the decision-making importance.

[0150] In the embodiments of the present application, when evaluating models in different fields, the sample features for attribution analysis are different; for example, in the image field, the sample features are the pixels in the image; in the field of natural language processing, the sample features are the tokens of each word vector after the sentence is segmented. Specifically, the input of the gradient integral algorithm is a single sample data to be evaluated (the test sample in the test data), and the output is the distribution of the contribution degree corresponding to the sample data in the sample features.

[0151] Combine the following Figure 4 Explanation: Taking the input image as an example of a camera, in the distribution map output by the gradient integration algorithm, the pixels with a larger contribution are concentrated in the camera area in the image.

[0152] Step C2: Determine the result of the evaluation on the abnormal problem based on the distribution information.

[0153] Specifically, attribution analysis can also be a visual analysis method that can provide visualization results of the model decision focus area at the sample level (such as Figure 4 The output picture of the gradient integration algorithm shown in Figure 1 provides a strong basis for the optimization algorithm model.

[0154] Combine the following Figure 5 The gradient integral visualization evaluation results in the attribution analysis module of the embodiment of the present application are explained.

[0155] For the evaluation of BERT (Bidirectional Encoder Representation from Transformers, sentiment classification model), the gradient scoring algorithm is used to perform attribution analysis on the abnormal bad cases generated in the model.

[0156] Among them, the test sample for the current abnormal problem bad case is: This is a book that is very unlike Anne Baby. Although it is difficult to pronounce, I feel that this is the only way to describe it. Anne Baby has always given me the impression of a silent woman with a complicated and heavy mood, who is used to using a pen instead of language, and quietly reading gloomy words in a corner. So when I saw a collection of essays from her, I subconsciously ignored it. I guess this must be a replica of those unchanging words, cold and chilly. But why not open the first page and look at the table of contents first. This book is a leap, a change, which I found from the table of contents. The label corresponding to this test sample is: positive emotion.

[0157] The model's prediction is to truncate the text directly, from Figure 5 It can be analyzed from (Visualization diagram) that the first half of the test sample (the above sentence) corresponding to the above abnormal problem ( Figure 5 The corresponding content shown in the figure is filled with a large number of negative sentiment words, which makes the model judge that it has negative sentiment as a whole, without paying attention to the discarded transition content in the second half (the discarded transition content in the above sentence includes: replacing language, sitting alone in a corner and quietly reading the gloomy text. So when I saw a collection of essays from Yi, I subconsciously ignored it. I guess this must be another copy of those unchanging texts, cold and full of chill. But you might as well open the first page and look at the table of contents first. This book is a leap, a change, which I found from the table of contents), and then the discarded transition content represents the true emotional orientation of this test sample.

[0158] From the results of the above evaluation on abnormal problems, it can be found that the direct truncation operation of text makes it difficult for the model to make correct judgments on certain sample data. Therefore, an improved optimization method for the model of direct truncation operation of text can be fed back.

[0159] The following describes the specific process of using the visual analysis module to evaluate abnormal problems.

[0160] Optionally, when using the visual analysis module to evaluate abnormal problems, at least one of the following steps D1-D2 may be used:

[0161] Step D1: Use a convolutional heat map algorithm to determine the heat map corresponding to each test sample image in the test data and the annotation information of the test sample image; determine the result of the evaluation on the abnormal problem based on the heat map; wherein the focus position in the heat map represents the focus area of ​​the classification task under the category corresponding to the annotation information.

[0162] Specifically, Figure 6 As shown in , when the visualization analysis module adopts the convolutional heat map algorithm, its input includes the sample image to be evaluated and the target category for the sample image (annotated by the category label), and its output is a heat map. The focus position in the heat map represents the area that the model focuses on when performing the classification task under the target category. Figure 6 As shown in Figure (a), the input image includes dogs and cats, and the category label (annotation information) corresponding to Figure (a) is cat. The convolution heat map algorithm can focus the key area of ​​the model classification decision on the location of the cat in the image. As can be seen from Figure (c), the model has a good ability to distinguish and process this category (cat). Figure 6 As shown in the middle figure (g), the input image includes dogs and cats, and the category label corresponding to figure (g) is dog. The convolution heat map algorithm can focus the key area of ​​the model classification decision on the position of the dog in the picture. From figure (i), it can be seen that the model has a good ability to distinguish and process this category (dog). In other words, the convolution heat map algorithm can be used to analyze model behavior, infer the model's classification decision from the model's focus area for a certain category and a certain input, and then study the reasons for the current abnormal problem bad case and the reason why the model cannot make an accurate judgment on the sample of this category.

[0163] Combine the following Figure 7a and Figure 7b The visualization evaluation results of the convolutional heat map grad-cam algorithm are explained.

[0164] The following example uses the convolutional heat map algorithm to perform visualization analysis in the eating broadcast vertical category, which is not suitable for the image classification model. Specifically, given a negative class (normal image, such as Figure 7a ) as the target category, it can be found that when the model processes the negative class image, the corresponding output heat map can focus on the food, that is, the model's classification processing is consistent with the focus area when the operation classification is considered. However, when a positive class (unsuitable image, such as Figure 7b As shown in the figure, when the model uses the positive image as the target category, it can be found that the corresponding output heat map focuses on meaningless pixels when processing the positive image. It can be inferred that in the vertical category of eating broadcasting, the model's decision to judge it as a positive sample is flawed. At this time, feedback can be given to suggest adding more positive samples or building a small vertical model.

[0165] Step D2: Use a dimensionality reduction algorithm to monitor distribution change information of test data during model testing; determine the results of evaluation for abnormal problems based on the distribution change information of test data.

[0166] Specifically, the manifold learning dimensionality reduction algorithm UMAP can be used, or the data dimensionality reduction algorithm t-SNE can be used. The principle of the t-SNE algorithm is to convert the similarity between data points into probabilities, and observe the algorithm results by projecting them in a two-dimensional or three-dimensional space (visualization process). During the model testing process, the data distribution changes of the test samples during the model testing process can be observed from a visual perspective, which helps to optimize the model based on the test situation.

[0167] The following describes the specific process of using the influence analysis module to evaluate abnormal problems.

[0168] In one embodiment, using the influence analysis module to evaluate abnormal issues includes the following steps E1-E2:

[0169] Step E1: Determine the importance score of each training sample in the training data relative to the test data based on the training data and the test data.

[0170] Step E2: Determine the result of the evaluation on the abnormal problem based on the importance score.

[0171] Specifically, the input of the influence function algorithm used in the influence analysis module is the test data to be evaluated and the training data, and the output is the importance score of each training sample in the training data relative to the test data. After obtaining the importance score calculated by the algorithm, on the one hand, the algorithm can find the most important training sample subset for the abnormal problem bad case and check whether there is any information with incorrect labeling; on the other hand, it can dig out common problems in the training sample subset with higher importance, and then infer the decision-making mode of the model.

[0172] Combine the following Figure 8 The evaluation results of the influence analysis module in the embodiments of the present application are illustrated by way of example.

[0173] Specifically, the influence analysis of the training data and test data of the discomfort image classification model can be obtained as follows: Figure 8 The analysis results are shown in Figure 2. Figure 8 Harmful training samples mean that such training samples have a negative impact when used to evaluate test samples, and helpful training samples mean that such training samples have a positive impact when used to evaluate test samples.

[0174] Among them, Figure 8 As shown in the figure, in the test sample image, the image is judged as a negative sample after model processing, but it is actually a positive sample. When calculating the influence score of the training sample corresponding to the test sample, most of the training samples that help it the most (helpful samples) are positive samples and are related to the field of eating broadcasts, but in the harmful samples, the data that also has a great influence on it is also related to the field of eating broadcasts. The content of the person eating in the picture is also the same as that in the test sample, but the corresponding actual annotation information (label) is a negative sample. Therefore, it can be inferred that the relevant training samples may be incorrectly labeled.

[0175] In a feasible embodiment, the model is trained using training data in step S102, and further includes steps S1021-S1022:

[0176] Step S1021: using a dimensionality reduction algorithm to monitor distribution change information of training data during model training.

[0177] Step S1022: Determine the result of evaluating the model based on the distribution change information of the training data.

[0178] In an embodiment of the present application, the visualization module introduced in the evaluation system can specifically be a model iteration visualization module, which can be used to monitor the training process, so that the algorithm not only focuses on the loss indicator during the training process, but also observes the changes in the data distribution of the training samples in the training data as the training process changes from a visualization perspective. The implementation of this algorithm is conducive to improving the training of the model.

[0179] Specifically, the same as the dimensionality reduction algorithm shown in the above embodiment, the manifold learning dimensionality reduction algorithm UMAP and the data dimensionality reduction algorithm t-SNE can also be used in step S1021. The following uses the UMAP algorithm as an example to illustrate that in the scenario where the amount of training data is large, the UMAP algorithm has the characteristics of being able to retain the local results and overall structure of the training data, and running fast and occupying little memory. After the UMAP algorithm is integrated into the evaluation system, the algorithm can be used in the model training phase to monitor the changes in the distribution of training data during the training process, and to present the dynamic process of algorithm training in a visual manner.

[0180] In order to further illustrate the evaluation results of the visual analysis module provided in the embodiment of the present application in the model training iteration process, the following is combined with Figure 9a and Figure 9b , taking the training process of the sentiment classification model BERT as an example. Figure 9a It is the first epoch graph obtained during model training. Figure 9b This is the second epoch graph obtained during model training. Figure 9a and Figure 9b It can be found that as the model training deepens, the model shows better ability to distinguish between the two categories.

[0181] Based on the various algorithms provided in the above embodiments, the embodiments of the present application can integrate various algorithms into an evaluation system (AI-DL toolbox) by using a computer programming language package PYTHON package to call the training and test evaluation parts of the algorithm model.

[0182] The AI-DL toolbox provided in the embodiment of the present application can make full use of the white-box evaluation capabilities in the AI-DL toolbox for multi-dimensional evaluation in the automated link of algorithm white-box evaluation, based on the benchmark of conventional quality red-line evaluation: when the quality red-line does not meet the standard (the preset indicator value is less than or equal to the preset threshold), the model coverage analysis can be performed from the model dimension to examine the coverage of the neurons inside the model by the sample data, and feedback on the model problems; it can also be based on the data dimension to construct a specific "difficult" evaluation data set by performing uncertainty analysis on the test data to observe the performance of the model on this type of evaluation set, so as to obtain certain common problems that lead to algorithm failure. On the other hand, the AI-DL toolbox can also conduct multi-dimensional analysis based on the abnormal problem bad case (the preset indicator value is greater than the preset threshold). For example, perform attribution analysis on bad cases, analyze the contribution of different features in the test samples to the model decision, and explore the potential patterns of algorithm model classification; visualize the intermediate features of the bad case input, and judge the model's focus (or focus area, focus position) through information such as heat maps; perform influence analysis on training samples to find the sample data in the training samples that has the greatest influence on the generation of bad cases, in the hope of discovering errors. It can be seen that the AI-DL toolbox provided in the embodiment of the present application is applied to the automated link of algorithm evaluation, and the content fed back to the algorithm is expanded in depth and breadth.

[0183] The present application embodiment provides a model evaluation device, such as Fig.10 As shown, the model evaluation device 100 may include: an acquisition module 101, a training module 102, a test module 103 and an evaluation module 104. The acquisition module 101 is used to acquire training data and test data; the training module 102 is used to train the model using the training data; the test module 103 is used to test the trained model using the test data and calculate the preset index value on the test data; the evaluation module 104 is used to determine to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data and the model based on the preset index value.

[0184] In one embodiment, when the evaluation module 104 is used to perform the step of determining to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data, and the model based on the preset indicator value, it is also used to perform the following steps:

[0185] If the preset indicator value is less than or equal to the preset threshold, at least one of the following is used for evaluation: using the coverage analysis module in the evaluation system for measuring the integrity of the test data to evaluate the model; using the uncertainty analysis module in the evaluation system for measuring the uncertainty of the test data to evaluate the test data;

[0186] Abnormal problems are evaluated by using at least one of the following: an attribution analysis module for analyzing the importance of different features of test data to model decisions, a visualization analysis module for analyzing the importance of different features of test data to model decisions, and an influence analysis module for analyzing the importance of training data to the model's decision-making based on test data.

[0187] In one embodiment, when the evaluation module 104 is used to perform the step of using the coverage analysis module to evaluate the model, it includes performing at least one of the following:

[0188] Determine the boundary range of each neuron in the model based on the training data, and evaluate the model based on the boundary range and test data;

[0189] Construct neuron pairs between model network layers and evaluate the model based on neuron pairs and test data.

[0190] In one embodiment, the evaluation module 104 is used to perform the steps of determining the boundary range of each neuron in the model based on the training data and evaluating the model based on the boundary range and the test data, and is also used to perform the following steps:

[0191] Perform forward propagation processing based on the training data to determine the boundary range of each neuron in the model;

[0192] Perform forward propagation processing based on the test data to determine the number of neurons corresponding to each test sample in the test data that exceeds the boundary range;

[0193] Determine the confidence level of the model's decision on each test sample based on the number of neurons;

[0194] The results of the evaluation of the model are determined based on the confidence level.

[0195] In one embodiment, when the evaluation module 104 is used to execute the steps of constructing neuron pairs and evaluating the model based on the neuron pairs and the test data, it is also used to execute the following steps:

[0196] Extract neurons from different layers of neurons in the model to construct multiple neuron pairs;

[0197] Determine the ratio of the number of neuron pairs in all activation states among the neuron pairs triggered by the test data to the number of all neuron pairs;

[0198] The results of the evaluation of the model are determined based on the ratio.

[0199] In one embodiment, when the evaluation module 104 is used to perform the step of determining the result of evaluating the model based on the ratio, it is also used to perform the following steps:

[0200] Construct attack sample data based on the state information of neurons activated by untested data;

[0201] Based on the attack sample data, determine the boundary vulnerability information of the model.

[0202] In one embodiment, when the evaluation module 104 is used to perform the step of using the uncertainty analysis module to evaluate the test data, it is also used to perform the following steps:

[0203] Perform forward propagation processing based on the test data to determine the prediction results of the test data;

[0204] Determining an uncertainty value of the test data based on at least one of a variance, an entropy, and a mutual information of the prediction result;

[0205] The results of the evaluation of the test data are determined based on the uncertainty value.

[0206] In one embodiment, when the evaluation module 104 is used to perform the step of using the attribution analysis module to evaluate the abnormal problem, it is used to perform the following steps:

[0207] The gradient integral algorithm is used to determine the distribution information of the contribution degree corresponding to each test sample in the test data in the sample characteristics;

[0208] The results of the evaluation for the abnormal problem are determined based on the distribution information.

[0209] In one embodiment, when the evaluation module 104 is used to perform the step of using the visual analysis module to evaluate the abnormal problem, it is also used to perform at least one of the following:

[0210] A convolutional heat map algorithm is used to determine the heat map corresponding to each test sample image in the test data and the annotation information corresponding to the test sample image; the result of the evaluation on the abnormal problem is determined based on the heat map; the focus position in the heat map represents the focus area of ​​the classification task under the category corresponding to the annotation information;

[0211] In the process of model testing, a dimensionality reduction algorithm is used to monitor the distribution change information of the test data; based on the distribution change information of the test data, the results of the evaluation on the abnormal problem are determined.

[0212] In one embodiment, when the evaluation module 104 is used to execute the step of using the influence analysis module to evaluate the abnormal problem, it is also used to execute:

[0213] Determine the importance score of each training sample in the training data relative to the test data based on the training data and the test data;

[0214] The results of the evaluation of the abnormal issues are determined based on the importance scores.

[0215] In one embodiment, when the training module 102 is used to perform the step of training the model using the training data, it is also used to perform the following steps:

[0216] Use dimensionality reduction algorithms to monitor distribution changes in training data during model training;

[0217] The results of evaluating the model are determined based on the distribution change information of the training data.

[0218] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module in the device in each embodiment of the present application correspond to the steps in the method in each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0219] In an embodiment of the present application, an electronic device is provided, which includes: a memory and a processor; at least one program, stored in the memory, for being executed by the processor, compared with the prior art, it can be achieved: an evaluation system is provided in the present application, and the evaluation system is used to perform the whole process of model training and testing during model evaluation; during implementation, training data and test data are obtained, the model is trained with the training data, the trained model is tested with the test data, and the preset index value obtained by evaluating the model performance during the test is calculated, and then based on the preset index value, at least one analysis module in the evaluation system is determined to evaluate at least one of the training data, test data and model. The implementation of the present application can be adapted to various models, without limiting the field of the model, and improves the adaptability of model evaluation in various fields; on the other hand, the present application can also realize evaluation of various dimensions of training data, test data and models, provide multi-dimensional analysis results, and improve the efficiency of model optimization and improvement.

[0220] In an alternative embodiment, an electronic device is provided, such as Fig.11 As shown, Fig.11 The electronic device 1100 shown includes: a processor 1101 and a memory 1103. The processor 1101 and the memory 1103 are connected, such as through a bus 1102. Optionally, the electronic device 1100 may also include a transceiver 1104, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 1104 is not limited to one, and the structure of the electronic device 1100 does not constitute a limitation on the embodiments of the present application.

[0221] Processor 1101 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1101 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0222] The bus 1102 may include a path to transmit information between the above components. The bus 1102 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1102 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0223] The memory 1103 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0224] The memory 1103 is used to store the application code (computer program) for executing the solution of the present application, and the execution is controlled by the processor 1101. The processor 1101 is used to execute the application code stored in the memory 1103 to implement the contents shown in the above method embodiment.

[0225] Among them, electronic devices include but are not limited to: smart phones, tablets, laptops, smart speakers, smart watches, car-mounted equipment, etc.

[0226] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the model evaluation method provided in the above-mentioned various optional implementations.

[0227] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0228] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0229] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A model evaluation method, characterized in that: include: Get training data and test data; Using the training data to train the model; Using the test data to test the trained model, and calculating a preset index value obtained by evaluating the performance of the model during the test; Determining based on the preset indicator value to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data, and the model includes: If the preset indicator value is less than or equal to the preset threshold, at least one of the following is used for evaluation: The model is evaluated by using a coverage analysis module in the evaluation system for measuring the integrity of the test data; The uncertainty analysis module in the evaluation system for measuring the uncertainty of the test data is used to evaluate the test data; Abnormal problems are evaluated by using at least one of the following: an attribution analysis module for analyzing the importance of different features of test data to model decisions, a visualization analysis module for analyzing the importance of different features of test data to model decisions, and an influence analysis module for analyzing the importance of training data to the model's decision-making based on test data.

2. The method according to claim 1, characterized in that The model is evaluated using the coverage analysis module, including at least one of the following: Determine the boundary range of each neuron in the model based on the training data, and evaluate the model based on the boundary range and the test data; Construct neuron pairs between network layers of the model, and evaluate the model based on the neuron pairs and the test data.

3. The method according to claim 2, characterized in that The step of determining a boundary range of each neuron in the model based on the training data, and evaluating the model based on the boundary range and the test data, comprises: Performing forward propagation processing based on the training data to determine the boundary range of each neuron in the model; Performing forward propagation processing based on the test data to determine the number of neurons corresponding to each test sample in the test data that exceeds the boundary range; Determining the confidence of the model's decision on each test sample based on the number of neurons; A result of evaluating the model is determined based on the confidence level.

4. The method according to claim 2, characterized in that: The step of constructing the neuron pairs between the network layers of the model and evaluating the model based on the neuron pairs and the test data includes: Extracting neurons from different layers of neurons in the model to construct a plurality of neuron pairs; Determine the ratio of the number of all activated neuron pairs in the test data-triggered neuron pairs to the number of all neuron pairs; A result of evaluating the model is determined based on the ratio.

5. The method according to claim 4, characterized in that The determining of the result of evaluating the model based on the ratio further includes: constructing attack sample data based on state information of the neuron pairs that are not activated by the test data; Based on the attack sample data, boundary vulnerability information of the model is determined.

6. The method according to claim 1, characterized in that The uncertainty analysis module is used to evaluate the test data, including: Perform forward propagation processing based on the test data to determine a prediction result of the test data; Determining an uncertainty value of the test data based on at least one of the variance, entropy, and mutual information of the prediction result; A result of evaluating the test data is determined based on the uncertainty value.

7. The method according to claim 1, characterized in that The attribution analysis module is used to evaluate abnormal problems, including: Using a gradient integral algorithm to determine the distribution information of the contribution degree corresponding to each test sample in the test data in the sample characteristics; A result of evaluating the abnormal problem is determined based on the distribution information.

8. The method according to claim 1, characterized in that: The visual analysis module is used to evaluate abnormal problems, including at least one of the following: Using a convolution heat map algorithm for each test sample image in the test data and the annotation information corresponding to the test sample image, determine a heat map corresponding to the test sample image; Determine the result of the evaluation for the abnormal problem based on the heat map; the focus position in the heat map represents the focus area of ​​the model for the classification task under the category corresponding to the labeled information; In the process of model testing, a dimensionality reduction algorithm is used to monitor distribution change information of test data; and a result of evaluation on abnormal problems is determined based on the distribution change information of the test data.

9. The method according to claim 1, characterized in that: The impact analysis module is used to evaluate abnormal issues, including: Determine, based on the training data and the test data, an importance score of each training sample in the training data relative to the test data; A result of evaluating the abnormal problem is determined based on the importance score.

10. The method according to any one of claims 1 to 9, characterized in that Using the training data to train the model also includes: Use dimensionality reduction algorithms to monitor distribution changes in training data during model training; The result of evaluating the model is determined based on the distribution change information of the training data.

11. A model evaluation device, characterized in that: include: An acquisition module is used to obtain training data and test data; A training module, used to train the model using the training data; A testing module, used to test the trained model using the test data and calculate a preset indicator value on the test data; An evaluation module, used to determine, based on the preset indicator value, to use at least one analysis module in the evaluation system to evaluate at least one of the training data, the test data, and the model, including: If the preset indicator value is less than or equal to the preset threshold, at least one of the following is used for evaluation: The model is evaluated by using a coverage analysis module in the evaluation system for measuring the integrity of the test data; the test data is evaluated by using an uncertainty analysis module in the evaluation system for measuring the uncertainty of the test data; Abnormal problems are evaluated by using at least one of the following: an attribution analysis module for analyzing the importance of different features of test data to model decisions, a visualization analysis module for analyzing the importance of different features of test data to model decisions, and an influence analysis module for analyzing the importance of training data to the model's decision-making based on test data.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; One or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer storage medium is used to store computer instructions. When the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 10.

Images