A deep learning model information evaluation method

Abstract

The present application relates to a kind of deep learning model information evaluation method, can solve the problem that current information difference of different models cannot be evaluated in the environment of zero sample, and provide performance evaluation index for active learning method. Including the following steps: first, prepare two image classification models, then create initial random image, and the image is normalized preprocessing, then the image is input into two models to be evaluated respectively, and the classification probability vector is obtained. Then the classification probability vector is used to set loss function, the image is optimized using loss function, and finally the optimized image is denormalized, and the trained deep dream is obtained. Finally, the obtained deep dream is input into two models to be evaluated, and two groups of predicted class logits vectors are obtained, and the class logits vector can be used to calculate model information difference value. The present application can accurately evaluate the information difference of two models without real samples.

Description

Technical Field

[0001] This invention relates to the field of deep learning econometrics, and proposes a method for measuring the information differences between different models through quantitative indicators, particularly a model information evaluation method based on deep learning that does not require the participation of real samples. Background Technology

[0002] Currently, in deep learning applications, it is necessary to measure the information differences between different models to provide performance evaluation metrics for methods such as data domain adaptation, active learning, and semi-supervised learning. However, there is currently no zero-shot method for evaluating model information differences. Current methods generally apply performance evaluation metrics designed for a single model to two models, and then obtain the information differences between the models by calculating the difference in these metrics. For example, in image classification tasks, model differences can be evaluated by comparing the accuracy difference between two models on the same dataset. However, this method requires the test dataset to be involved in the measurement of model information differences. If the dataset is inaccessible due to privacy or other issues, this method cannot be used, thus failing to meet the requirements of zero-shot scenarios. Summary of the Invention

[0003] The purpose of this invention is to fill the gap in methods for evaluating information differences in zero-shot models by proposing a method for evaluating information in deep learning models. The technical solution is as follows:

[0004] A method for evaluating information in deep learning models:

[0005] (1) Prepare two image classification models M that need to be evaluated for differences in model information. A M B ;

[0006] (2) Create an initial random image and perform normalization preprocessing on it for training deep dreams;

[0007] (3) Input the normalized initial random images into model M respectively. A M B Obtain the classification probability vector P A P B ;

[0008] (4) Using the classification probability vector P A P B Set the loss function L, then use the loss function to optimize the initial random image in multiple rounds, and finally denormalize the optimized image to obtain the trained deep dream;

[0009] (5) Re-input the obtained deep dream into model M A M BTwo sets of predicted category logits vectors are obtained. A l B ;

[0010] (6) Utilizing category logits vectors A l B Calculate the information difference value s of the model.

[0011] The beneficial effects of this invention are:

[0012] (1) This invention is a specially designed method for evaluating model information differences, which can more accurately quantify model information differences compared to current accuracy difference methods.

[0013] (2) This invention is a zero-sample method that can obtain the model information difference value without the participation of real images in the calculation.

[0014] (3) This method has been tested and verified through a large number of experiments, which has effectively improved the reliability of the method. Attached Figure Description

[0015] Figure 1 This is an overall structural diagram of the present invention; Detailed Implementation

[0016] To make the technical solution of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings. The present invention is implemented in specific steps:

[0017] (1) Prepare two image classification models M that need to be evaluated for differences in model information. A M B The two models need to serve the same image classification task, that is, the classification categories need to be consistent;

[0018] (2) Create N c *N initial random images following a uniform distribution Where N c Let N represent the number of categories, and N represent the number of deep dreams in a category. Then, based on the mean μ (μ = [0.485, 0.456, 0.406]) and standard deviation σ (σ = [0.229, 0.224, 0.225]) of the ImageNet dataset, we can further divide the data into categories. Normalized to x, used for training deep dreams;

[0019]

[0020] (3) Input the initial random image x into model M respectively A M B The category logits vector is obtained, and then the classification probability vector P is obtained after normalization by the softmax function.A P B :

[0021]

[0022] Where N c Represents the total number of categories. Representative model M A Predict the probability that image x belongs to class j;

[0023] (4) Using the classification probability vector P A P B Set the loss function L:

[0024]

[0025] Where y represents the y-th type of depth dream. Then, the image x is optimized in multiple rounds using a loss function:

[0026]

[0027] Where α represents the learning rate, x i This represents the image optimized to round i. This process is repeated until a preset number of rounds is reached. Finally, the optimized image is denormalized to obtain the trained deep dream image x. final ;

[0028] x final =σ*x+μ

[0029] (5) Take a deep dream x belonging to the i-th class i Re-enter model M A M B This yields two sets of predicted category logits vectors.

[0030]

[0031] in The i-th type of deep dream is represented by model M A The logits value when identified as class j. The i-th type of deep dream is represented by model M B The logits value when identified as class j;

[0032] (6) Repeat step (5) until all N are obtained. c * The logits values ​​of N deep dreams predicted by the model to each category are then calculated, and the average of these logits values ​​is obtained to get the mean logits value of all deep dreams of category i predicted to be category j.

[0033]

[0034] Utilize all categories The sum of the absolute values ​​of the differences can be used to calculate the model information difference value s.

[0035]

Claims

1. A method for evaluating the information of a deep learning model, characterized in that, The steps for assessing differences in model information include: (1) Prepare two image classification models whose information differences need to be evaluated. , ; (2) Create initial random images that follow a uniform distribution and normalize them according to the mean and variance of the ImageNet dataset for use in training deep dreams; (3) Normalize the initial random image Input the model respectively , The class logits vector is obtained, and then the classification probability vector is obtained after normalization by the softmax function. , ; (4) Using classification probability vectors , Set the loss function : in Representing the Deep dream-like Representing the total number of categories, then applying a loss function to the image. Perform multiple rounds of optimization: in Represents the learning rate. Representative optimized to the number The process is repeated for each round of images until a preset number of rounds is reached. Finally, the optimized images are normalized to obtain the trained deep dream. (5) Re-input the trained deep dreams into the model to obtain the logits value when each deep dream is predicted by the model to be of each category, and average the logits value corresponding to the deep dreams of the same category to obtain the average logits value when the i-th deep dream is predicted to be of the j-th category. (6) Based on the mean logits obtained by different models for the same category of deep dreams, calculate the absolute value of the difference between the mean logits of the two models, and sum the absolute value differences of all categories to obtain the model information difference value.

Images