Deep learning model information evaluation method

Abstract

The invention relates to a deep learning model information evaluation method, which can solve the problem that the information difference of different models cannot be evaluated in a zero-sample environment at present, and provides performance evaluation indexes for active learning and other methods. Comprising the following steps: firstly, preparing two image classification models, then creating an initial random image, carrying out normalization preprocessing on the image, respectively inputting the image into two models to be evaluated, and obtaining a classification probability vector; then, a loss function is set by utilizing the classification probability vector, the image is optimized by utilizing the loss function, and finally, the optimized image is subjected to normalization removal to obtain a trained deep dream. And finally, inputting the obtained deep dream into the two models to be evaluated to obtain two groups of predicted class logs vectors, and calculating a model information difference value by using the class logs vectors. The information difference of the two models can be accurately evaluated without participation of a real sample.

Description

Technical Field

[0001] The present invention relates to the field of deep learning measurement, and proposes a method for measuring information differences between different models through quantitative indicators, and particularly relates to a model information evaluation method based on deep learning without the participation of real samples. Background Art

[0002] At present, in the application of deep learning, it is necessary to measure the information differences between different models and provide performance evaluation indicators for methods such as data domain adaptation, active learning, and semi-supervised learning. However, there is currently no zero-shot method for evaluating model information differences. The current method generally applies the performance evaluation indicators designed for a single model to two models, and then calculates the difference in the indicators to obtain the information differences of different models. For example, in image classification tasks, the model differences can be evaluated by comparing the accuracy differences between two models on the same dataset. However, this method requires the test dataset to participate in the measurement process of model information differences. If the dataset cannot be accessed due to privacy or other issues, this method cannot be used and cannot meet the zero-shot scenario. Summary of the invention

[0003] The purpose of the present invention is to fill the gap in the zero-sample model information difference evaluation method and propose a deep learning model information evaluation method. The technical solution is as follows: A method for evaluating deep learning model information: (1) Prepare two image classification models M that need to evaluate the model information difference A 、M B ; (2) Create initial random images and normalize and preprocess them for training Deep Dream; (3) Input the normalized initial random images into the model M A 、M B , get the classification probability vector P A , P B ; (4) Using the classification probability vector P A , P B Set the loss function L, and then use the loss function to perform multiple rounds of optimization on the initial random image. Finally, denormalize the optimized image to obtain the trained Deep Dream. (5) Re-input the obtained deep dream into model M A 、M B , get two sets of predicted category logits vectors l A , l B ; (6) Using the category logits vector l A, l B Calculate the model information difference value s. The beneficial effects of the present invention are: (1) The present invention is a specially designed model information difference evaluation method, which can quantify model information differences more accurately than the current accuracy difference method. (2) The present invention is a zero-sample method that can obtain the model information difference value without the need for real images to participate in the calculation. (3) This method has been verified through a large number of experiments, which effectively improves the reliability of the method. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is the overall structure diagram of the present invention; DETAILED DESCRIPTION To make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. The present invention is specifically implemented according to the following steps: (1) Prepare two image classification models M that need to evaluate the model information difference A 、M B , the image classification tasks served by the two models need to be the same, that is, the classification categories need to be consistent; (2) Create N c *N initial random images that follow a uniform distribution Where N c Represents the number of categories, N represents the number of deep dreams in a category, and then according to the mean μ (μ = [0.485, 0.456, 0.406]) and standard deviation σ (σ = [0.229, 0.224, 0.225]) of the ImageNet dataset Normalized to x for training Deep Dream; (3) Input the initial random image x into the model M A 、M B Get the category logits vector, and then normalize it with the softmax function to get the classification probability vector P A , P B : Where N c Represents the total number of categories, Representative Model M A Predict the probability that image x belongs to the jth class; (4) Using the classification probability vector P A , P B Set the loss function L: Where y represents the yth type of deep dream. Then use the loss function to perform multiple rounds of optimization on image x: Where α represents the learning rate, x i represents the image optimized to the i-th round. This process is repeated until the preset number of rounds is reached, and finally the optimized image is denormalized to obtain the trained deep dream x final ; x final =σ*x+μ (5) Take a deep dream x belonging to the i-th category i Re-enter model M A 、M B , get two sets of predicted categories log its vector in Represents the i-th type of deep dream model M A The logits value when identified as the jth class, Represents the i-th type of deep dream model M B The logits value when it is identified as the jth class; (6) Repeat step (5) until all N c *N deep dreams are predicted by the model as the logits values ​​of each category, and then the average of these logits values ​​is calculated to get the mean of the logits values ​​of all deep dreams of category i when they are predicted as category j Take advantage of all categories The model information difference value s can be calculated by summing up the absolute values ​​of the differences.

Claims

1. A deep learning model information evaluation method, characterized in that: The steps to assess model information discrepancy include: (1) Prepare two image classification models M that need to evaluate the model information difference A 、M B ; (2) Create initial random images that follow a uniform distribution and normalize them according to the mean and variance of the ImageNet dataset for training Deep Dream. (3) Input the normalized initial random images into the model M A 、M B , get the classification probability vector P A , P B ; (4) Using the classification probability vector P A , P B Set the loss function L, and then use the loss function to perform multiple rounds of optimization on the initial random image. Finally, denormalize the optimized image to obtain the trained Deep Dream. (5) Re-input the obtained deep dream into model M A 、M B , get two sets of predicted category logits vectors l A , l B ; (6) Using the category logits vector l A , l B Calculate the model information difference value s.

2. A deep learning model information evaluation method as claimed in claim 1, characterized in that: The method for obtaining deep dreams in step (4) is as follows: Using Model M A 、M B The classification probability vector P for the initial random image x A , P B Set the loss function L: Where y represents the yth type of deep dream, N c Represents the total number of categories. Then the loss function is used to perform multiple rounds of optimization on the image x: Where α represents the learning rate, x i represents the image optimized to the i-th round. This process is repeated until the preset number of rounds is reached, and finally the optimized image is denormalized to obtain the trained deep dream.

3. A deep learning model information evaluation method as claimed in claim 1, characterized in that: The method for calculating the model information difference value in step (6) is: Re-input the trained deep dream into model M A 、M B , get the logits value of each deep dream when it is predicted by the model as each category, then calculate the average of these logits values, and get the mean of the logits values ​​of all deep dreams of category i when they are predicted as category j in Represents the i-th type of deep dream model M A The logits value when identified as the jth class, Represents the i-th type of deep dream model M B The logits value when it is identified as the jth class. Then use all the categories The model information difference value s can be calculated by summing up the absolute values ​​of the differences.

Images