Evaluation method of vision transformer classification model in vertical scene

Abstract

The application provides a Vision Transformer (ViT) classification model automatic performance evaluation method suitable for vertical industry application scenarios, and in view of the problem that the black box model structure is unknown and the inference process cannot be obtained, a multi-dimensional, cross-task and migratable evaluation process is designed. The method comprises the following steps: selecting a vertical benchmark data set, constructing a lightweight proxy model based on knowledge distillation, performing transfer learning fine tuning and dynamically recording the training process, calculating the universality index through the multi-task training efficiency and classification accuracy, and comparing the horizontal performance of different ViT models. The method can scientifically evaluate the generalization ability and application adaptability of the ViT model without relying on the internal information of the model, support model optimization and deployment decision, and is especially suitable for application fields such as medical images, industrial quality inspection, remote sensing monitoring and the like which have strict requirements on the performance of the model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision performance evaluation, specifically relating to an automated performance evaluation method for Vision Transformer (ViT) classification models in vertical industry application scenarios. This method is suitable for evaluating black-box models with unknown model structures or inference processes that are not readily available, especially in application areas such as medical imaging, industrial quality inspection, and remote sensing monitoring, where high classification accuracy and generalization ability are required. Background Technology

[0002] With the development of deep learning technology, Vision Transformer (ViT), as a visual classification model based on self-attention mechanism, has demonstrated strong expressive power and cross-task adaptability when handling complex scenes and large-scale data. Due to its excellent feature modeling ability and good transfer learning performance, it has been widely applied in various vertical industries such as image recognition, medical image diagnosis, and industrial defect detection. However, existing ViT model evaluation methods have significant shortcomings in practical applications, mainly in the following aspects: First, there is the limitation of model agnosticness. In many industry applications, users typically access models through cloud services or inference APIs, without obtaining the model's specific structure, parameters, and intermediate features. In this black-box environment, traditional evaluation methods based on internal model information (such as gradient visualization and attention heatmap analysis) are inapplicable, making it difficult for users to effectively assess model performance.

[0003] Secondly, the evaluation metrics lack comprehensiveness. Current mainstream evaluations typically focus on the model's inference speed, parameter size, or single accuracy, neglecting multi-dimensional comprehensive evaluation, such as the model's robustness, generalization ability, and adaptability to application scenarios, thus failing to provide a comprehensive reference for the model's real-world application.

[0004] In summary, there is an urgent need for an evaluation method for the Vision Transformer classification model that is applicable to black-box models, supports cross-scenario transfer evaluation, has multi-dimensional comprehensive indicators, and has automated execution capabilities, in order to meet practical application needs and improve the scientificity and reliability of model selection and deployment. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose an evaluation method for the VisionTransformer classification model in vertical scenarios.

[0006] The technical solution of this invention is: An evaluation method for the Vision Transformer classification model in vertical scenarios, comprising the following steps: Step 1: For the vertical industry application scenario to be evaluated, select a representative benchmark dataset. Based on the specific vertical characteristics and requirements, select a standard dataset that covers the task of that vertical category as the evaluation benchmark. The selected representative benchmark datasets should have a sufficient sample size and clear annotations to truly reflect the classification task characteristics of the target vertical category, ensuring the accuracy and practicality of the evaluation results. For example, datasets for identifying benign and malignant brain tumors in the medical field, defect detection datasets in the industrial field, and gesture recognition datasets in the field of intelligent interaction can all serve as typical representatives of specific vertical categories. The selection of the datasets should be combined with the actual application requirements and task difficulty to ensure that the evaluation method has good applicability and generalization ability. Step 2: Based on the black-box target ViT model to be evaluated, construct a lightweight proxy model with a known structure. Input the general vision dataset ImageNet-1K into the black-box target ViT model, collect the inference output of the black-box target ViT model as training sample labels, and use knowledge distillation technology to train the proxy model. The surrogate model is designed with known structural features and has a computational complexity significantly lower than the black-box target ViT model, in order to facilitate efficient transfer learning and performance analysis in the future. Step 3: Based on the surrogate model designed in Step 2, perform transfer learning fine-tuning training on the benchmark dataset in Step 1, and dynamically record the training process information. Set the maximum number of iterations and the expected accuracy threshold as training termination conditions, and stop training when either condition is met. During the training process, record in detail the number of training stops and the accuracy on the validation set on the vertical task for subsequent comprehensive performance evaluation. Step 4: Calculate multi-dimensional generality indicators based on the fine-tuned surrogate model in Step 3. These indicators comprehensively reflect the training efficiency and classification accuracy of the surrogate model in multiple vertical tasks. The indicators integrate the number of training stops and accuracy information for each task to quantify the generalization ability and adaptability of the surrogate model, serving as a key reference for evaluating the performance of the target ViT model. Step 5: Based on the generality indicators in Step 4, conduct a horizontal performance comparison of different target ViT models. By comparing the generality indicator values ​​of each target ViT model, select the target model with higher indicator values ​​and better performance. This supports model selection and deployment decisions in vertical scenarios and improves the scientificity and reliability of model application.

[0007] In step 2, knowledge distillation includes two methods: soft distillation and hard distillation. When the target ViT model outputs a class probability distribution, soft distillation is used to train the surrogate model by learning the class probabilities output by the target model as soft labels. When the target ViT model outputs class labels, hard distillation is used to train the surrogate model by learning the class labels output by the target model as hard labels; In step 2, the surrogate model refers to a classification model with a known structure and significantly lower computational complexity than the target ViT model, which facilitates subsequent transfer training and metric calculation. In step 3, the transfer learning fine-tuning training includes setting the maximum number of iterations MaxEpoch and the expected accuracy ExpectedAcc. When the number of training iterations reaches MaxEpoch or the validation set accuracy reaches ExpectedAcc, training is stopped, and the number of iterations StopEpoch and the final accuracy are recorded. In step 4, the calculation method for the generality index is as follows:

[0008] Where N is the number of tasks; It is the round number when the model stops on the i-th task; This is the maximum number of iteration rounds set on the i-th task; It is the accuracy achieved by the model on the i-th task; It is the maximum expected accuracy set on the i-th task; In step 5, by comparing the generality index (GI) values ​​of different target ViT models, the model with the higher GI value is selected as the target ViT model with better performance.

[0009] Compared with existing technologies, the present invention has the following advantages: 1. In black-box model evaluation, this invention breaks through the limitation of dependence on the target ViT model structure and reasoning process, and constructs a lightweight proxy model through knowledge distillation, which effectively solves the problem that existing methods cannot evaluate models with unknown structures.

[0010] 2. In cross-vertical task adaptation, this invention, through surrogate model transfer learning and multi-task generality index design, can comprehensively measure the generalization ability and adaptability of the target model, significantly improving the objectivity and practical value of the evaluation results.

[0011] 3. In the horizontal comparison of models, this invention enables quantitative comparison of the performance of different ViT models based on the general index GI value, supports scientific and reasonable model selection and deployment decisions in vertical scenarios, and has good application and promotion value. Attached Figure Description

[0012] Figure 1 This is a flowchart of the evaluation method of the present invention. Detailed Implementation

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0014] Example like Figure 1 As shown in the figure, this embodiment provides a method for automated performance evaluation of the Vision Transformer (ViT) classification model in medical imaging scenarios. The specific process is as follows: Step 1: Select a benchmark dataset for medical imaging. For the medical scenario of classifying benign and malignant brain tumors, the publicly available BraTS 2021 (Brain Tumor Segmentation Challenge) dataset is selected as the evaluation benchmark dataset. This dataset contains MRI images (T1, T1ce, T2, FLAIR) and tumor region annotation information. In this embodiment, it is transformed into a binary classification task (benign / malignant).

[0015] Step 2: Construct the surrogate model and perform knowledge distillation. With the black-box target model to be evaluated as ViT-B / 16 (Vision Transformer Base), a lightweight surrogate model, ResNet-18, is distilled using the ImageNet-1K dataset. Its parameter count is only 1 / 5 of the target ViT model, significantly reducing computational complexity.

[0016] Knowledge distillation method selection: Soft distillation: When the target ViT model outputs a class probability distribution, the surrogate model is trained by learning soft labels, and the loss function is a weighted sum of cross-entropy loss and KL divergence loss.

[0017] in, Predict the output for the surrogate model; For actual category labels; Represents cross-entropy loss; Indicates KL divergence; The logit output of the target ViT model; For the logit output of the proxy model; For the Softmax function; As a weighting factor; Temperature coefficient; Hard distillation: If only the final classification labels of the black-box model can be obtained, then the class labels are directly used as hard labels to train the surrogate model, and the loss function is:

[0018] in: Predict the output for the surrogate model; For actual category labels; In this embodiment, we assume that the target model cannot output a class probability distribution, and therefore hard distillation is employed.

[0019] Step 3, Transfer Learning and Fine-tuning. The distilled surrogate model ResNet-18 is fine-tuned using transfer learning based on the BraTS dataset.

[0020] Parameter settings: Maximum number of iterations MaxEpoch = 100, expected accuracy ExpectedAcc = 0.95; Stopping condition: Training stops when the validation set accuracy is ≥0.95 or after 100 epochs. Recorded information: Number of training stops (StopEpoch), final validation set accuracy.

[0021] For example, in the experiment, the surrogate model achieved a validation set accuracy of 0.953 in the 72nd round, satisfying the stopping condition.

[0022] Step 4, calculate the generalizability index (GI):

[0023] Step 5: Horizontal comparison of different target ViT models. In this embodiment, three typical structures—ViT-B / 16, ViT-L / 16, and DeiT-S—were selected as comparison objects. Experimental conditions were kept consistent, and the experimental results for each model are shown in the table below:

[0024] As shown in the table, although ViT-L / 16 achieved the highest accuracy on the validation set (0.962), its high convergence requirement resulted in a low GI value of only 0.511. The DeiT-S model demonstrated superior convergence efficiency but suffered from insufficient accuracy, ultimately achieving a GI value of 0.548. Considering both efficiency and accuracy, ViT-B / 16 demonstrated the best performance in terms of GI (0.554). Therefore, in this medical imaging task, ViT-B / 16 is recommended as the optimal target model for clinical deployment and application.

[0025] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An evaluation method for the Vision Transformer classification model in vertical scenarios, characterized by... The method includes the following steps: Step 1: For the vertical industry application scenario to be evaluated, select a representative benchmark dataset. Based on the specific vertical characteristics and requirements, select a standard dataset that covers the task of that vertical category as the evaluation benchmark. Step 2: Based on the black-box target ViT model to be evaluated, construct a lightweight proxy model with a known structure. Input the general vision dataset ImageNet-1K into the black-box target ViT model, collect the inference output of the black-box target ViT model as training sample labels, and use knowledge distillation technology to train the proxy model. Step 3: Based on the surrogate model designed in Step 2, perform transfer learning fine-tuning training on the benchmark dataset in Step 1, and dynamically record training process information. Set the maximum number of iterations and the expected accuracy threshold as training termination conditions, and stop training when either condition is met. During the training process, record in detail the number of training termination rounds and the validation set accuracy on the vertical task. Step 4: Calculate multi-dimensional generality indicators based on the fine-tuned surrogate model in Step 3. These indicators comprehensively reflect the training efficiency and classification accuracy of the surrogate model in multiple vertical tasks. The indicators integrate the number of training stops and accuracy information for each task to quantify the generalization ability and adaptability of the surrogate model. Step 5: Based on the generality indicators in Step 4, conduct a horizontal performance comparison of different target ViT models. By comparing the generality indicator values ​​corresponding to each target ViT model, select the target model with higher indicator values ​​and better performance to support model selection and deployment decisions in vertical scenarios.

2. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 1, characterized in that: In step 1, the selected representative benchmark dataset should have a sufficient sample size and clear labeling to truly reflect the classification task characteristics of the target vertical category, ensuring the accuracy and practicality of the evaluation results.

3. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 2, characterized in that: The benchmark datasets are a dataset for identifying benign and malignant brain tumors in the medical field, a dataset for defect detection in the industrial field, and a dataset for gesture recognition in the field of intelligent interaction.

4. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 1, characterized in that: In step 2, the surrogate model is designed with known structure and computational complexity significantly lower than the black-box target ViT model, so as to facilitate efficient transfer learning and performance analysis in the future.

5. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 1, characterized in that: In step 2, knowledge distillation includes two methods: soft distillation and hard distillation. When the target ViT model outputs a class probability distribution, soft distillation is used to train the surrogate model by learning the class probabilities output by the target model as soft labels. When the target ViT model outputs class labels, hard distillation is used to train the surrogate model by learning the class labels output by the target model as hard labels.

6. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 1, characterized in that: In step 2, the surrogate model refers to a classification model with a known structure and significantly lower computational complexity than the target ViT model, which facilitates subsequent transfer training and metric calculation.

7. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 1, characterized in that: In step 3, the transfer learning fine-tuning training includes setting the maximum number of iterations MaxEpoch and the expected accuracy ExpectedAcc. When the number of training iterations reaches MaxEpoch or the validation set accuracy reaches ExpectedAcc, training is stopped, and the number of iterations StopEpoch and the final accuracy are recorded.

8. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 1, characterized in that: In step 4, the calculation method for the generality index is as follows: Where N is the number of tasks; It is the round number when the model stops on the i-th task; This is the maximum number of iteration rounds set on the i-th task; It is the accuracy achieved by the model on the i-th task; It is the maximum expected accuracy set on the i-th task.

9. The evaluation method for the Vision Transformer classification model in vertical scenarios according to claim 2, characterized in that: In step 5, by comparing the generality index (GI) values ​​of different target ViT models, the model with the higher GI value is selected as the target ViT model with better performance.

Images