Adaptation device, adaptation method, and adaptation program

Abstract

This adaptation device regards a classification model and an image enhancement model as one model (image classification model), and applies the TTA method to update the model. For example, the adaptation device converts a target image into an enhanced image using the image enhancement model. The adaptation device also enters the enhanced image and the target image into the classification model. The adaptation device then compares the uncertainties (e.g., confidences) of the model outputs, selects the output with lower uncertainty (e.g., higher confidence), and calculates the loss. The adaptation device updates the parameters of the image enhancement model and the classification model on the basis of the calculated loss.

Description

Adaptation device, adaptation method, and adaptation program

[0001] The present invention relates to an adaptation device, an adaptation method, and an adaptation program for improving the accuracy of a model in test-time adaptation.

[0002] Deep Neural Networks (DNNs), which are conventionally used for image classification, can experience significant degradation in classification accuracy when the input data distribution shifts due to various factors such as weather and noise. For example, as shown in Figure 1, when noise is added to the input data (image data) to a Classification Network, the input data may not be classified correctly.

[0003] Test-time adaptation (TTA) is a technique that can maintain the classification accuracy of a model even under distribution shifts. TTA is a technique that simultaneously performs inference on unlabeled target data (e.g., data from the model's operational environment) and model updating. TTA can maintain classification accuracy even under distribution shifts.

[0004] Dequan Wang, Evan Shelhamery, Shaoteng Liu, Bruno Olshausen, Trevor Darrell, "TENT:FULLY TEST-TIME ADAPTATION BY ENTROPY MINIMIZATION", ICLR'21

[0005] However, TTA has the problem that updating a model using uncertain samples can lead to accumulated errors, which can cause the model to collapse or become catastrophically oblivious. Therefore, the present invention aims to improve the accuracy of a model even when uncertain samples are used when updating a model using TTA.

[0006] In order to solve the above-mentioned problems, the present invention is characterized by comprising a data accepting unit that accepts input of an image to be classified; an image enhancing unit that performs enhancement processing on the image to be classified using an image enhancement model that performs enhancement processing on the input image; a classification unit that classifies the image to be classified and the image that has been enhanced using the classification model that classifies the input image; a selection unit that selects a classification result of an image that has lower output uncertainty from the classification model between the image to be classified and the image that has been enhanced using the image enhancement model; and a model updating unit that calculates a TTA (Test-time Adaptation) loss using the classification result of the selected image, and updates parameters of the image enhancement model and the classification model based on the calculated loss.

[0007] The present invention allows for improved model accuracy even when using uncertain samples.

[0008] FIG. 1 illustrates an example of image classification under a distribution shift. FIG. 2 is a diagram for explaining an overview of an adaptation device. FIG. 3 illustrates an example of an enhanced image, an original image, and example confidence values ​​for the enhanced image and the original image. FIG. 4 illustrates an example configuration of an adaptation device. FIG. 5 is a flowchart illustrating an example of a processing procedure performed by the adaptation device. FIG. 6 illustrates pseudocode showing the processing performed by the adaptation device. FIG. 7 illustrates the error rate for the CCTA Standard Task. FIG. 8 illustrates the error rate for the CCTA Diverse Task. FIG. 9 illustrates the error rate for the CCTA Gradual Task. FIG. 10 illustrates the error rate for the PTTA Task. FIG. 11 illustrates the error rates when TECA is used for BN Adapt, Tent, EATA, and CoTTA. FIG. 12 illustrates the error rates when only the image enhancement model, only the classification model, and both models are updated for BN Adapt, Tent, EATA, and CoTTA. Fig. 13 shows the error rates for BN Adapt, Tent, EATA, and CoTTA when SL (Switching Logits), SPUS (Synchronizing Parameter Updating Speed), and FBNS (Freezing BN Statistics) used in TECA are removed. Fig. 14 shows the error rates when the uncertainty metric used in SL (Switching Logits) is changed from confidence to entropy. Fig. 15 shows an example configuration of a computer that executes an adaptation program.

[0009] Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings. The present invention is not limited to this embodiment. In the following description, the adaptive device of this embodiment will be described taking as an example a case where the uncertainty of image classification is evaluated using confidence, but the present invention is not limited to this. For example, the adaptive device may evaluate the uncertainty of image classification using an entropy value.

[0010] [Overview] First, an overview of the adaptation device will be explained using Figure 2. The adaptation device uses an image classification model (fΘ ) is adapted to the operational environment of the image classification model. Θ ) is a model that classifies input images. An image classification model is a classification model (f θ cls ) and an image enhancement model (f φ enh The adaptation device treats the classification model and the image enhancement model as a single model, applies the TTA method, and updates the model.

[0011] The process executed by the adaptive device will be outlined below, taking as an example a case where the target image (image to be classified) is a corrupted image.

[0012] First, the adaptive device converts the corrupted image into an enhanced image using an image enhancement model, and then inputs the enhanced image and the original image (target image) of the enhanced image into a classification model.

[0013] Next, the adaptive device obtains the evaluation values ​​of the uncertainty (e.g., confidence values) of the enhanced image and the original image in the classification model, selects the output (classification result) with the lower uncertainty (e.g., the higher confidence value), calculates a loss using the selected output, and updates the image enhancement model and classification model using the calculated loss.

[0014] For example, the adaptive device may use the gradient ∇ Θ Based on this, the gradient (∇ φ ) and the gradient (∇ θ Here, the adaptive device determines n: the number of data to be batch-processed by the image classification model (batch size), and n enh : The number of samples used to update the image enhancement model, and the gradient of the image enhancement model (∇ φ ) to n / n enh ∇ φ (details below).

[0015] Furthermore, to update a classification model, the adaptive device updates the Batch Normalization (BN) statistics of the classification model, but does not update (freeze) the BN statistics of the image enhancement model, because classification accuracy may be reduced if an image output from an image enhancement model updated based on the BN statistics is classified using the classification model (details will be described later).

[0016] [Image Enhancement Model] The image enhancement model uses, for example, URIE. The adaptation device trains the pre-trained image enhancement model to learn an image transformation that minimizes loss (for example, cross-entropy).

[0017] [Switching Logits (SL)] It is also known that enhancing an image using an image enhancement model does not necessarily improve the image uncertainty. For example, as shown in Figure 3, when comparing the confidence scores of an enhanced image and the original image, the original image may have a higher confidence score.

[0018] Therefore, the adaptive device selects the output of the classification model for the image with higher confidence (lower uncertainty) between the enhanced image generated by the image enhancement model and the original image as the output to be used to update the image classification model (see equation (1) below).

[0019]

[0020] [Synchronizing Parameter Updating Speed ​​(SPUS)] When updating an image classification model, if output for the original image is selected instead of output for the enhanced image, there is no backward path to the image enhancement model. As a result, the image enhancement model will not be trained. This results in a difference between the update speed of the parameters of the classification model and the image enhancement model, which may have a negative impact on the classification accuracy of the image classification model.

[0021] Therefore, the adaptation device rescales the gradient used to update the parameters of the image enhancement model in order to match the update speed of the parameters of the classification model with the update speed of the parameters of the image enhancement model.

[0022] For example, the smaller the ratio of the batch size used to update the parameters of the image enhancement model to the batch size used to update the parameters of the classification model, the larger the gradient used to update the parameters of the image enhancement model. For example, the adaptation device rescales the gradient of the image enhancement model based on the following equation (2):

[0023]

[0024] [Freezing BN Statistics (FBNS)] Furthermore, the authors have experimentally found that using BN Adapt (model updating using BN statistics) to update the image enhancement model results in a deterioration in accuracy (see the error rate when FBNS (Freezing BN Statistics) is removed in Figure 13, which will be described later). Therefore, the adaptive device updates the BN statistics of the classification model but does not update (freeze) the BN statistics of the image enhancement model.

[0025] [Configuration Example] Next, a configuration example of the adaptation device 10 will be described with reference to Fig. 4. The adaptation device 10 includes an input / output unit 11, a storage unit 12, and a control unit 13.

[0026] The input / output unit 11 is an interface that controls the input and output of various data. For example, the input / output unit 11 receives input of image data to be classified by an image classification model.

[0027] The storage unit 12 stores data, programs, etc. that are referenced when the control unit 13 executes various processes. The storage unit 12 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0028] For example, the storage unit 12 stores parameters of an image classification model (parameters of an image enhancement model and parameters of a classification model). The image enhancement model enhances an input image. The classification model classifies the input image. The image enhancement model and the classification model are assumed to have been trained in advance using training data.

[0029] The control unit 13 is responsible for overall control of the adaptation device 10. The functions of the control unit 13 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the storage unit 12.

[0030] The control unit 13 includes, for example, a data receiving unit 131 , an image enhancing unit 132 , a classifying unit 133 , a selecting unit 134 , and a model updating unit 135 .

[0031] The data receiving unit 131 receives input of an image to be classified by the image classification model. This image is, for example, an image in the operating environment of the image classification model. The image enhancement unit 132 inputs the image received by the data receiving unit 131 into the image enhancement model, thereby performing enhancement processing on the image.

[0032] The classification unit 133 classifies the image that has been enhanced by the image enhancement unit 132 and the original image using a classification model.

[0033] The selection unit 134 obtains an evaluation value (e.g., a confidence value) of the uncertainty of the output of each of the image that has been enhanced by the image enhancement model and the original image in the classification model, and selects the output for the image with a lower evaluation value of uncertainty (e.g., a higher confidence value) (see the above-mentioned formula (1)).

[0034] The model update unit 135 calculates the loss of the TTA using the output selected by the selection unit 134, and updates the parameters of the image enhancement model and the classification model based on the calculated loss.

[0035] For example, the model update unit 135 may use a batch size n used to update the parameters of the image enhancement model relative to a batch size n used to update the parameters of the classification model. enhThe smaller the ratio of , the larger the gradient used to update the parameters of the image enhancement model (see equation (2) above).

[0036] The model update unit 135 updates the parameters of the classification model using BN statistics, but freezes the BN statistics of the image enhancement model, i.e., the model update unit 135 does not update the parameters of the image enhancement model using BN statistics.

[0037] According to the adaptive device 10, when updating an image classification model by TTA, the accuracy of the image classification model can be improved even when uncertain samples are used.

[0038] [Example of Processing Procedure] Next, an example of the processing procedure executed by the adaptation device 10 will be described with reference to Fig. 5. First, the data accepting unit 131 of the adaptation device 10 accepts input of an image (an image to be classified) (S1). Next, the image enhancing unit 132 transforms the image accepted in S1 using an image enhancement model (S2). Thereafter, the classification unit 133 inputs the image after transformation in S2 and the original image (the image input in S1) into the classification model (S3).

[0039] After S3, the selection unit 134 obtains the confidence of the transformed image and the confidence of the original image in the classification model, and selects the output (classification result) with the higher confidence (S4).The model update unit 135 then calculates the loss of the output selected in S4 (S5), and updates the parameters of the image enhancement model and the classification model based on the calculated loss (S6).

[0040] By performing the above process, the adaptation device 10 can improve the accuracy of the image classification model even when uncertain samples are used when updating the image classification model using TTA. The process performed by the adaptation device 10 can be represented in pseudocode as shown in Figure 6, for example.

[0041] [Evaluation Experiment] The results of an evaluation experiment of the image classification model updated by the adaptation device 10 are shown below.

[0042] [Dataset] The dataset used for evaluation was ImageNet-C, with corrupted data at level 5 out of 5 corruption levels. 5,000 samples were extracted from each corrupted dataset using RobustBench.

[0043] [Tasks] The tasks performed by the image classification model were as follows: Continual TTA task (standard, diverse, gradual) Standard: After evaluation of one damage, evaluate the next damage without resetting the model. Diverse: Vary the order of damage and evaluate the average 10 times. Gradual: The order of damage is the same as standard, and evaluation is performed by gradually changing the damage level (1 → 5 → 1).

[0044] - Practical TTA Task (PTTA) Evaluation on correlated non-iid streaming data simulating real-world applications.

[0045] [TTA methods] ・Source: Classification model only, no model update. ・Input Adapt: ​​Classification model + URIE, no model update. ・BN Adapt: ​​Updates BN statistics only. ・Tent: Updates BN layer by entropy minimization. ・EATA: A method that extends Tent using active sample selection and EWC. ・CoTTA: A method that is robust in continuously changing environments using a self-training framework, augmentation-averaged pseudo-labels, and stochastic restoration. ・RoTTA: A method that is stable even in non-iid environments using a memory bank.

[0046] [TTA] The details of the TTA implementation are as follows: SGD (Stochastic Gradient Descent), momentum 0.9, batch size 64 CoTTA & RoTTA: lr (learning rate) = 0.01, Tent & CoTTA: lr = 0.00025

[0047] [URIE] The image enhancement model used was URIE. URIE was trained using AugMix + DeepAugment, a data augmentation method that improves robustness. URIE was also trained to minimize the loss of the classification model. ResNet-50 was used as the classification model, with the exception of some experiments.

[0048] [Evaluation Results] The evaluation results of the image classification model updated using the model updating method (TECA) executed by the adaptive device 10 are described below. As shown in Figure 7, applying TECA to the Continual TTA standard task reduced the error rate of each TTA method. In particular, CoTTA with TECA achieved the best error rate.

[0049] Furthermore, as shown in Figure 8, the error rate of each TTA method also decreased when TECA was applied to the Continual TTA diverse task. Here too, CoTTA w / TECA achieved the best error rate.

[0050] Furthermore, as shown in Figure 9, applying TECA also reduced the error rate of each TTA method except for Tent in the Continual TTA gradual task. Here too, CoTTA w / TECA achieved the best error rate.

[0051] Furthermore, as shown in Figure 10, the average error rate of each TTA method also decreased for the Practical TTA task when TECA was applied.

[0052] Furthermore, Figure 11 shows the error rates when using TECA for BN Adapt, Tent, EATA, and CoTTA. The model used was ResNet (R18, 34, 50, 101, 152). As shown in Figure 11, it was confirmed that combining TECA reduces the error rate more than increasing the model size. This confirms that TECA is a method with good model training efficiency.

[0053] We also compared the error rates for each of BN Adapt, Tent, EATA, and CoTTA for three patterns: updating only the image enhancement model parameter φ, updating only the classification model parameter θ, and updating both the image enhancement model parameter and the classification model parameter Θ (see Figure 12). The numbers in parentheses indicate the error rate after parameter update - the error rate without parameter update.

[0054] As shown in Figure 12, we confirmed that the pattern of updating both the image enhancement model parameters and the classification model parameters resulted in the lowest error rate. Furthermore, updating only the image enhancement model parameters resulted in a decrease in accuracy, except for EATA. This indicates that BN Adapt has a negative impact on updating the image enhancement model parameters.

[0055] Furthermore, the error rates for BN Adapt, Tent, EATA, and CoTTA when SL (Switching Logits), SPUS (Synchronizing Parameter Updating Speed), and FBNS (Freezing BN Statistics) used in TECA are removed are shown in Figure 13. The values ​​in parentheses indicate the error rate when SL, SPUS, and FBNS are removed - the error rate when SL, SPUS, and FBNS are not removed.

[0056] As shown in Figure 13, it was confirmed that the error rate was high when any of SL, SPUS, and FBNS was missing. This confirmed that the accuracy of the model improved when all of SL, SPUS, and FBNS were included.

[0057] The error rate when the SL selection criterion (uncertainty) in TECA is changed from confidence to entropy is shown in Figure 14. The value in parentheses indicates (error rate when entropy is used - error rate when confidence is used).

[0058] As shown in Figure 14, in all cases of BN Adapt, Tent, EATA, and CoTTA, the error rate was higher when entropy was used as the SL selection criterion (uncertainty). In other words, it was found that the error rate was lower when confidence was used as the SL selection criterion than when entropy was used.

[0059] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0060] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0061] [Program] The adaptation device 10 can be implemented by installing a program (adaptation program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the adaptation device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0062] 15 is a diagram showing an example of a computer that executes an adaptation program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0063] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0064] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the adaptation device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the adaptation device 10 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0065] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0066] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0067] REFERENCE SIGNS LIST 10 Adaptation device 11 Input / output unit 12 Storage unit 13 Control unit 131 Data reception unit 132 Image enhancement unit 133 Classification unit 134 Selection unit 135 Model update unit

Claims

1. An adaptation device comprising: a data accepting unit that accepts input of an image to be classified; an image enhancement unit that performs enhancement processing on the image to be classified using an image enhancement model that performs enhancement processing on the input image; a classification unit that classifies the image to be classified and the image on which the enhancement processing has been performed using a classification model that classifies the input image; a selection unit that selects a classification result of an image with lower output uncertainty by the classification model from among the image to be classified and the image on which the enhancement processing has been performed using the image enhancement model; and a model updating unit that calculates a TTA (Test-time Adaptation) loss using the classification result of the selected image, and updates parameters of the image enhancement model and the classification model based on the calculated loss.

2. The adaptive device described in claim 1, characterized in that the model update unit increases the gradient used to update the parameters of the image enhancement model as the ratio of the batch size used to update the parameters of the image enhancement model to the batch size used to update the parameters of the classification model decreases.

3. The adaptive device according to claim 1, characterized in that the model update unit updates parameters of the classification model using Batch Normalization (BN) statistics, but does not update parameters of the image enhancement model using BN statistics.

4. The adaptive device according to claim 1, characterized in that the selection unit uses a value indicating the confidence of the image to be classified by the classification model as a value indicating the uncertainty of the output by the classification model.

5. An adaptation method executed by an adaptation device, comprising: a step of accepting input of an image to be classified; a step of performing enhancement processing of the image to be classified using an image enhancement model that performs enhancement processing of the input image; a step of classifying the image to be classified and the image on which the enhancement processing has been performed using a classification model that classifies the input image; a step of selecting a classification result of an image having a lower output uncertainty by the classification model from among the image to be classified and the image on which the enhancement processing has been performed by the image enhancement model; and a step of calculating a TTA (Test-time Adaptation) loss using the classification result of the selected image, and updating parameters of the image enhancement model and the classification model based on the calculated loss.

6. An adaptation program for causing a computer to execute the steps of: accepting input of an image to be classified; enhancing the image to be classified using an image enhancement model that enhances the input image; classifying the image to be classified and the image that has been enhanced using a classification model that classifies the input image; selecting a classification result of the image to be classified and the image that has been enhanced using the image enhancement model, which has a lower uncertainty in output by the classification model; calculating a TTA (Test-time Adaptation) loss using the classification result of the selected image, and updating parameters of the image enhancement model and the classification model based on the calculated loss.

Images