Information processing apparatus, information processing method, and program

Abstract

To suppress an increase in inference time when test time adaptation is used in inference using a model of a neural network.SOLUTION: An information processing device includes a path selection unit that determines, on the basis of input data, which of an initial model in which parameter update is not performed and an update model in which parameter update is performed is to be used, and an inference unit that performs inference processing on the input data by using the initial model or the update model on the basis of a determination result in the path selection unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to machine learning technology using neural networks. [Background technology]

[0002] In recent years, machine learning technology, particularly convolutional neural networks (CNNs), has significantly improved the accuracy of techniques such as identifying and detecting objects in images and segmenting areas. Technologies that use these machine learning technologies to promote the automation of visual inspection processes in various business processes are attracting attention.

[0003] When promoting the automation of visual inspection processes by processing captured images, it is desirable for CNN image processing to be in line with the intuition of the humans who originally perform the visual inspection. For example, it is considered desirable for CNN image processing to produce consistent processing results that are not affected by changes in the way the image is captured or the lighting environment.

[0004] However, image processing models based on machine learning techniques such as CNN are vulnerable to changes known as domain shifts, such as changes in the appearance of images or lighting conditions, as described above, and are known to significantly degrade processing accuracy.

[0005] This vulnerability of CNNs to domain shifts implies that CNNs must be used in the same environment for training and inference, which can pose a major practical challenge. Therefore, a method called domain adaptation is being actively researched to enable CNN models to perform image processing appropriately even when domain shifts occur.

[0006] The most commonly used method for domain adaptation in image processing using CNN models is unsupervised domain adaptation.Unsupervised domain adaptation assumes the existence of supervised data (source domain data) in an environment where no domain shift has occurred and unsupervised data (target domain data) in which a domain shift has occurred during CNN training, and aims to adapt the CNN model to the domain shift using the unsupervised target domain data.

[0007] Unsupervised domain adaptation is currently being researched as a promising method to combat domain shift, but it has the limitation that it assumes the existence of target domain data, so the environment at the time of inference must be known in advance.

[0008] Since CNNs are used in a variety of applications and environments, it is not realistic to grasp all the environments at the time of inference in advance. To alleviate this constraint, a method called test-time adaptation (Non-Patent Document 1, Non-Patent Document 2) has been attracting attention in recent years.

[0009] In test-time adaptation, domain adaptation is performed during inference using inference data acquired in the inference environment. This eliminates the need to collect target domain data before inference and enables domain adaptation in any environment. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Y. Sun et al., "Test-Time Training with Self-Supervisionfor Generalization under Distribution Shifts", 2020. [Non-patent document 2] S. Goyal et al., "Test-Time Adaptation via Conjugate Pseudo-labels", 2022. Summary of the Invention [Problem to be solved by the invention]

[0011] Test-time adaptation is currently attracting attention due to its wide range of applications, but because domain adaptation is performed during inference, the time required for inference of a CNN model is, in principle, longer than if adaptation were not performed.Since image processing tasks using CNNs often have strict constraints on the speed required for inference, the significant increase in inference time poses a major challenge to the practical application of test-time adaptation.

[0012] The present invention has been made in consideration of the above points, and aims to provide a technique for suppressing an increase in inference time when test-time adaptation is used in inference using a neural network model. [Means for solving the problem]

[0013] According to the disclosed technology, a path selection unit determines whether to use an initial model that does not perform parameter updating or an updated model that performs parameter updating based on input data; an inference unit that performs inference processing on the input data using the initial model or the updated model based on the determination result of the path selection unit; An information processing device is provided. [Effects of the Invention]

[0014] According to the disclosed technology, it is possible to suppress an increase in inference time when test-time adaptation is used in inference using a neural network model. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing device 100 according to a first embodiment. [Figure 2] 1 is a process flow for explaining the overall operation of the information processing device 100. [Figure 3]FIG. 10 is a diagram showing a processing flow of a path selection unit 110. [Figure 4] FIG. 10 is a diagram showing a processing flow of an inference unit 120. [Figure 5] FIG. 10 is a diagram showing a processing flow of the initial model processing unit 121. [Figure 6] FIG. 10 is a diagram showing a processing flow of an updated model processing unit 122. [Figure 7] FIG. 10 is a diagram illustrating a configuration of an inference unit 120 in an information processing device 100 according to a second embodiment. [Figure 8] 10 is a processing flow of an inference unit 120 in the second embodiment. [Figure 9] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0017] (Outline of the embodiment) As mentioned above, test-time adaptation is currently attracting attention due to its wide range of applications, but because domain adaptation is performed during inference, the time required for inference of a CNN model will, in principle, be longer than if adaptation were not performed.

[0018] The technology according to this embodiment improves the inference speed of test-time adaptation of CNN. In this embodiment, a determination module (a path selection unit, described later) is used to determine which inference images require test-time adaptation and which do not, and test-time adaptation is performed as much as possible only on inference images in which a domain shift that would affect accuracy has occurred. This prevents unnecessary increases in inference time and is expected to enable high-speed, highly accurate test-time adaptation even when a domain shift occurs.

[0019] Below, a first embodiment and a second embodiment will be described as examples of specific device configurations and device operations. Note that in the embodiments, the input data to be processed (inference target) is an image, but the technology according to the present embodiments can also be applied to input data other than images. Also, in the embodiments, the model used for inference processing is a neural network model, but the technology according to the present embodiments can also be applied to models other than neural network models.

[0020] (First embodiment) First, the first embodiment will be described.

[0021] <Device configuration example> 1 shows an example of the configuration of an information processing device 100 in the first embodiment. The information processing device 100 includes a route selection unit 110, an inference unit 120, an update unit 130, and a model parameter DB 140. Note that the model parameter DB 140 may be a component external to the information processing device 100.

[0022] The inference unit 120 includes an initial model processing unit 121, an updated model processing unit 122, and an output unit 123. The model parameter DB 140 includes a model parameter DB 140A and a model parameter DB 140B.

[0023] Note that the model parameter DB 140A and the model parameter DB 140B indicate that the data in the model parameter DB 140 is divided. For ease of understanding, the model parameter DB 140A and the model parameter DB 140B are depicted outside the model parameter DB 140 in Fig. 1. However, the model parameter DB 140A and the model parameter DB 140B may be provided outside the model parameter DB 140.

[0024] <Operation overview> The overall operation of the information processing device 100 will be described with reference to the flowchart of FIG.

[0025] In S1 (step 1), an input image, which is an image to be inferred, is sent to the path selection unit 110. In S2, the path selection unit 110 determines a path to use in test-time adaptation based on the characteristics of the input image. Specifically, the path selection unit 110 determines whether to use an initial model or an updated model for inference.

[0026] In S3, the inference unit 120 performs inference processing on the input image based on the determination result by the path selection unit 110. More specifically, if it is determined that an initial model is to be used for inference, the initial model processing unit 121 acquires model parameters from the model parameter DB 140A that stores the model parameters of the initial model, performs inference processing based on the model parameters, and sends the inference result to the output unit 123.

[0027] If it is determined that an updated model should be used for inference, the updated model processing unit 122 acquires model parameters from the model parameter DB 140B that stores the model parameters of the updated model, performs inference based on the model parameters while performing test-time adaptation on the input image, and sends the inference results to the output unit 123.

[0028] In S4, the output unit 123 outputs the inference result. In S5, if there is a next input image to be inferred, the process moves to processing of that input image. When processing of all input images is completed, the process ends.

[0029] The processing of each part will be described in more detail below with reference to the processing flow of each part.

[0030] <Route Selection Unit 110> 3 shows a processing flow of the route selection unit 110. The operation of the route selection unit 110 will be described with reference to FIG.

[0031] In S101, the path selection unit 110 acquires and inputs an input image of an inference target.

[0032] In S102, the path selection unit 110 determines whether to perform processing using the initial model or the updated model based on the characteristics of the input image.

[0033] The path selection unit 110 makes this determination based on, for example, the frequency components of the input image. Specifically, for example, the input image is Fourier transformed, and the power of the high-frequency components obtained by the Fourier transform is compared with a threshold value to determine whether to process using the initial model or the updated model. In this way, by making a determination based on the frequency components, it is possible to select targets with high learning effectiveness quickly and accurately.

[0034] The determination method is not limited to the above method, and any method may be used as long as it allows for appropriate determination.

[0035] In S103, the path selection unit 110 sends the determination result obtained in S102 and the input image to the inference unit 120.

[0036] <Inference part 120> Next, the processing of the inference unit 120 will be described with reference to the processing flow of FIG.

[0037] In S201, the inference unit 120 receives the input image and the determination result for the path from the path determination unit 110. In S202, if the determination result is the initial model, the process proceeds to S203, and if the determination result is the updated model, the process proceeds to S204.

[0038] In S203, the input image is sent to the initial model processing unit 121. In S204, the input image is sent to the updated model processing unit 122.

[0039] <Initial model processing unit 121> Next, the processing of the initial model processing unit 121 will be described with reference to the processing flow of FIG.

[0040] In S301, the initial model processing unit 121 acquires model parameters from the model parameter DB 140 A. In S302, the initial model processing unit 121 acquires an input image to be inferred.

[0041] In S303, the initial model processing unit 121 performs inference on the input image based on the model parameters acquired from the model parameter DB 140A, and sends the result to the output unit 123. More specifically, the initial model processing unit 121 has a neural network model, sets the model parameters acquired from the model parameter DB 140A in the model, and performs inference using the model in which the model parameters are set. In other words, the initial model processing unit 121 inputs the image to be inferred into the model, and obtains the output from the model as the inference result.

[0042] <Update model processing unit 122> Next, the processing of the update model processing unit 122 will be described with reference to the processing flow of FIG.

[0043] In S401, the update model processing unit 122 acquires model parameters from the model parameter DB 140 B. In S402, the update model processing unit 122 acquires an input image to be inferred.

[0044] In S403, the update model processing unit 122 performs inference based on the model parameters while performing test-time adaptation on the input image. More specifically, the update model processing unit 122 has a neural network model, sets model parameters acquired from the model parameter DB 140B in the model, and performs inference using the model in which the model parameters are set. However, by performing test-time adaptation, the model parameters are updated (adapted) to perform better inference.

[0045] Regarding the test-time adaptation in this case, the update model processing unit 122 performs the test-time adaptation using, for example, a method based on self-supervised learning disclosed in Non-Patent Document 1. Note that the test-time adaptation method used by the update model processing unit 122 is not limited to a specific method, and any test-time adaptation method may be used as long as it appropriately improves inference accuracy.

[0046] In S404, the update model processing unit 122 stores the adapted model parameters that have been subjected to the test-time adaptation in the model parameter DB 140B via the update unit .

[0047] (Second embodiment) In some test-time adaptation methods, in order to prevent overfitting and an increase in computational cost, only some parameters within the model are adapted at test time, while other parameters remain unchanged. Among these, the most commonly adopted method is to adapt only the normalization process (normalization layer) within the model at test time.

[0048] Therefore, in the second embodiment, test-time adaptation is applied only to the normalization layer within the model. More specifically, path selection is performed for the normalization layer that performs normalization processing within the model.

[0049] The overall configuration of the information processing device 100 in the second embodiment is the same as that of the information processing device 100 in the first embodiment (FIG. 1), and includes a path selection unit 110, an inference unit 120, an update unit 130, and a model parameter DB 140. As in the first embodiment, the model parameter DB 140 may be an external component of the information processing device 100. The operation of the path selection unit 110 is also the same as that in the first embodiment.

[0050] In the second embodiment, the configurations of the inference unit 120 and the model parameter DB 140 differ from those in the first embodiment.

[0051] Fig. 7 shows a detailed configuration of the inference unit 120 and the model parameter DB 140 in the second embodiment. Note that Fig. 7 shows an example in which a model parameter DB 140A and a model parameter DB 140B are provided.

[0052] In the second embodiment, it is assumed that an inference result is output after N stages of feature extraction are performed inside the inference unit 120, and that each feature extraction processing layer is followed by processing in a normalization layer.

[0053] 7, the inference unit 120 has a first feature extraction layer 125-1, ..., an N-th feature extraction layer 125-N, and an output layer 126. Each feature extraction layer has a similar configuration, and FIG. 7 shows the configuration of the first feature extraction layer 125-1.

[0054] 7, the first feature extraction layer 125-1 has a feature extraction processing layer 127, an initial normalization layer 128, and an update normalization layer 129. That is, the normalization layer has an initial normalization layer 128 that uses an initial model for normalization, and an update normalization layer 129 that has an update model to which test-time adaptation is applied (parameter updating is performed). The other feature extraction sections also have a configuration similar to that of the first feature extraction layer 125-1. Note that the initial normalization layer 128 may also be called the initial model, and the update normalization layer 129 may also be called the update model.

[0055] In the example of Figure 7, it is assumed that model parameters of the initial model of each layer are stored in model parameter DB140A, and in Figure 7, the fact that the model parameters of the initial model of each layer are stored in model parameter DB140A is shown as model parameter DB140A1, model parameter DB140A2, ...

[0056] In the example of Figure 7, it is assumed that model parameters of each layer to be updated by test-time adaptation are stored in model parameter DB140B, and in Figure 7, the fact that the model parameters of each layer to be updated are stored in model parameter DB140B is shown as model parameter DB140B2, model parameter DB140B4, ...

[0057] <Inference part 120> The processing of the inference unit 120 in the second embodiment will be described with reference to the processing flow in Fig. 8. Note that since the outline of the processing is similar for all feature extractions, the processing in the first feature extraction layer 125-1 will be described in detail below.

[0058] In S501, the first feature extraction layer 125-1 receives an input image and a determination result from the path determination unit 110.

[0059] In S502, the feature extraction processing layer 127 performs feature extraction processing to extract features from the input image. As shown in Fig. 7, the feature extraction processing layer 127 performs feature extraction processing using model parameters read out from the model parameter DB 140A1.

[0060] In S503, if the determination result obtained in S501 is "use the initial model", the process proceeds to S504, and if it is "use the updated model", the process proceeds to S506.

[0061] In S504, the inference unit 120 acquires normalization parameters from the model parameter DB 140A2 and sets the normalization parameters in the initial normalization layer 128.

[0062] In S505, the initial normalization layer 128 performs normalization processing on the extracted features output by the processing in S502 using normalization parameters. After the processing, the process proceeds to S509.

[0063] In S506, the inference unit 120 acquires normalization parameters from the model parameter DB 140B2 and sets the normalization parameters in the update normalization layer 129.

[0064] In S507, the inference unit 120 (or the update normalization layer 129) uses the extracted features output by the process of S502 to perform test-time adaptation of the normalization parameters in the update normalization layer 129. In the adaptation process of S507, for example, the parameters are updated using the "mean and variance" of the extracted features output by S502. However, this updating method is merely an example. Any updating may be performed as long as an appropriate update can be performed. Furthermore, the updating method may involve rewriting all of the parameters themselves, or may involve taking a moving average.

[0065] In S508, the update normalization layer 129 performs normalization processing on the extracted features based on the model parameters that have been adapted during testing (updated model parameters). After processing, the process proceeds to S509.

[0066] In S509, the first feature extraction layer 125-1 sends the extracted features that have been normalized in S505 or S508 to the second feature extraction layer 125-2. Note that although it is assumed that the above processing is performed continuously in the second feature extraction layer 125-2 as well, it is also possible to perform processing in which no test-time adaptation is performed at a specific layer, or to perform test-time adaptation regardless of the input.

[0067] (Other examples common to the first and second embodiments) In the first and second embodiments, the path selection unit 110 determines whether to use the initial model or the updated model based on, for example, the frequency components of the input image, but this is just one example.

[0068] The path selection unit 110 may determine whether to use the initial model or the updated model based on the validity of the output from the model in the inference unit 120 for the input image. The model used for the determination may be the initial model, the updated model, or a model that is not the target of path selection (for example, a feature extraction processing layer). In addition, the validity of the output may be measured by any method. In this way, by making a determination based on the validity of the output from the model, it is possible to select a target with a high learning effect with high accuracy.

[0069] Furthermore, in the first and second embodiments, the update model, which is a model that updates parameters by test-time adaptation, may be one that has undergone prior learning to become more resistant to noise. Since it is assumed that the data processed by the update model will contain a lot of noise, it is possible to improve accuracy by performing prior learning to become more resistant to noise.

[0070] (Example of hardware configuration) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0071] That is, the information processing device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0072] Fig. 9 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 9 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0073] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0074] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0075] (Effects of the Technology According to the Embodiments) According to the technology of this embodiment, it is possible to suppress an increase in inference time when test-time adaptation is used in inference using a neural network model. This is expected to prevent an unnecessary increase in inference time and enable high-speed and highly accurate test-time adaptation even when a domain shift occurs.

[0076] The following additional notes are provided regarding the above-described embodiments.

[0077] <Additional Notes> (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Determine whether to use an initial model that does not perform parameter updates or an updated model that performs parameter updates based on the input data; Based on the determination result, an inference process is performed on the input data using the initial model or the updated model. Information processing device. (Additional note 2) Both the initial model and the updated model are models for normalization processing. Item 1. An information processing device according to item 1. (Additional note 3) The processor determines whether to use the initial model or the updated model based on the frequency content of the input data. Item 1. An information processing device according to item 1. (Additional note 4) The processor determines whether to use the initial model or the updated model based on the validity of an output from a model in the inference unit with respect to the input data. Item 1. An information processing device according to item 1. (Additional note 5) The updated model is a model that has been trained in advance to be more resistant to noise. Item 1. An information processing device according to item 1. (Additional note 6) An information processing method executed by an information processing device, a path selection step for determining whether to use an initial model without parameter updates or an updated model with parameter updates based on input data; an inference step of performing inference processing on the input data using the initial model or the updated model based on the determination result in the path selection step; An information processing method comprising: (Additional note 7) A non-transitory storage medium storing a program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 5.

[0078] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0079] 100 Information processing device 110 Route selection unit 120 Reasoning Department 121 Initial model processing section 122 Update model processing section 123 Output section 125-1 to 125-N 1st feature extraction layer to Nth feature extraction layer 126 Output Layer 127 Feature Extraction Processing Layer 128 Initial normalization layer 129 Update normalization layer 130 Update Department 140 Model Parameter DB 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. a path selection unit that determines whether to use an initial model that does not perform parameter updating or an updated model that performs parameter updating based on input data; an inference unit that performs inference processing on the input data using the initial model or the updated model based on the determination result of the path selection unit; An information processing device comprising:

2. Both the initial model and the updated model are models for normalization processing. The information processing device according to claim 1 .

3. The path selection unit determines whether to use the initial model or the updated model based on a frequency component of the input data. The information processing device according to claim 1 .

4. The path selection unit determines whether to use the initial model or the updated model based on the validity of an output from a model in the inference unit with respect to the input data. The information processing device according to claim 1 .

5. The updated model is a model that has been trained in advance to be more resistant to noise. The information processing device according to claim 1 .

6. An information processing method executed by an information processing device, a path selection step for determining whether to use an initial model without parameter updates or an updated model with parameter updates based on input data; an inference step of performing inference processing on the input data using the initial model or the updated model based on the determination result in the path selection step; An information processing method comprising:

7. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 5.

Images