IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

The image processing technique addresses catastrophic forgetting in CNNs by combining continuous learning with metric learning and centroid vector correction, ensuring high accuracy for both old and new classes through a dual neural network approach.

JP7582127B2Active Publication Date: 2024-11-13JVC KENWOOD CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2021140819
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-08-31
Publication Date
2024-11-13
Estimated Expiration
2041-08-31

AI Technical Summary

Technical Problem

Existing image processing techniques based on Convolutional Neural Networks (CNNs) suffer from catastrophic forgetting, where the accuracy for old tasks decreases as they learn new tasks, leading to insufficient reduction of fatal forgetting.

Method used

An image processing device and method that combines continuous learning with metric learning, using a basic neural network to maintain old classes and an additional neural network to learn new classes, with centroid vector correction to minimize forgetting, and a classification process that selects classes based on proximity to embedding vectors.

Benefits of technology

The approach reduces fatal forgetting by maintaining high accuracy for both old and new classes, improving overall classification performance by leveraging both networks' strengths and minimizing overfitting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582127000001
    Figure 0007582127000001
  • Figure 0007582127000002
    Figure 0007582127000002
  • Figure 0007582127000003
    Figure 0007582127000003
Patent Text Reader

Abstract

To provide an image processing technique based on machine learning which can reduce catastrophic forgetting.SOLUTION: A basic class selection unit 20 selects a basic class, with respect to input data, on the basis of an embedding vector outputted from a basic neural network having learned basic classes and centroid vectors of the basic classes. An incremental learning unit 50 uses an additional neural network having learned the basic classes to perform incremental learning of additional classes. An additional class selection unit 40 selects, with respect to input data, an additional class on the basis of an embedding vector outputted from the additional neural network having performed incremental learning, the basic classes, and centroid vectors of additional classes. A classification determination unit 80 classifies the input data into classes on the basis of the basic class selected by the basic class selection unit 20 and the additional class selected by the additional class selection unit 40.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image processing technique based on machine learning. [Background technology]

[0002] Humans can learn new knowledge through long-term experience and can retain old knowledge without forgetting it. On the other hand, the knowledge of a Convolutional Neural Network (CNN) depends on the dataset used for training, and in order to adapt to changes in the data distribution, the parameters of the CNN must be retrained for the entire dataset. As CNN learns new tasks, its estimation accuracy for old tasks decreases. Thus, when CNN performs continuous training, it is inevitable that it will suffer from catastrophic forgetting, in which the learning results of old tasks are forgotten while learning a new task.

[0003] Incremental learning or continual learning has been proposed as a method to avoid fatal forgetting. Incremental learning is a learning method in which, when a new task or new data occurs, the model is not trained from scratch, but is trained by improving the currently trained model. Regularization-based continuous learning is one method of continuous learning, which uses regularization loss for learning (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 145852 [Non-patent literature]

[0005] [Non-Patent Document 1] Thomas Mensink, Jakob Verbeek, Florent Perronnin, Gabriela Csurka, "Distance-Based Image Classification: Generalizing to new classes at near-zero cost", IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, 2013, 35 (11), pp.2624-2637. [Non-Patent Document 2] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, Joost van de Weijer, "Semantic Drift Compensation for Class-Incremental Learning", 2020 Computer Vision and Pattern Recognition, pp 6982-6991. [Non-Patent Document 3] Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, ​​Fei Wu, Xi Li, "MgSvF: Multi-Grained Slow vs. Fast Framework for Few-Shot Class-Incremental Learning", arXiv:2006.15524, 2021. Summary of the Invention [Problem to be solved by the invention]

[0006] The technique described in Patent Document 1 has a problem in that it is not possible to sufficiently reduce fatal forgetting.

[0007] The present invention has been made in view of the above circumstances, and has a purpose to provide an image processing technique based on machine learning that can reduce fatal forgetting. [Means for solving the problem]

[0008] In order to solve the above problem, an image processing device of one embodiment of the present invention includes a base class selection unit that selects a base class for input data based on an embedding vector output by a base neural network that has learned a base class and a centroid vector of the base class; a continuous learning unit that continuously learns an additional class using an additional neural network that has learned a base class; an additional class selection unit that selects an additional class for the input data based on the embedding vector output by the continuously trained additional neural network and the centroid vectors of the base class and the additional class; and a classification decision unit that classifies the input data based on the base class selected by the base class selection unit and the additional class selected by the additional class selection unit.

[0009] Another aspect of the present invention is an image processing method, which includes a base class selection step of selecting a base class for input data based on an embedding vector output by a base neural network that has learned a base class and a centroid vector of the base class, a continuous learning step of continuously learning an additional class using an additional neural network that has learned a base class, an additional class selection step of selecting an additional class for the input data based on an embedding vector output by the additional neural network that has been continuously trained and the centroid vectors of the base class and the additional class, and a classification decision step of classifying the input data based on the base class selected by the base class selection step and the additional class selected by the additional class selection step.

[0010] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0011] According to the present invention, it is possible to provide an image processing technique based on machine learning that can reduce fatal forgetting. [Brief description of the drawings]

[0012] [Figure 1] 1 is a configuration diagram of an image processing device according to an embodiment; [Diagram 2] 4 is a flowchart illustrating a continuous learning process by the image processing device of FIG. 1. [Diagram 3] 2 is a diagram for explaining the structure of a neural network model used in the basic neural network processing unit and the additional neural network processing unit of FIG. 1. FIG. [Figure 4] 4 is a flowchart illustrating a classification determination process by the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] 1 is a configuration diagram of an image processing device 100 according to an embodiment. The image processing device 100 includes a basic neural network processing unit 10, a basic class selection unit 20, an additional neural network processing unit 30, an additional class selection unit 40, a continuous learning unit 50, a center of gravity derivation unit 60, a center of gravity vector correction unit 70, and a classification determination unit 80.

[0014] In this embodiment, machine learning is performed by combining continuous learning and metric learning. Here, an example of an image is described as input data, but the input data is not limited to images. Metric learning is a method for learning an embedding space (feature space) that takes into account the relationship between images (see, for example, Non-Patent Document 1). Metric learning is used in various fields such as information retrieval, data classification, and image recognition. Continuous learning, which learns using regularization loss, can be combined with metric learning, which uses metric loss.

[0015] In this embodiment, class incremental learning, which is one type of continuous learning, is used (for example, see Non-Patent Document 2 and Non-Patent Document 3). Non-Patent Document 2 performs class incremental learning using one neural network. Non-Patent Document 3 performs class incremental learning using two neural networks with different learning rates, and performs classification in a combined feature space that combines the feature spaces of the two neural networks.

[0016] In this embodiment, the basic neural network that has learned the basic classes is not changed, and the additional neural network that has learned the basic classes and continues to learn the additional classes is updated. Classification (class selection) is performed on the input image using each of the basic neural network and the additional neural network, and the input image is classified into the class with the higher accuracy (closer distance).

[0017] 2 is a flowchart for explaining the continuous learning by the image processing device 100. The configuration and overall operation of the continuous learning will be described with reference to FIGS.

[0018] First, a neural network in which the base classes have been trained and a centroid vector of the base class derived using the neural network are obtained. The neural network in which the base classes have been trained may be obtained from a network, or may be trained using a dataset including the base classes. It is preferable that the neural network in which the base classes have been trained is not trained for class classification, but is trained by metric training (embedding training). The centroid vector of the base class may be obtained from a network, or may be derived by inputting an image of the base class into the trained neural network, finding the centroid for each class for the embedding vector output from the trained neural network, and deriving it as the centroid vector for each class. Here, the number of centroid vectors for each class is set to 1, but there may be more than one.

[0019] A neural network that has already learned a basic class is set in the basic neural network processing unit 10 and the additional neural network processing unit 30 (S10).

[0020] The centroid vectors of the basic classes derived using a neural network that has already learned the basic classes are set in the basic class selection unit 20 and the additional class selection unit 40 (S20). The basic class selection unit 20 and the additional class selection unit 40 each store the centroid vectors of the basic classes.

[0021] Next, the learning session i, which is continuous learning, is repeated N times (i=1, 2, . . . , N) (S30).

[0022] First, for each additional class included in the additional training data set, the additional neural network processing unit 30 inputs all images of a certain additional class into the additional neural network before the learning session i is performed, and derives embedding vectors of all images of the additional class. The centroid derivation unit 60 derives the centroid vector of the additional class from the embedding vectors of all images of the additional class (S40). The centroid vector of the additional class here is the centroid vector before learning. Note that the centroid vector of the additional class is derived for all additional classes.

[0023] Next, the continuous training unit 50 continues training the additional neural network using the additional training data set including the additional class as a training session i (S50).

[0024] Next, for each additional class included in the additional training data set, the additional neural network processing unit 30 inputs all images of a certain additional class into the additional neural network after the learning session i, and derives embedding vectors of all images of the additional class. The centroid derivation unit 60 derives the centroid vector of the additional class from the embedding vectors of all images of the additional class (S60). The centroid vector of the additional class here is the centroid vector after learning. Note that the centroid vector of the additional class is derived for all additional classes.

[0025] Next, the additional class selection unit 40 deletes the centroid vectors of the stored basic classes (S70). Here, the number of centroid vectors of the basic classes to be deleted is the number of additional classes added in learning session i. The centroid vectors of the basic classes to be deleted are the ones that are closest to the centroid vectors of the additional classes added in learning session i. After all the centroid vectors of the basic classes have been deleted, no more centroid vectors are deleted. This makes it possible to make the number of centroid vectors stored by the basic class selection unit 20 and the number of centroid vectors stored by the additional class selection unit 40 the same.

[0026] Next, the centroid vector correction unit 70 corrects the centroid vectors of the known classes stored by the additional class selection unit 40 (S80). The known classes include the basic classes and the additional classes of the learning session (i-1). There is no need to correct the additional classes of the learning session i. i is incremented by 1 (S90), and the process returns to step S30, and steps S40 to S80 are repeated until i=N. When i exceeds N, the process ends.

[0027] For correction of the centroid vectors of the learned (known) classes, the method described with reference to FIG. 3 in Non-Patent Document 2 is improved and used.

[0028] The center of gravity vector correction unit 70 corrects the center of gravity vector of the learned class based on the center of gravity vector of the class before the continued learning and the center of gravity vector of the class after the continued learning that are within a predetermined distance of the center of gravity vector of the learned class (known class). Specifically, the center of gravity vector correction unit 70 obtains the amount of movement of the center of gravity vector of the class after the continued learning from the center of gravity vector of the class before the continued learning, and calculates the average movement amount of these movements. The center of gravity vector correction unit 70 corrects the center of gravity vector of the learned class by adding the average movement amount to the center of gravity vector of the learned class.

[0029] In Non-Patent Document 2, correction is performed using an embedding vector before learning that is within a radius R of the centroid vector of a known class, but in this embodiment, correction is performed using both the centroid vector of the class before continued learning and the centroid vector of the class after continued learning. In calculating the average movement amount, using more centroid vectors makes it less susceptible to small fluctuations in each image, so in this embodiment, correction is performed using both the centroid vector of the class before continued learning that is within a predetermined distance of the centroid vector of the learned class and the centroid vector of the class after continued learning.

[0030] The configuration and operation of the continuous learning unit 50 will now be described in more detail.

[0031] The basic training dataset is a supervised dataset that contains a large number of basic classes (e.g., 100 to 1000 classes) and each class consists of a large number of images (e.g., 3000 images). The basic training dataset is assumed to be a sufficient amount of data for learning a general classification task alone.

[0032] In contrast, an additional training dataset is a supervised dataset that includes a small number of additional classes (e.g., about 2 to 10 classes), with each additional class consisting of a small number of images (e.g., about 1 to 5 images). Training data consisting of a set of three images, an anchor image belonging to a certain class, a positive image belonging to the same class as the anchor image, and a negative image belonging to a class different from the anchor image, is input to the neural network to be trained. Here, the number of minor classes is set to 2 because even if the class to be trained is 1, a class that is not to be trained as a negative image is necessary. Also, although a small number of images is used here, a large number of images may be used as long as it is a small class.

[0033] 3 is a diagram for explaining the structure of a neural network model used in the basic neural network processing unit 10 and the additional neural network processing unit 30. The neural network is a deep neural network that includes a convolutional layer and a pooling layer, but does not include a fully connected layer. It includes CONV-1 to CONV-5, which are convolutional layers of ResNet-18 shown in FIG. 3, followed by a global average pooling layer, and outputs a 512-dimensional embedding vector.

[0034] The continuous learning unit 50 calculates the overall loss L by adding the metric loss Lml and the regularization loss Lr as shown in the following equation, and learns the neural network so as to minimize the overall loss L. L = Σ(Lml + Lr) Here, Σ indicates that the sum is taken for the input image.

[0035] The triplet loss is used as the metric loss. The triplet loss Lml is calculated based on the embedding vector of the anchor image, the embedding vector of the positive image, and the embedding vector of the negative image as follows: Lml=dp-dn+α where dp is the Euclidean distance between the embedding vector of the anchor image and the embedding vector between the positive image, dn is the Euclidean distance between the embedding vector of the anchor image and the embedding vector between the negative image, and α is the offset.

[0036] The regularization loss Lr is an embedding vector loss Lrv for minimizing the difference between before and after a learning session of the embedding vector output when an image is input to a neural network, as shown in the following equation. Lrv = || V(i) - V(i-1) || Here, V(i) is the embedding vector output by the neural network in the learning session i. V(i-1) is the embedding vector output by the neural network in the learning session (i-1). ||·|| is the symbol that indicates the calculation of the Frobenius norm.

[0037] 4 is a flowchart for explaining the classification decision by the image processing device 100. The configuration and overall operation of the classification decision will be described with reference to FIGS.

[0038] The basic neural network processing unit 10 inputs an image to be classified to the basic neural network, and the additional neural network processing unit 30 inputs the image to be classified to the additional neural network that has been continuously trained (S100).

[0039] The basic neural network processing unit 10 supplies the embedding vector of the image to be classified output from the basic neural network to the basic class selection unit 20, and the additional neural network processing unit 30 supplies the embedding vector of the image to be classified output from the additional neural network to the additional class selection unit 40 (S110).

[0040] The base class selection unit 20 selects a base class based on the base embedding vector output by the base neural network (S120). Specifically, the base class having the centroid vector that is closest to the base embedding vector is selected.

[0041] The additional class selection unit 40 selects an additional class based on the additional embedding vector output by the additional neural network (S130). Specifically, the additional class having the centroid vector that is closest to the additional embedding vector is selected. Note that the additional class selection unit 40 does not select the base class even if the class having the centroid vector that is closest to the additional embedding vector is the base class.

[0042] The classification determination unit 80 compares the basic class selected by the basic class selection unit 20 with the additional class selected by the additional class selection unit 40, and determines the class with the closer distance between the centroid vector and the embedding vector as the class of the classification result of the image to be classified (S140). The distance between the centroid vector and the embedding vector may be treated as a probability as an inverse number, and the probability may be determined to be high or low, and the class with the higher probability may be determined as the class of the classification result. Here, if the distance between the centroid vector and the embedding vector is the same between the selected basic class and the selected additional class, the additional class is selected as the class of the classification result.

[0043] (Modification) A modified example of the additional class selection unit 40 and the classification determination unit 80 will be described. Only operations different from the embodiment will be described. The additional class selection unit 40 selects the centroid vector that is closest to the additional embedding vector, regardless of whether it is a base class or an additional class. Here, if the base class selected by the base class selection unit 20 and the base class selected by the additional class selection unit 40 are different, the classification determination unit 80 selects the base class selected by the base class selection unit 20 as the class of the classification result. Here, the reason why the classification determination unit 80 selects the base class selected by the base class selection unit 20 as the class of the classification result is that the basic neural network has learned with more data for the basic class. In other words, the classification determination unit 80 selects the classification result of the neural network that has learned with more data.

[0044] The various processes of the image processing device 100 described above can be realized not only as a device using hardware such as a CPU and a memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as a data broadcast of terrestrial or satellite digital broadcasting.

[0045] As described above, according to the image processing device 100 of this embodiment, the basic neural network does not forget the base class because it does not continue learning. Therefore, even if the learning session progresses, the basic neural network can classify the base class with a high probability. Since the basic neural network does not continue learning the additional class, the basic neural network cannot select the additional class, but the additional neural network can learn while taking into account the characteristics of both the basic class and the additional class by continuing to learn the additional class for the base class, and can select the additional class.

[0046] According to this embodiment, the classification result obtained by a basic neural network that does not forget the basic classes and the classification result obtained by an additional neural network that has continuously learned the additional classes are evaluated, and the classification result with the higher accuracy is selected, thereby making it possible to improve classification accuracy while reducing fatal forgetting.

[0047] Here, when the additional neural network learns only the additional class, the centroid vector of the additional class is likely to be overfitted because the number of data of the additional class is small. In addition, the correction of the centroid vector is also likely to be over-corrected. Therefore, in the learning of the additional neural network, by taking into account the embedding vector output by the basic neural network that has learned the basic class with a large number of data together with the additional class, the centroid vector of the additional class and the correction of the centroid vector are prevented from fluctuating greatly due to overfitting, and the overfitting of the centroid vector of the additional class and the correction of the centroid vector is significantly reduced.

[0048] Furthermore, by keeping the total number of classes, that is, the basic classes and the additional classes, constant in the additional class selection unit 40, the embedding spaces of the basic neural network and the additional neural network can be kept at approximately the same density, and the distances in the embedding spaces in the basic class selection unit 20 and the additional class selection unit 40 can be handled at approximately the same level. This makes it possible to prevent bias in class selection between the basic class selection unit 20 and the additional class selection unit 40.

[0049] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0050] 10 basic neural network processing unit, 20 basic class selection unit, 30 additional neural network processing unit, 40 additional class selection unit, 50 continuous learning unit, 60 centroid derivation unit, 70 centroid vector correction unit, 80 classification determination unit, 100 image processing device.

Claims

1. a base class selection unit that selects a base class for input data based on an embedding vector output by a base neural network that has learned base classes and a centroid vector of the base class; a continuous learning unit that continuously learns an additional class using an additional neural network that has already learned a basic class; an additional class selection unit that selects an additional class based on an embedding vector output by the additional neural network that has been continuously trained and on centroid vectors of a base class and an additional class for the input data; an image processing device comprising: a classification determination unit that classifies the input data based on the basic class selected by the basic class selection unit and the additional class selected by the additional class selection unit.

2. a centroid deriving unit that derives a centroid vector from the embedding vector output by the additional neural network; 2. The image processing device according to claim 1, further comprising a center of gravity correction unit that corrects the center of gravity vector of a known class before the continued learning based on the center of gravity vector before the continued learning and the center of gravity vector after the continued learning derived by the center of gravity derivation unit.

3. 2 . The image processing apparatus according to claim 1 , wherein the additional class selection unit deletes the centroid vectors of the basic classes by the number of classes to be added during continuous learning.

4. A base class selection step of selecting a base class for input data based on an embedding vector output by a base neural network that has learned base classes and a centroid vector of the base class; a continuous learning step of continuously learning additional classes using an additional neural network that has already learned the base classes; an additional class selection step of selecting an additional class for the input data based on an embedding vector output by the additional neural network that has been continuously trained and on centroid vectors of a base class and an additional class; a classification decision step of classifying the input data based on the basic class selected in the basic class selection step and the additional class selected in the additional class selection step.

5. A base class selection step of selecting a base class for input data based on an embedding vector output by a base neural network that has learned base classes and a centroid vector of the base class; a continuous learning step of continuously learning additional classes using an additional neural network that has already learned the base classes; an additional class selection step of selecting an additional class for the input data based on an embedding vector output by the additional neural network that has been continuously trained and on centroid vectors of a base class and an additional class; a classification determination step of classifying the input data based on the basic class selected in the basic class selection step and the additional class selected in the additional class selection step.

Citation Information

Patent Citations

  • Classifying digital images in few-shot tasks based on neural networks trained using manifold mixup regularization and self-supervision

    US20210124993A1

  • Neural network learning device, neural network learning method and storage medium storing program

    WO2017145852A1