Machine learning device, machine learning method, and machine learning program

Abstract

To provide a machine learning technology with which it is possible to reduce fatal oblivion.SOLUTION: Embedment conversion units 20, 50 convert embedment outputted by a neural network into a first embedding matrix and a second embedding matrix. A regularization loss calculation unit 80 calculates a regularization loss on the basis of a first embedding matrix derived by converting the embedment outputted by the neural network to be trained, a first embedding matrix derived by converting the embedment outputted by a trained neural network, a second embedding matrix derived by converting the embedment outputted by the neural network to be trained, and a second embedding matrix derived by converting the embedment outputted by the trained neural network. A metric loss calculation unit 70 calculates the metric loss of the neural network to be trained. An optimization unit 90 optimizes the neural network to be trained on the basis of the metric loss and the regularization loss.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to machine learning techniques. [Background technology]

[0002] Humans can learn new knowledge through long-term experience and can retain old knowledge without forgetting it. On the other hand, the knowledge of a Convolutional Neural Network (CNN) depends on the dataset used for training, and in order to adapt to changes in the data distribution, the parameters of the CNN must be retrained for the entire dataset. As CNN learns new tasks, its estimation accuracy for old tasks decreases. Thus, when CNN performs continuous training, it is inevitable that it will suffer from catastrophic forgetting, in which the learning results of old tasks are forgotten while learning a new task.

[0003] Incremental learning or continual learning has been proposed as a method to avoid fatal forgetting. Incremental learning is a learning method in which, when a new task or new data occurs, the model is not trained from scratch, but is trained by improving the currently trained model. Regularization-based continuous learning is one method of continuous learning, which uses regularization loss for learning (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 145852 [Non-patent literature]

[0005] [Non-Patent Document 1] Thomas Mensink, Jakob Verbeek, Florent Perronnin, Gabriela Csurka, "Distance-Based Image Classification: Generalizing to new classes at near-zero cost", IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, 2013, 35 (11), pp.2624-2637. Summary of the Invention [Problem to be solved by the invention]

[0006] The technique described in Patent Document 1 has a problem in that it is not possible to sufficiently reduce fatal forgetting.

[0007] The present invention has been made in view of the above circumstances, and has an object to provide a machine learning technique capable of reducing fatal forgetting. [Means for solving the problem]

[0008] In order to solve the above problem, a machine learning device according to an embodiment of the present invention is a machine learning device using a neural network that continuously learns, and includes an embedding conversion unit that converts embeddings output by a neural network into a first embedding matrix and a second embedding matrix, a first embedding matrix obtained by converting embeddings output by a neural network to be trained, a first embedding matrix obtained by converting embeddings output by a trained neural network, a second embedding matrix obtained by converting embeddings output by the neural network to be trained, and a regularization loss calculation unit that calculates a regularization loss based on the second embedding matrix obtained by converting embeddings output by the trained neural network, a metric loss calculation unit that calculates a metric loss of the neural network to be trained, and an optimization unit that optimizes the neural network to be trained based on the metric loss and the regularization loss. When the learning of the neural network to be trained is completed, the neural network to be trained is set as a trained neural network.

[0009] Another aspect of the present invention is a machine learning method. This method is a machine learning method using a neural network that continuously learns, and includes an embedding conversion step of converting an embedding output by a neural network into a first embedding matrix and a second embedding matrix, a regularization loss calculation step of calculating a regularization loss based on the first embedding matrix obtained by converting an embedding output by a neural network to be trained, the first embedding matrix obtained by converting an embedding output by a trained neural network, the second embedding matrix obtained by converting an embedding output by the neural network to be trained, and the second embedding matrix obtained by converting an embedding output by the trained neural network, a metric loss calculation step of calculating a metric loss of the neural network to be trained, and an optimization step of optimizing the neural network to be trained based on the metric loss and the regularization loss. When the learning of the neural network to be trained is completed, the neural network to be trained is set as a trained neural network.

[0010] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0011] According to the present invention, it is possible to provide a machine learning technique capable of reducing fatal forgetting. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram of a machine learning device according to a first embodiment. [Diagram 2] 1 is a flowchart illustrating an overall flow of learning by a machine learning device. [Diagram 3] FIG. 2 is a diagram illustrating the structure of a neural network model to be learned in a learning session i. [Figure 4] 13 is a flowchart illustrating a continuous learning operation. [Diagram 5] 5(a) to 5(f) are diagrams illustrating examples of transformations applied to a 7×7 embedded map. [Figure 6] FIG. 13 is a configuration diagram of an image processing device according to a second embodiment. [Figure 7] 7 is a flowchart illustrating an overall flow of learning by the image processing device of FIG. 6. [Figure 8A] FIG. 7 is a diagram illustrating a configuration of the outline embedded learning unit of FIG. 6. [Figure 8B] 7 is a diagram illustrating a configuration of a detailed embedding learning unit in FIG. 6. [Figure 9A] 7 is a flowchart illustrating the operation of continuous learning by the outline embedded learning unit of FIG. 6. [Figure 9B] 7 is a flowchart illustrating the operation of the continuous learning by the detailed embedded learning unit in FIG. 6. [Figure 10] 7 is a flowchart illustrating the operation of classifying by the classifying unit in FIG. 6. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (First embodiment) 1 is a configuration diagram of a machine learning device 100 according to a first embodiment. The machine learning device 100 includes a training neural network processing unit 10, an embedding conversion unit 20, a representative vector storage unit 30, a trained neural network processing unit 40, an embedding conversion unit 50, a loss calculation unit 60, a metric loss calculation unit 70, a regularization loss calculation unit 80, and an optimization unit 90.

[0014] In this embodiment, machine learning is performed by combining continuous learning and metric learning. Here, an example of an image is described as input data, but the input data is not limited to images. Metric learning is a method for learning an embedding space (feature space) that takes into account the relationship between images (see, for example, Non-Patent Document 1). Metric learning is used in various fields such as information retrieval, data classification, and image recognition. Continuous learning, which learns using regularization loss, can be combined with metric learning, which uses metric loss.

[0015] 2 is a flowchart illustrating the overall flow of learning by the machine learning device 100. The configuration and operation of the machine learning will be described with reference to FIGS.

[0016] First, the training neural network processing unit 10 trains basic classes using a basic training data set and updates the training neural network (S10). A training session i is set to 0. This is also called an initial session.

[0017] Next, the embedding conversion unit 20 derives representative vectors of basic classes using the basic training data set based on the neural network to be trained, and stores the representative vectors in the representative vector storage unit 30 (S20).

[0018] Next, the training neural network processing unit 10 saves the training neural network as a trained neural network, and provides it to the trained neural network processing unit 40 (S30).

[0019] Next, the learning session i of the continuous learning is repeated N times (i=1, 2, ..., N) (S40). The trained neural network saved in the learning session (i-1) is used as the neural network to be trained in the learning session i.

[0020] The training neural network processing unit 10 continues to train additional classes using the additional training data set, and updates the training neural network (S50).

[0021] Next, the embedding conversion unit 20 derives representative vectors of additional classes using the additional training data set based on the neural network to be trained, and stores the representative vectors in the representative vector storage unit 30 (S60).

[0022] Next, the training neural network processing unit 10 saves the training neural network as a trained neural network, and provides it to the trained neural network processing unit 40 (S70).

[0023] i is incremented by 1 (S80), the process returns to step S40, and steps S50 to S70 are repeated until i=N. When i exceeds N, the process ends.

[0024] The configuration and operation of continuous learning will be explained in more detail.

[0025] The basic training dataset is a supervised dataset that contains a large number of basic classes (e.g., 100 to 1000 classes) and each class consists of a large number of images (e.g., 3000 images). The basic training dataset is assumed to be a sufficient amount of data for learning a general classification task alone.

[0026] In contrast, an additional training dataset is a supervised dataset that includes a small number of additional classes (e.g., about 2 to 10 classes), with each additional class consisting of a small number of images (e.g., about 1 to 5 images). Training data consisting of a set of three images, an anchor image belonging to a certain class, a positive image belonging to the same class as the anchor image, and a negative image belonging to a class different from the anchor image, is input to the neural network to be trained. Here, the number of minor classes is set to 2 because even if the class to be trained is 1, a class that is not to be trained as a negative image is necessary. Also, although a small number of images is used here, a large number of images may be used as long as it is a small class.

[0027] Figure 3 is a diagram explaining the structure of the neural network model to be trained in training session i. The neural network to be trained is a deep neural network including a convolutional layer and a pooling layer. It includes CONV-1 to CONV-5, which are convolutional layers of ResNet-18 shown in Figure 3, and outputs a 7x7x512-dimensional embedding map (hereinafter, simply referred to as "embedding").

[0028] The trained neural network has the same configuration as the neural network to be trained. The trained neural network is a neural network that has already completed training in the training session (i-1).

[0029] The embedding conversion unit 20 performs smoothing by global average pooling for each 7×7 embedding map output by the neural network to be trained, and outputs a 512-dimensional embedding vector. The embedding conversion unit 50 performs smoothing by global average pooling for each 7×7 embedding map output by the trained neural network, and outputs a 512-dimensional embedding vector. The embedding conversion unit 20 and the embedding conversion unit 50 have the same configuration and perform the same operation, so they can be configured as a single unit.

[0030] The loss calculation unit 60 adds the metric loss Lml calculated by the metric loss calculation unit 70 and the regularization loss Lr calculated by the regularization loss calculation unit 80 to calculate the total loss L as shown in the following equation. L = Σ(Lml + Lr) Here, Σ indicates that the sum is taken for the input image.

[0031] The metric loss calculation unit 70 calculates the metric loss. Here, the triplet loss is used as the metric loss. The triplet loss Lml is calculated based on the embedding vector of the anchor image, the embedding vector of the positive image, and the embedding vector of the negative image by the following formula. Lml=dp-dn+α where dp is the Euclidean distance between the embedding vector of the anchor image and the embedding vector between the positive image, dn is the Euclidean distance between the embedding vector of the anchor image and the embedding vector between the negative image, and α is the offset.

[0032] The regularization loss calculation unit 80 calculates the regularization loss. The regularization loss Lr is the sum of the embedding loss Lre and the embedding vector loss Lrv. As shown in the following formula, the embedding loss Lre is the difference between before and after the learning session of the embedding output when an image is input to the neural network, and the embedding vector loss Lrv is the difference between before and after the learning session of the embedding vector output when an image is input to the neural network. Lr=Lre+Lrv Lre = ||E(i)-E(i-1)|| Lrv = || V(i) - V(i-1) || Here, E(i) is the embedding output by the neural network to be trained, and E(i-1) is the embedding output by the trained neural network. V(i) is the embedding vector output by the neural network to be trained, and V(i-1) is the embedding vector output by the trained neural network. ||·|| is the symbol that indicates the calculation of the Frobenius norm.

[0033] The regularization loss is used to maintain the classification results for the training data learned in the previous training session (i-1) rather than optimizing only for the additional training data learned in the training session i. The embedding loss Lre is a loss that takes into account the detailed characteristics of the image, and the embedding vector loss Lrv is a loss that takes into account the overview characteristics of the image.

[0034] The optimization unit 90 optimizes the weights of the neural network to be trained so as to minimize the total loss L. When the optimization is completed, the total loss L is reset.

[0035] The representative vectors of the classes are calculated as follows.

[0036] The training neural network processing unit 10 sequentially inputs images in a training data set to the training neural network to derive embeddings, and the embedding conversion unit 20 converts the embeddings into embedding vectors. The embedding conversion unit 20 calculates representative vectors for each class based on the embeddings or embedding vectors, and stores them in the representative vector storage unit 30.

[0037] The embedding conversion unit 20 calculates the average of the embedding vectors of all images belonging to the class and sets it as the representative vector of the class. The embedding conversion unit 20 associates the class with the calculated representative vector and stores it in the representative vector storage unit 30. The representative vector of the class may be the center of gravity of the embedding vectors of all images belonging to the class, the average of the embeddings of all images belonging to the class, or the center of gravity of the embeddings of all images belonging to the class. Here, the embedding vectors of all images belonging to the class are used to calculate the representative vector of the class, but for example, N embedding vectors may be randomly selected from all images belonging to the class and used. If the number of embedding vectors of all images is NA, N≦NA.

[0038] FIG. 4 is a flowchart illustrating the operation of the continuous learning.

[0039] The training target neural network processing unit 10 inputs training data consisting of a set of three images, an anchor image, a positive image, and a negative image, to the training target neural network (S100). The metric loss is calculated using the three images, the anchor image, the positive image, and the negative image. The anchor image is the image to be trained. The positive image and the negative image are not the training target. In other words, the anchor image is used to calculate the regularization loss and the representative vector. The positive image and the negative image are not used to calculate the regularization loss and the representative vector. The training data is randomly selected from the training dataset.

[0040] The training object neural network processing unit 10 supplies the embedding output by the training object neural network to the embedding conversion unit 20 and the regularization loss calculation unit 80 (S110).

[0041] The embedding conversion unit 20 smoothes the embedding output by the training neural network to calculate an embedding vector, and supplies the calculated embedding vector to the metric loss calculation unit 70 and the regularization loss calculation unit 80 (S120).

[0042] The metric loss calculation unit 70 calculates the metric loss based on the embedding vector supplied from the embedding conversion unit 20 (S130).

[0043] The trained neural network processing unit 40 inputs the training data to the trained neural network (S140).

[0044] The trained neural network processing unit 40 supplies the embedding output by the trained neural network to the embedding conversion unit 50 and the regularization loss calculation unit 80 (S150).

[0045] The embedding conversion unit 50 smoothes the embedding output by the trained neural network to calculate an embedding vector, and supplies the calculated embedding vector to the regularization loss calculation unit 80 (S160).

[0046] The regularization loss calculation unit 80 calculates an embedding loss based on the embeddings supplied from the training neural network processing unit 10 and the trained neural network processing unit 40, calculates an embedding vector loss based on the embedding vectors supplied from the embedding conversion unit 20 and the embedding conversion unit 50, and calculates the regularization loss by adding the embedding loss and the embedding vector loss (S170).

[0047] The loss calculation unit 60 calculates the overall loss by adding the metric loss and the regularization loss (S180).

[0048] The optimization unit 90 decides whether or not to optimize (S190). If not optimizing, the process returns to step S100, and if optimizing, the process proceeds to step S200. Optimization is performed on an epoch basis. An epoch is a unit in which all images in the training dataset are used as anchor images. If there are a large number of images, optimization may be performed on a batch basis. If not optimizing, the overall loss is added up, and if optimizing, learning is performed to minimize the sum of the overall loss, and the overall loss is reset when optimization is complete.

[0049] The optimization unit 90 optimizes the neural network to be trained based on the total sum of the overall losses (S200).

[0050] The optimization unit 90 determines whether to complete the learning (S210), and if the learning is to be continued, the process returns to step S100, and if the learning is to be completed, the process proceeds to step S220. When the learning of a predetermined number of epochs is completed, the learning is completed. The predetermined number of epochs is set before the training.

[0051] When the learning is completed, the training neural network processing unit 10 sets the training neural network as a trained neural network in the trained neural network processing unit 40 (S220).

[0052] As described above, according to the machine learning device 100 of the present embodiment, by using the embedding vector loss that considers the overview characteristics of the image together with the embedding loss that considers the detailed characteristics of the image as the regularization loss, it is possible to perform continuous learning by considering both the detailed characteristics of the image and the overview characteristics of the image. Furthermore, by using the detailed characteristics of the image together with the overview characteristics of the image, the overview characteristics are emphasized, and it is possible to suppress entanglement of embedding in the embedding space.

[0053] Furthermore, according to the machine learning device 100 of this embodiment, the representative vectors from the previous learning session are saved and used in the next learning session, making it possible to continue learning additional classes while retaining the classes learned in the previous session.

[0054] Some variations of this embodiment will be described below.

[0055] (Variation 1) When calculating the regularization loss Lr, only the embedding loss Lre is used without using the embedding vector loss Lrv.

[0056] By using an embedding loss that takes into account the detailed characteristics of the image as the regularization loss, it is possible to continue learning while taking into account the detailed characteristics of the image. In other words, it is possible to continue learning while taking into account the individual components of the 7x7 embedding.

[0057] (Variation 2) When calculating the regularization loss Lr, the embedding loss Lre and the embedding vector loss Lrv are weighted and added together. Lr=β*Lre+Lrv where β is a weighting parameter ranging from 0.0 to 1.0.

[0058] The weight parameter β can be used to control the degree to which the embedding space of the neural network eliminates the situation where classes are mixed.

[0059] (Variation 3) In the second modification, the initial value of the weighting parameter β is set large, and β is decreased as the number of learning sessions increases. β may be set to 0 in a given learning session.

[0060] Although detailed features are taken into account in the initial stage of learning, the embedding space generally becomes more entangled as the learning session progresses. Increasing the proportion of overview features can suppress this entanglement in the embedding space.

[0061] (Variation 4) Initially, the representative vectors of the classes may be used as embeddings output by the neural network, and then in a given training session, the embeddings may be switched to smoothed embedding vectors.

[0062] In the first half of the learning process, the representative vector that takes into account detailed characteristics is left using embedding, but in the second half of the learning process, the embedding space generally becomes more tangled, so the embedding space can be suppressed by switching to a representative vector that takes into account overview characteristics using embedding vectors. This switching can also reduce memory capacity.

[0063] (Variation 5) In the above embodiment, the embedding loss taking into account the detailed characteristics of the image is calculated using the embedding output by the neural network, and the embedding vector loss taking into account the general characteristics of the image is calculated using an embedding vector obtained by smoothing the embedding, but this may be generalized to calculate two types of losses using the following two embedding matrices. In this modification, the configurations and operations of the embedding conversion units 20 and 50, the metric loss calculation unit 70, and the regularization loss calculation unit 80 in the embodiment are different as follows.

[0064] The embedding transformation units 20, 50 apply two different transformations to the 7x7 embedding map, converting it into two embedding matrices.

[0065] Figures 5(a) to 5(f) are diagrams illustrating an example of a transformation applied to a 7 x 7 embedding map. The values ​​within each black box in Figures 5(a) to 5(f) are smoothed (averaged), and the smoothed parts become the columns of the embedding matrix.

[0066] Figure 5(a) shows global average pooling, which smooths all values ​​in the embedding map, which is equivalent to smoothing the 7x7 embedding to find the embedding vector, as in the above embodiment.

[0067] Fig. 5(b) shows horizontal average pooling, which smooths the horizontal components in the embedding map. Here, all horizontal components are smoothed and used, but only even or odd rows can be used. Fig. 5(c) shows vertical average pooling, which smooths the vertical components in the embedding map. Here, all vertical components are smoothed and used, but only even or odd columns can be used.

[0068] Figures 5(d) and 5(e) show vertical average pooling to smooth a given rectangular region in the embedding map. It is not necessary to use all values ​​in the embedding map as in Figure 5(d), and values ​​in the embedding map may be used in duplicate as in Figure 5(e). In Figure 5(d), each of the 3x3 rectangular regions in the upper left, lower left, upper right, and lower right is smoothed, and the central rows and columns are not used. In Figure 5(e), each of the 4x4 rectangular regions in the upper left, lower left, upper right, and lower right is smoothed, allowing overlaps in the central rows and columns. The size of each rectangular region is not limited to these.

[0069] Figure 5(f) shows concentric average pooling, which divides and smooths the embedding map based on the distance from the center of the embedding map. In Figure 5(f), 3x3, 5x5, and 7x7 rectangular regions are overlapped and smoothed according to the distance from the center. Here, 3x3, 5x5, and 7x7 rectangular regions are overlapped and smoothed, but they may be smoothed without overlapping. Also, only the 5x5 rectangular region in Figure 5(f) may be used. In this case, it corresponds to the case where the region of global average pooling in Figure 5(a) is reduced.

[0070] The number and shapes of regions to be smoothed are not limited to those exemplified in FIGS. 5(a) to 5(f).

[0071] For example, as a first conversion, the embedding conversion unit 20, 50 converts the embedding output by the neural network into a first embedding matrix by global average pooling in FIG. 5(a). The first embedding matrix is ​​a 512×1 matrix. As a second conversion, the embedding conversion unit 20, 50 converts the embedding output by the neural network into a second embedding matrix by four-region equal average pooling that smooths the inside of the black frame shown in FIG. 5(d). The second embedding matrix is ​​a 512×4 matrix.

[0072] Here, the global average pooling and the four-region equal average pooling are combined, but the combination is not limited to this. For example, the transformations in FIG. 5(a) to FIG. 5(f) may be freely combined. In addition, the embedding map itself may be included as an equivalent transformation as a combination target. In the first embodiment, the embedding loss Lre is calculated using the equivalent transformation as the first transformation with the first embedding matrix as the embedding itself, and the embedding vector loss Lrv is calculated using the global average pooling in FIG. 5(a) as the second transformation with the second embedding matrix as the embedding vector, and the sum of the embedding loss Lre and the embedding vector loss Lrv is calculated as the regularization loss Lr.

[0073] In calculating the metric loss, the metric loss calculation unit 70 uses, instead of the embedding vector in the embodiment, an embedding matrix with the fewer number of columns out of the first embedding matrix or the second embedding matrix obtained by converting the embedding output by the neural network to be trained.

[0074] The regularization loss calculation unit 80 determines the regularization loss Lr to be the sum of the first embedding matrix loss Lr1 and the second embedding matrix loss Lr2. Lr = Lr1 + Lr2 Lr1 = ||M1(i)-M1(i-1)|| Lr2 = ||M2(i)-M2(i-1)|| Here, M1(i) is the first embedding matrix converted from the embedding output by the neural network to be trained, M1(i-1) is the first embedding matrix converted from the embedding output by the trained neural network, M2(i) is the second embedding matrix converted from the embedding output by the neural network to be trained, and M2(i-1) is the second embedding matrix converted from the embedding output by the trained neural network.

[0075] The embedding conversion unit 20 uses the embedding matrix used by the metric loss calculation unit 70 instead of the embedding vector to obtain a representative vector of the class, and stores the representative vector in the representative vector storage unit 30. However, the metric loss calculation unit 70 may use an embedding vector as in the embodiment, and in that case, the representative vector obtained from the embedding vector is stored in the representative vector storage unit 30.

[0076] By allowing any combination of regularization losses, it is possible to suppress tangles in embedding in the embedding space according to the image characteristics to be classified by combining two different characteristics and calculating and optimizing the loss, without being limited to the combination of a detailed characteristic and an overview characteristic. Furthermore, it is also possible to calculate and optimize the loss by combining two or more different characteristics. For example, for the transforms shown in Figures 5(a) to 5(f), it is also possible to arbitrarily combine three or more transforms by changing the number of divisions, etc.

[0077] (Second embodiment) 6 is a configuration diagram of an image processing device 200 according to the second embodiment. The image processing device 200 includes an overview embedding learning unit 210, a detail embedding learning unit 220, a synthesis unit 230, and a classification unit 240.

[0078] Below, the configuration and operation of the image processing device 200 of the second embodiment will be described. However, descriptions of the configuration and operation common to the machine learning device 100 of the first embodiment will be omitted as appropriate, and only the configuration and operation different from the machine learning device 100 of the first embodiment will be described.

[0079] As in the first embodiment, in the second embodiment, machine learning is performed by combining continuous learning and metric learning.

[0080] Fig. 7 is a flowchart explaining the overall flow of learning by the image processing device 200. The configuration and operation of the machine learning will be explained with reference to Figs.

[0081] First, we use the basic training dataset to learn base classes and update the neural network (S300). Let learning session i be 0.

[0082] Next, representative vectors of the basic classes are derived and stored using the basic training data set based on the neural network (S310). The method of deriving the representative vectors will be described later.

[0083] This is referred to as learning session 0, also known as the basic learning session.

[0084] Next, the learning session i, which is continuous learning, is repeated N times (i=1, 2, ..., N) (S320). The trained neural network in the learning session (i-1) is used as the neural network to be trained in the learning session i.

[0085] In a learning session i (i≧1), the following processing is executed in the overview embedding learning unit 210 and the detail embedding learning unit 220.

[0086] An embedding or embedding vector is derived from the neural network before continued learning using the additional training data set and stored (S330). The overview embedding learning unit 210 derives and stores the embedding vector, and the detailed embedding learning unit 220 derives and stores the embedding. Here, the neural network before continued learning refers to the trained neural network in the learning session (i-1).

[0087] Next, the overview embedding training unit 210 and the detail embedding training unit 220 continue to train additional classes using additional training data sets to update the neural network (S340).

[0088] Next, the overview embedding learning unit 210 and the detail embedding learning unit 220 derive and store representative vectors of additional classes using the additional training data set based on the neural network (S350).

[0089] i is incremented by 1 (S360), the process returns to step S320, and steps S330 to S350 are repeated until i=N. When i exceeds N, the process ends.

[0090] The configuration and operation of continuous learning will be described in more detail. Learning session 0 is assumed to be completed, and learning session i (i≧1) of continuous learning will be described.

[0091] When a training data set is input, the overview embedding learning unit 210 learns overview embeddings of classes included in the training data set and outputs a representative vector for each class. When an image to be classified is input, the overview embedding learning unit 210 outputs an embedding vector.

[0092] When a training data set is input, the detail embedding learning unit 220 learns the detail embedding of the classes included in the training data set and outputs a representative vector for each class. When an image to be classified is input, the detail embedding learning unit 220 outputs an embedding vector.

[0093] The learning rate of the detail embedding learning unit 220 is set to be larger than the learning rate of the overview embedding learning unit 210. For example, the learning rate of the overview embedding learning unit 210 is set to 1e -6 , and the learning rate of the detailed embedding learning unit 220 is set to 1e -4 Set to.

[0094] The synthesis unit 230 operates only when an image to be classified is input. The synthesis unit 230 concatenates the embedding vector output by the overview embedding learning unit 210 and the embedding vector output by the detail embedding learning unit 220 to derive a composite embedding vector. The synthesis unit 230 also concatenates the representative vector output by the overview embedding learning unit 210 and the representative vector output by the detail embedding learning unit 220 for each class to derive a composite representative vector.

[0095] The classification unit 240 operates only when an image to be classified is input. The classification unit 240 performs class classification using a composite representative vector and a composite embedding vector. For class classification, a Nearest Class Mean (NCM) method is used. That is, the input image is classified into a class having a composite representative vector that has the shortest distance from the composite embedding vector. The distance is the Mahalanobis distance as shown below. d M (x,x')=(x-x') T M(x-x') Here, x is the composite embedding vector, x' is the composite representative vector, and M is the variance-covariance matrix. The variance-covariance matrix may be any positive semidefinite matrix.

[0096] FIG. 8A shows the configuration of general embedding learning unit 210, and FIG. 8B shows the configuration of detail embedding learning unit 220. As shown in FIG.

[0097] The memories 330a and 330b are connected to the overview embedding learning unit 210 and the detail embedding learning unit 220. Before the start of learning session 1, the representative vectors of the base classes derived in learning session 0 are stored in the memories 330a and 330b.

[0098] Before the neural networks 310a, 310b begin training in each training session, an additional training data set is used to derive embeddings and embedding vectors from the neural networks 310a, 310b before further training and store them in the memory units 330a, 330b.

[0099] The neural networks 310a and 310b are trained neural networks in the learning session i. The structure of the trained neural network model is the same as that in the first embodiment, as shown in Fig. 3. After the learning session 0 is completed, the trained neural network of the learning session 0 is copied to the neural network of the learning session 1.

[0100] The embedding conversion units 320a and 320b perform smoothing by global average pooling for each 7×7 embedding map output by the training target neural networks 310a and 310b, and output a 512-dimensional embedding vector. The embedding conversion units 320a and 320b perform smoothing by global average pooling for each 7×7 embedding map output by the neural networks 310a and 310b before continued learning, and output a 512-dimensional embedding vector.

[0101] The loss calculation units 340a and 340b add the metric loss Lml calculated by the metric loss calculation units 350a and 350b and the regularization loss Lr calculated by the regularization loss calculation units 360a and 360b to calculate the total loss L as shown in the following equation. L = Σ(Lml + Lr) Here, Σ indicates that the sum is taken for the input image.

[0102] The metric loss calculation units 350a and 350b calculate the metric loss. Here, the triplet loss is used as the metric loss. The triplet loss Lml is calculated based on the embedding vector of the anchor image, the embedding vector of the positive image, and the embedding vector of the negative image by the following formula. Lml=dp-dn+α where dp is the Euclidean distance between the embedding vector of the anchor image and the embedding vector between the positive image, dn is the Euclidean distance between the embedding vector of the anchor image and the embedding vector between the negative image, and α is the offset.

[0103] The regularization loss calculation units 360a and 360b calculate the regularization loss. The regularization loss Lr is the embedding loss Lre or the embedding vector loss Lrv. In the regularization loss calculation unit 360a of the overview embedding learning unit 210, the regularization loss Lr is the embedding vector loss Lrv. In the regularization loss calculation unit 360b of the detailed embedding learning unit 220, the regularization loss Lr is the embedding loss Lre.

[0104] As shown in the following equation, the embedding loss Lre is the difference between before and after a learning session of the embedding output when an image is input to a neural network, and the embedding vector loss Lrv is the difference between before and after a learning session of the embedding vector output when an image is input to a neural network. Lre = ||E(i)-E(i-1)|| Lrv = || V(i) - V(i-1) || Here, E(i) is the embedding output by the neural network to be trained, and E(i-1) is the embedding output by the neural network before further training. V(i) is the embedding vector output by the neural network to be trained, and V(i-1) is the embedding vector output by the neural network before further training. E(i-1) and V(i-1) are obtained from the storage units 330a and 330b.

[0105] The optimization units 380a and 380b optimize the weights of the neural network to be trained so as to minimize the total loss L. When the optimization is completed, the total loss L is reset.

[0106] The representative vectors of the classes are calculated as follows.

[0107] Images in the training data set are input sequentially to the training neural networks 310a and 310b to derive embeddings, and the embedding conversion units 320a and 320b convert the embeddings into embedding vectors. The representative vector derivation units 370a and 370b calculate representative vectors for each class based on the embeddings or embedding vectors, and supply them to the synthesis unit 230.

[0108] The representative vector derivation units 370a and 370b calculate the average of the embedding vectors of all images belonging to the class and set it as the representative vector of the class. The representative vector derivation units 370a and 370b associate the class with the calculated representative vector and store it in the storage units 330a and 330b. The representative vector of the class may be the center of gravity of the embedding vectors of all images belonging to the class, the average of the embeddings of all images belonging to the class, or the center of gravity of the embeddings of all images belonging to the class.

[0109] FIG. 9A is a flowchart illustrating the operation of continuous learning by summary embedding learning unit 210.

[0110] Training data consisting of a set of three images, an anchor image, a positive image, and a negative image, is input to the neural network 310a to be trained (S400).

[0111] The trained neural network 310a outputs the embeddings (S410).

[0112] The embedding conversion unit 320a smoothes the embeddings output by the training neural network 310a to calculate embedding vectors, and supplies the calculated embedding vectors to the metric loss calculation unit 350a and the regularization loss calculation unit 360a (S420). The embedding conversion unit 320a supplies the embedding vectors of the anchor image, positive image, and negative image to the metric loss calculation unit 350a, and supplies the embedding vector of the anchor image to the regularization loss calculation unit 360a.

[0113] The metric loss calculation unit 350a calculates a metric loss based on the embedding vector supplied from the embedding conversion unit 320a (S430).

[0114] The embedding conversion unit 320a obtains an embedding vector obtained by smoothing the embedding output by the neural network 310a before the continuous learning stored in S330, and supplies the obtained embedding vector to the regularization loss calculation unit 360a (S440).

[0115] The regularization loss calculation unit 360a calculates an embedding vector loss based on the embedding vector obtained by smoothing the embedding output by the training neural network 310a supplied from the embedding conversion unit 320a and the embedding vector obtained by smoothing the embedding output by the neural network 310a before continued training, and outputs the embedding vector loss as a regularization loss (S450).

[0116] The loss calculation unit 340a calculates the overall loss by adding the metric loss and the regularization loss (S460).

[0117] The optimization unit 380a decides whether or not to optimize (S470). If not, the process returns to step S400. If optimized, the process proceeds to step S480. Optimization is performed on an epoch basis, but if there are a large number of images, optimization may be performed on a batch basis. If not, the overall loss is added up, and if optimized, learning is performed to minimize the sum of the overall losses, and the overall loss is reset when optimization is complete.

[0118] The optimization unit 380a optimizes the neural network to be trained based on the total sum of the overall losses (S480).

[0119] The optimization unit 380a determines whether to complete the learning (S490), and if the learning is to be continued, the process returns to step S400, and if the learning is to be completed, the process ends. When the learning for a predetermined number of epochs is completed, the learning is completed.

[0120] FIG. 9B is a flowchart illustrating the operation of detailed embedded learning unit 220 in continuous learning.

[0121] Training data consisting of a set of three images, an anchor image, a positive image, and a negative image, is input to the neural network to be trained 310b (S500).

[0122] The training neural network 310b outputs the embeddings (S510). The embeddings of the anchor image, positive image, and negative image are provided to the embedding conversion unit 320b. The embeddings of the anchor image are provided to the regularization loss calculation unit 360b.

[0123] The embedding conversion unit 320b smoothes the embedding output by the training neural network 310b to calculate an embedding vector, and supplies the calculated embedding vector to the metric loss calculation unit 350b (S520).

[0124] The metric loss calculation unit 350b calculates a metric loss based on the embedding vector supplied from the embedding conversion unit 320b (S530).

[0125] The embedding conversion unit 320b supplies the embedding output by the neural network 310b before continued training, which was saved in S330, to the regularization loss calculation unit 360b (S540).

[0126] Regularization loss calculation unit 360b calculates an embedding loss based on the embedding output from training neural network 310b and neural network 310b before continued training, and outputs the embedding loss as a regularization loss (S550).

[0127] The loss calculation unit 340b calculates the overall loss by adding the metric loss and the regularization loss (S560).

[0128] The optimization unit 380b decides whether or not to optimize (S570). If not, the process returns to step S500. If optimized, the process proceeds to step S580. Optimization is performed on an epoch basis, but if there are a large number of images, optimization may be performed on a batch basis. If not, the overall loss is added up, and if optimized, learning is performed to minimize the sum of the overall losses, and the overall loss is reset when optimization is complete.

[0129] The optimization unit 380b optimizes the neural network to be trained based on the total sum of the overall losses (S580).

[0130] Optimization unit 380b determines whether to complete the learning (S590), and if the learning is to be continued, returns to step S500, and if the learning is to be completed, ends. When the learning of a predetermined number of epochs is completed, the learning is completed.

[0131] FIG. 10 is a flowchart illustrating the class classification operation performed by the classification unit 240 of the image processing device 200. As shown in FIG.

[0132] An image is input to the overview embedding training unit 210 and the detail embedding training unit 220 (S600).

[0133] The neural network 310a of the overview embedding learning unit 210 outputs the embedding (S610). The embedding conversion unit 320a of the overview embedding learning unit 210 smoothes the embedding output by the neural network 310a and outputs an embedding vector (S620).

[0134] Neural network 310b of detail embedding learning unit 220 outputs the embedding (S630). Embedding conversion unit 320b of detail embedding learning unit 220 smoothes the embedding output by neural network 310b and outputs an embedding vector (S640).

[0135] The synthesis unit 230 synthesizes the embedding vector output by the general embedding learning unit 210 and the embedding vector output by the detailed embedding learning unit 220 to derive a synthesized embedding vector (S650).

[0136] The synthesis unit 230 synthesizes the representative vector output by the general embedding learning unit 210 and the representative vector output by the detailed embedding learning unit 220 to derive a synthesized representative vector (S660).

[0137] The classification unit 240 predicts a class based on the composite embedding vector and the composite representative vector, and classifies the image (S670).

[0138] As described above, according to the image processing device 200 of this embodiment, by combining an embedding output by a neural network that learns based on an embedding loss that takes into account the detailed characteristics of the image and an embedding output by a neural network that learns based on an embedding vector loss that takes into account the conceptual characteristics of the image, it is possible to continue learning while preserving the conceptual characteristics of the image while preserving the detailed characteristics of the image.

[0139] Furthermore, according to the image processing device 200 of this embodiment, the representative vectors from the previous learning session are saved and used in the next learning session, making it possible to continue learning additional classes while retaining the classes learned in the previous session.

[0140] Some variations of this embodiment will be described below.

[0141] (Variation 1) A modification of the synthesis unit 230 will be described.

[0142] The synthesis unit 230 derives a synthesized embedding vector V by taking a weighted average of the embedding vector Vo output by the general embedding learning unit 210 and the embedding vector Vd output by the detail embedding learning unit 220 .

[0143] The synthesis unit 230 derives a synthesized representative vector U by taking a weighted average of the representative vector Uo output by the overview embedding learning unit 210 and the representative vector Ud output by the detail embedding learning unit 220 for each class.

[0144] V = γVo + (1-γ)Vd U = γUo + (1-γ)Ud where γ is a weighting parameter ranging from 0.0 to 1.0.

[0145] The weight parameter γ allows us to control the importance of detail features and concept features in the embedding space.

[0146] (Variation 2) In the first modification, the initial value of the weighting parameter γ is set small, and γ is increased as the number of learning sessions increases. γ may be set to 1 in a given learning session.

[0147] In the initial stage of learning, emphasis is placed on detailed features, but as the learning session progresses, the embedding space generally becomes more entangled. Increasing the proportion of conceptual features can reduce this entanglement in the embedding space.

[0148] (Variation 3) In the above embodiment, the embedding loss taking into account the detailed characteristics of the image is calculated using the embedding output by the neural network, and the embedding vector loss taking into account the general characteristics of the image is calculated using an embedding vector obtained by smoothing the embedding, but this may be generalized to calculate two types of losses using the following two embedding matrices. In this modification, the configurations and operations of the embedding conversion units 320a and 320b, the metric loss calculation units 350a and 350b, and the regularization loss calculation units 360a and 360b in the embodiment are different as follows.

[0149] The embedding conversion unit 320a of the general embedding training unit 210 and the embedding conversion unit 320b of the detailed embedding training unit 220 apply two different transformations to the 7×7 embedding map, converting it into two embedding matrices.

[0150] An example of the transformation applied to the 7×7 embedded map is as described in FIG. 5(a) to FIG. 5(f) in the same manner as in the fifth modification of the first embodiment.

[0151] For example, as a first conversion in the overview embedding learning unit 210, the embedding conversion unit 320a converts the embeddings output by the neural network into a first embedding matrix by global average pooling in FIG. 5(a). The first embedding matrix is ​​a 512×1 matrix. As a second conversion in the detailed embedding learning unit 220, the embedding conversion unit 320b converts the embeddings output by the neural network into a second embedding matrix by four-region equal average pooling that smooths the inside of the black frame shown in FIG. 5(d). The second embedding matrix is ​​a 512×4 matrix.

[0152] Here, global average pooling and four-region equal average pooling are combined, but the combination is not limited to this. For example, the transformations shown in FIG. 5(a) to FIG. 5(f) may be freely combined by changing the number of divisions so that the dimension of the first embedding matrix is ​​less than the dimension of the second embedding matrix. In addition, the embedding map itself may be included as an equivalent transformation as a combination target. In the second embodiment, the overview embedding learning unit 210 uses the global average pooling of FIG. 5(a) as the first transformation to calculate the embedding vector loss Lrv using the first embedding matrix as the embedding vector, and the detailed embedding learning unit 220 uses the equivalent transformation as the second transformation to calculate the embedding loss Lre using the second embedding matrix as the embedding itself.

[0153] Overview In calculating the metric loss, the metric loss calculation unit 350a of the embedding learning unit 210 uses a first embedding matrix obtained by converting the embedding output by the neural network to be trained, instead of the embedding vector in the embodiment.

[0154] In calculating the metric loss, the metric loss calculation unit 350b of the detailed embedding learning unit 220 uses a second embedding matrix obtained by converting the embedding output by the neural network to be trained, instead of the embedding vector in the embodiment.

[0155] The regularization loss calculation unit 360a of the overview embedding learning unit 210 sets the regularization loss Lr to a first embedding matrix loss Lr1.

[0156] Regularization loss calculation unit 360b of detailed embedding learning unit 220 sets the regularization loss Lr to a second embedding matrix loss Lr2.

[0157] Lr1 = ||M1(i)-M1(i-1)|| Lr2 = ||M2(i)-M2(i-1)|| Here, M1(i) is the first embedding matrix converted from the embedding output by the neural network being trained, M1(i-1) is the first embedding matrix converted from the embedding output by the trained neural network before continued training, M2(i) is the second embedding matrix converted from the embedding output by the neural network being trained, and M2(i-1) is the second embedding matrix converted from the embedding output by the neural network before continued training.

[0158] The representative vector derivation units 370a and 370b use the embedding matrix used in the metric loss calculation units 350a and 350b instead of the embedding vector to obtain representative vectors of the classes, and store them in the storage units 330a and 330b. However, the metric loss calculation units 350a and 350b may use embedding vectors as in the embodiment, and in that case, the representative vectors obtained from the embedding vectors are stored in the storage units 330a and 330b.

[0159] The combining unit 230 converts the embedding matrices output by the general embedding learning unit 210 and the detail embedding learning unit 220 into embedding vectors, and then combines the embedding vectors. The combining unit 230 converts the first embedding matrix output by the general embedding learning unit 210 into a first embedding vector, converts the second embedding matrix output by the detail embedding learning unit 220 into a second embedding vector, and combines the first embedding vector and the second embedding vector to derive a combined embedding vector.

[0160] By allowing any combination of regularization losses, it is possible to suppress tangles in embedding in the embedding space according to the image characteristics to be classified by combining two different characteristics and calculating and optimizing the loss, without being limited to the combination of a detailed characteristic and an overview characteristic. Furthermore, it is also possible to calculate and optimize the loss by combining two or more different characteristics. For example, for the transforms shown in Figures 5(a) to 5(f), it is also possible to arbitrarily combine three or more transforms by changing the number of divisions, etc.

[0161] The various processes of the machine learning device 100 and the image processing device 200 described above can be realized not only as devices using hardware such as a CPU and a memory, but also as firmware stored in a ROM (read-only memory) or a flash memory, or software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting of terrestrial or satellite digital broadcasting.

[0162] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0163] 10 learning target neural network processing unit, 20 embedding conversion unit, 30 representative vector memory unit, 40 trained neural network processing unit, 50 embedding conversion unit, 60 loss calculation unit, 70 metric loss calculation unit, 80 regularization loss calculation unit, 90 optimization unit, 100 machine learning device, 200 image processing device, 310a, 310b neural network, 320a, 320b embedding conversion unit, 330a, 330b memory unit, 340a, 340b loss calculation unit, 350a, 350b metric loss calculation unit, 360a, 360b regularization loss calculation unit, 370a, 370b representative vector derivation unit, 380a, 380b optimization unit.

Claims

1. an embedding conversion unit that converts the embedding output by the neural network into a first embedding matrix and a second embedding matrix; a regularization loss calculation unit that calculates a regularization loss based on a first embedding matrix obtained by converting an embedding output from a neural network to be trained, a first embedding matrix obtained by converting an embedding output from a trained neural network, a second embedding matrix obtained by converting an embedding output from the neural network to be trained, and a second embedding matrix obtained by converting an embedding output from the trained neural network; A metric loss calculation unit that calculates a metric loss of the neural network to be trained; An optimization unit that optimizes the neural network to be trained based on the metric loss and the regularization loss, A machine learning device characterized in that, when learning of the neural network to be trained is completed, the neural network to be trained is designated as a trained neural network.

2. The machine learning device according to claim 1 , wherein the embedding conversion unit generates the first embedding matrix and the second embedding matrix by smoothing different regions of the embedding output by a neural network.

3. 2. The machine learning device according to claim 1, wherein the first embedding matrix is ​​an equivalent transformation of an embedding output by a neural network, and the second embedding matrix is ​​an embedding vector obtained by smoothing the embedding output by the neural network.

4. an embedding transformation step of transforming the embeddings output by the neural network into a first embedding matrix and a second embedding matrix; a regularization loss calculation step of calculating a regularization loss based on a first embedding matrix obtained by converting an embedding output from the neural network to be trained, a first embedding matrix obtained by converting an embedding output from a trained neural network, a second embedding matrix obtained by converting an embedding output from the neural network to be trained, and a second embedding matrix obtained by converting an embedding output from the trained neural network; A metric loss calculation step of calculating a metric loss of the neural network to be trained; and optimizing the neural network to be trained based on the metric loss and the regularization loss; A machine learning method, characterized in that when learning of the neural network to be trained is completed, the neural network to be trained is designated as a trained neural network.

5. an embedding transformation step of transforming the embeddings output by the neural network into a first embedding matrix and a second embedding matrix; a regularization loss calculation step of calculating a regularization loss based on a first embedding matrix obtained by converting an embedding output from the neural network to be trained, a first embedding matrix obtained by converting an embedding output from a trained neural network, a second embedding matrix obtained by converting an embedding output from the neural network to be trained, and a second embedding matrix obtained by converting an embedding output from the trained neural network; A metric loss calculation step of calculating a metric loss of the neural network to be trained; optimizing the neural network to be trained based on the metric loss and the regularization loss; A machine learning program, characterized in that when learning of the neural network to be trained is completed, the neural network to be trained is designated as a trained neural network.

Images