Machine learning device, machine learning method, and machine learning program

Abstract

To provide a mechanical learning technique that allows a user to do a transfer learning according to characteristics of a domain.SOLUTION: A domain fitness determination unit 52 determines a domain fitness on the basis of the accuracy of an inference when an image of a second domain is inferred by using a first model of a neural network learned by using an image of a first domain as teacher data. A learning layer determination unit 54 determines a learning target layer of a second model formed by duplicating the first model on the basis of the domain fitness. A transfer learning execution unit 56 performs a transfer learning of a learning target layer of the second model by using the image of the second domain as teacher data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to machine learning techniques. [Background technology]

[0002] Transfer learning is a technique for applying a model trained in one domain to another domain. In transfer learning, the original domain is called the source domain, and the domain to which it is transferred is called the target domain. It is necessary to efficiently adapt the model trained in the source domain to the target domain.

[0003] Patent document 1 discloses a domain transformation neural network configured to receive an input image from a source domain and process a network input including the input image from the source domain to generate a transformed image, which is a transformation of the input image from the source domain to a target domain different from the source domain. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-502665 Summary of the Invention [Problem to be solved by the invention]

[0005] In transfer learning, when adapting a model trained in a source domain to a target domain, transfer learning is performed regardless of the characteristics of the domain. This leads to issues such as a decrease in the generalizability of inference accuracy and an increase in the amount of processing required.

[0006] The present invention has been made in view of these circumstances, and has a purpose to provide a machine learning technique capable of transfer learning according to the characteristics of a domain. [Means for solving the problem]

[0007] In order to solve the above problem, a machine learning device of one embodiment of the present invention includes a domain adaptability determination unit that determines domain adaptability based on inference accuracy when inferring images of a second domain using a first model of a neural network trained using images of a first domain as training data, a learning layer determination unit that determines a learning target layer of a second model that replicates the first model based on the domain adaptability, and a transfer learning unit that performs transfer learning of the learning target layer of the second model using images of the second domain as training data.

[0008] Another aspect of the present invention is a machine learning method, which includes a step of determining a domain adaptability based on an inference accuracy when inferring an image of a second domain using a first model of a neural network trained using an image of a first domain as training data, a step of determining a learning target layer of a second model obtained by duplicating the first model based on the domain adaptability, and a step of transfer learning the learning target layer of the second model using the image of the second domain as training data.

[0009] Yet another aspect of the present invention is a trained model, which causes a computer to infer an input image, and which has been transfer-learned by a step of determining domain adaptability based on an inference accuracy when inferring an image of a second domain using a first model of a neural network trained using an image of a first domain as training data, a step of determining a learning target layer of a second model obtained by duplicating the first model based on the domain adaptability, and a step of transfer learning the learning target layer of the second model using the image of the second domain as training data.

[0010] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0011] According to the present invention, it is possible to provide a machine learning technique capable of transfer learning according to the characteristics of a domain. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram of a machine learning device and an inference device according to an embodiment. [Diagram 2] FIG. 2 is a detailed configuration diagram of a transfer learning unit of the machine learning device in FIG. 1. [Diagram 3] FIG. 2 is a diagram showing the structure of a neural network model used as a source model and a target model in the machine learning device of FIG. 1. [Figure 4] FIG. 13 is a diagram for explaining layers to be trained in a target model according to domain adaptability. [Diagram 5] FIG. 13 is a diagram for explaining another example of layers to be learned according to domain adaptability. [Figure 6] FIG. 13 is a diagram illustrating yet another example of layers to be learned according to domain adaptability. [Figure 7] 2 is a flowchart illustrating a machine learning procedure performed by the machine learning device of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] 1 is a configuration diagram of a machine learning device 100 and an inference device 200 according to an embodiment. The machine learning device 100 includes a source model storage unit 30, a target domain acquisition unit 40, a transfer learning unit 50, and a target model storage unit 60. The inference device 200 includes a target domain acquisition unit 70, an inference unit 80, and an inference result output unit 90.

[0014] Transfer learning is a machine learning technique that applies a model trained for a first task, for which there is sufficient data, to learning a second task that is related to the first task but for which there is insufficient data. Because knowledge learned with sufficient data can be transferred to another task, it is possible to obtain highly accurate results even for the second task, for which there is little data.

[0015] In transfer learning, when the types of input domains of the first task and the second task are the same and only the probability distributions differ, this is called "domain adaptation."

[0016] The input domain of the first task, which is the source of the transfer, is called the "source domain," and the input domain of the second task, which is the destination of the transfer, is called the "target domain." Additionally, the trained model of the first task is called the "source model," and the trained model of the second task is called the "target model."

[0017] An example of domain adaptation is to use computer graphics (CG) or images on the web, from which data is easy to collect, as the source domain, and real images taken by a camera or other device as the target domain.

[0018] A source model is trained using a large amount of CG images as the source domain, and a target model is generated by transfer learning from the source model using domain adaptation by using images captured by a camera as the target domain. Here, the classes included in the domain are, for example, people, cars, bicycles, dogs, and motorcycles, and the task is classification.

[0019] The machine learning device 100 is a device that generates a target model from a trained source model and a target domain by transfer learning.

[0020] The source domain acquisition unit 10 acquires, as the source domain, a large number of CG images of people, cars, bicycles, dogs, and motorcycles that can train a source model so as to classify people, cars, bicycles, dogs, and motorcycles with high accuracy.

[0021] The learning unit 20 performs machine learning of a neural network model using the source domain to generate a source model, and stores the generated source model in the source model storage unit 30. The source model is capable of classifying the source domain with high accuracy.

[0022] The source model stored in the source model storage unit 30 is a trained model used as a transfer source model in transfer learning. The source model is a neural network model.

[0023] The target domain acquisition unit 40 acquires images of people, cars, bicycles, dogs, and motorcycles captured by a camera as the target domain. The target domain generally has less data than the source domain.

[0024] The transfer learning unit 50 uses the target domain acquired by the target domain acquisition unit 40 to transfer learn the source model stored in the source model storage unit 30 to generate a target model, and stores the generated target model in the target model storage unit 60.

[0025] The target model stored in the target model storage unit 60 is a trained model generated by transfer learning. The target model is a neural network model. The target model is a part of the source model that was once copied and re-trained using the target domain.

[0026] The inference device 200 is a device that performs inference and classifies images using a target model generated by the machine learning device 100. The inference device 200, for example, includes an imaging unit, performs inference on images acquired by the imaging unit, and outputs the inference result.

[0027] The target domain acquisition unit 70 acquires a target domain to be inferred, and supplies it to the inference unit 80. The inference unit 80 infers the target domain based on the target model stored in the target model storage unit 60, and supplies the inference result to the inference result output unit 90. The inference result output unit 90 outputs a classification that is the inference result.

[0028] 2 is a detailed configuration diagram of the transfer learning unit 50 of the machine learning device 100. The transfer learning unit 50 includes a domain adaptability determining unit 52, a learning layer determining unit 54, and a transfer learning executing unit 56.

[0029] The domain adaptability determining unit 52 determines the domain adaptability based on the inference accuracy when inferring an image of the target domain using a source model of a neural network trained using an image of the source domain as training data.

[0030] More specifically, the domain adaptability determination unit 52 determines the domain adaptability based on the inference (classification) accuracy of the source model stored in the source model storage unit 30 for the target domain acquired by the target domain acquisition unit 40. Here, the domain adaptability is the mAP (mean average precision) of the source model for the target domain. The mAP is a value obtained by averaging the average accuracy of all classes. The mAP is a real number between 0 and 1. In this embodiment, the accuracy is used as an index of the domain adaptability, but the F-measure, sensitivity, suitability, etc. may also be used. In addition, the accuracy of the domain adaptability is the accuracy of the source model, but it is not limited to this as long as it can execute the task of the target domain (classification in this embodiment). For example, if it is possible to classify people, cars, bicycles, dogs, and motorcycles, which are the classification targets in this embodiment, the accuracy of a trained model that exists in the cloud or the like and is capable of classifying more than the classification targets in this embodiment may be used.

[0031] The learning layer determination unit 54 determines a layer to be learned in a target model, which is a neural network model, based on the domain adaptability.

[0032] Specifically, the learning layer determination unit 54 may increase the number of learning layers of the target model as the domain adaptability decreases, and may decrease the number of learning layers of the target model as the domain adaptability increases.

[0033] The learning layer determination unit 54 may include more layers from upper layers (layers closer to the input) to lower layers (layers closer to the output) as learning targets as the domain adaptability becomes smaller, and may include fewer layers on the lower layer (layers closer to the output) side as learning targets as the domain adaptability becomes larger. In other words, the learning layer determination unit 54 may include more layers closer to the input layer as learning targets as the domain adaptability becomes smaller.

[0034] The learning layer determination unit 54 may determine that only the fully connected layer of the target model is the layer to be learned when the domain adaptability is equal to or greater than a predetermined value.

[0035] In this way, when the domain adaptability is large, that is, when the probability distributions of the target domain and the source domain are close, by training only the lower layers of the target model, the source model can be adapted to the target domain while maintaining the generalization performance of detailed feature extraction in the upper layers of the source model, and the accuracy of the target model after transfer learning can be maintained high. Conversely, when the domain adaptability is small, that is, when the probability distributions of the target domain and the source domain are far apart, by training more layers of the target model, the accuracy of the target model after transfer learning can be increased.

[0036] The transfer learning execution unit 56 uses the target domain images as training data to perform transfer learning on the layers to be learned of the target model determined by the learning layer determination unit 54. Layers other than the layers to be learned are not to be learned, and the layers of the copied source model are used as they are without being newly learned.

[0037] FIG. 3 is a diagram showing the structure of a neural network model used as a source model and a target model in the machine learning device 100.

[0038] In this embodiment, the source model and the target model are VGG16, a neural network model. VGG16 is composed of 13 convolutional layers (CONV), 3 fully connected layers (Dense), and 5 pooling layers. The layers to be trained are the convolutional layers and the fully connected layers. The pooling layer is a layer that subsamples the feature map, which is the output of the convolutional layer.

[0039] FIG. 4 is a diagram for explaining layers to be trained in a target model according to domain adaptability.

[0040] If the domain fitness is 0.00, all layers are used as learning targets. If the domain fitness is greater than 0.00 and less than or equal to 0.10, the layers excluding CONV1-1 are used as learning targets. CONV1-1 uses a direct copy of the source model's layers. If the domain fitness is greater than 0.10 and less than or equal to 0.20, the layers excluding CONV1-1 and CONV1-2 are used as learning targets. CONV1-1 and CONV1-2 use a direct copy of the source model's layers. Similarly, if the domain fitness is greater than 0.95 and less than or equal to 1.00, none of the layers are used as learning targets. In this case, all layers use a direct copy of the source model's layers.

[0041] In this way, the smaller the domain adaptability, the more convolutional layers closer to the input layer will be trained, and the larger the domain adaptability, the fewer convolutional layers closer to the output layer will be trained.

[0042] The relationship between domain adaptability and the layers to be learned is not limited to this. The lower the domain adaptability, the more layers from the upper layers (layers closest to the input) to the lower layers (layers closest to the output) should be included in the learning targets, and the higher the domain adaptability, the fewer layers on the lower layer (layers closest to the output) side should be the learning targets. Note that Dense refers to three fully connected layers. Here, three fully connected layers are controlled together, but they may be controlled one by one, as with the convolutional layers.

[0043] FIG. 5 is a diagram for explaining another example of layers to be learned according to domain adaptability.

[0044] This is an example of bundling and controlling multiple convolution layers at the pooling layer level. For example, two convolution layers CONV1-1 and CONV1-2 are bundled together and are the learning target when the domain fitness is 0.00 or more and 0.20 or less, and two convolution layers CONV2-1 and CONV2-2 are bundled together and are the learning target when the domain fitness is 0.00 or more and 0.40 or less. Similarly, two adjacent convolution layers are bundled together and are the learning target. In this way, the feature extraction of the source model can be maintained for each feature map resolution.

[0045] FIG. 6 is a diagram for explaining still another example of layers to be learned according to domain adaptability.

[0046] This is an example where layers close to the input layer are not always the target of training. Here, the four convolution layers CONV1-1, CONV1-2, CONV2-1, and CONV2-2 that are close to the input layer are not the target of training. As for the other convolution layers, the larger the domain fitness, the fewer convolution layers that are closer to the output layer are the target of training.

[0047] In this way, we can directly use the detailed edge-level feature extraction of the source domain and adapt it to the target domain from the somewhat abstracted features, thereby improving accuracy and reducing the amount of calculation.

[0048] FIG. 7 is a flowchart illustrating a machine learning procedure performed by the machine learning device 100 of FIG.

[0049] The domain adaptability determination unit 52 of the transfer learning unit 50 of the machine learning device 100 determines the domain adaptability based on the inference accuracy when inferring an image of a target domain using a source model trained using an image of the source domain as training data (S10).

[0050] The learning layer determination unit 54 determines a layer to be a learning target of the target model obtained by duplicating the source model, based on the domain adaptability (S20).

[0051] The transfer learning execution unit 56 performs transfer learning on the learning target layer of the target model using an image of the target domain as training data (S30).

[0052] The various processes of the machine learning device 100 and the inference device 200 described above can be realized not only as devices using hardware such as a CPU and a memory, but also as firmware stored in a ROM (read-only memory) or a flash memory, or as software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting of terrestrial or satellite digital broadcasting.

[0053] As described above, according to the machine learning device 100 of this embodiment, by changing the layers of the neural network of the target model that undergoes transfer learning in accordance with the domain fitness based on the inference accuracy of the source model for the target domain, it is possible to generate a target model that has high processing efficiency in accordance with the characteristics of the domain, as well as high inference accuracy and generalization performance.

[0054] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0055] 10 source domain acquisition unit, 20 learning unit, 30 source model memory unit, 40 target domain acquisition unit, 50 transfer learning unit, 52 domain adaptability determination unit, 54 learning layer determination unit, 56 transfer learning execution unit, 60 target model memory unit, 70 target domain acquisition unit, 80 inference unit, 90 inference result output unit, 100 machine learning device, 200 inference device.

Claims

1. a domain adaptability determination unit that determines domain adaptability based on inference accuracy when inferring an image of a second domain using a first model of a neural network trained using an image of a first domain as training data; a learning layer determination unit that determines a learning target layer of a second model obtained by duplicating the first model based on the domain fitness; and a transfer learning unit that performs transfer learning on a learning target layer of the second model using images of the second domain as training data.

2. The machine learning device according to claim 1 , wherein the learning layer determination unit increases the number of layers to be learned as the domain adaptability decreases, and decreases the number of layers to be learned as the domain adaptability increases.

3. The machine learning device according to claim 1 or 2, wherein the learning layer determination unit is configured to include a layer closer to the input layer as the domain adaptability is smaller in the learning target layer.

4. The machine learning device according to claim 1 , wherein the learning layer determination unit determines only a fully connected layer as a learning target layer when the domain adaptability is equal to or greater than a predetermined value.

5. determining domain adaptability based on inference accuracy when inferring an image of a second domain using a first model of a neural network trained using an image of the first domain as training data; determining a layer to be trained of a second model obtained by replicating the first model based on the domain fitness; and performing transfer learning on a learning target layer of the second model using images of the second domain as training data.

6. A step of determining domain adaptability based on inference accuracy when inferring an image of a second domain using a first model of a neural network trained using an image of a first domain as training data; determining a layer to be trained of a second model obtained by replicating the first model based on the domain fitness; and a step of performing transfer learning on a learning target layer of the second model using images of the second domain as training data.

Images