Machine learning device, machine learning method, and machine learning program

Abstract

To provide a machine learning technology capable of performing transfer learning depending on properties of domains.SOLUTION: A domain adaptable data adequacy determination section 52 determines domain adaptable data adequacy based on a count of teacher data for a second domain when a first model of a neural network trained using teacher data of a first domain performs transfer learning by the teacher data of the second domain. A learning layer determination section 54 determines a layer to be learned by a second model obtained by duplicating the first model, based on the domain adaptable data adequacy. A transfer learning execution section 56 performs the transfer learning on the layer to be learned by the second model using the teacher data of the second domain.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to machine learning techniques. [Background technology]

[0002] Transfer learning is a technique for applying a model trained in one domain to another domain. In transfer learning, the original domain is called the source domain, and the domain to which it is transferred is called the target domain. It is necessary to efficiently adapt the model trained in the source domain to the target domain.

[0003] Patent document 1 discloses a domain transformation neural network configured to receive an input image from a source domain and process a network input including the input image from the source domain to generate a transformed image, which is a transformation of the input image from the source domain to a target domain different from the source domain. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-502665 Summary of the Invention [Problem to be solved by the invention]

[0005] In transfer learning, when adapting a model trained in a source domain to a target domain, transfer learning is performed regardless of the characteristics of the domain, which leads to issues such as a decrease in the generalizability of inference accuracy and an unnecessarily large amount of processing.

[0006] The present invention has been made in view of these circumstances, and its purpose is to provide a machine learning technique that is capable of transfer learning according to the characteristics of a domain. [Means for solving the problem]

[0007] In order to solve the above problem, a machine learning device according to one embodiment of the present invention includes a domain adaptation data ampleness determination unit that, when transfer learning a first model of a neural network trained using training data of a first domain using training data of a second domain, determines a domain adaptation data ampleness based on the number of training data of the second domain; a learning layer determination unit that determines a learning target layer of a second model that is a replica of the first model based on the domain adaptation data ampleness; and a transfer learning unit that transfer learns the learning target layer of the second model using the training data of the second domain.

[0008] Another aspect of the present invention is a machine learning method, which includes, when a first model of a neural network trained using training data of a first domain is subjected to transfer learning using training data of a second domain, a step of determining domain adaptation data richness based on the number of training data of the second domain, a step of determining a training target layer of a second model obtained by replicating the first model based on the domain adaptation data richness, and a step of transfer learning the training target layer of the second model using the training data of the second domain.

[0009] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a machine learning technique that is capable of transfer learning according to the characteristics of a domain. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of a machine learning device and an inference device according to an embodiment. [Figure 2] FIG. 2 is a detailed configuration diagram of a transfer learning unit of the machine learning device of FIG. 1. [Figure 3]FIG. 2 is a diagram showing the structure of a neural network model used as a source model and a target model in the machine learning device of FIG. 1. [Figure 4] FIG. 10 is a diagram illustrating a layer to be trained as a target model according to the degree of domain adaptation data richness. [Figure 5] FIG. 10 is a diagram illustrating another example of a layer to be learned according to the domain adaptation data richness. [Figure 6] FIG. 10 is a diagram illustrating yet another example of layers to be learned according to the domain adaptation data richness. [Figure 7] 2 is a flowchart illustrating a machine learning procedure performed by the machine learning device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1 is a configuration diagram of a machine learning device 100 and an inference device 200 according to an embodiment. The machine learning device 100 includes a source model storage unit 30, a target domain acquisition unit 40, a transfer learning unit 50, and a target model storage unit 60. The inference device 200 includes a target domain acquisition unit 70, an inference unit 80, and an inference result output unit 90.

[0013] Transfer learning is a machine learning technique that applies a model trained for a first task with sufficient data to a second task that is related to the first task but for which there is insufficient data. Because knowledge learned with sufficient data can be transferred to another task, highly accurate results can be obtained even for the second task with little data.

[0014] In transfer learning, when the input domain of the first task and the input domain of the second task are the same type and only the probability distribution is different, this is called "domain adaptation."

[0015] The input domain of the first task, which is the source of the transfer, is called the "source domain," and the input domain of the second task, which is the destination of the transfer, is called the "target domain." Also, the trained model for the first task is called the "source model," and the trained model for the second task is called the "target model."

[0016] As an example of domain adaptation, it is preferable to use computer graphics (CG) or images on the web, which are easy to collect data from, as the source domain, and real images taken with a camera or the like as the target domain, but the form of the images in the source domain and target domain is not limited to these.

[0017] A source model is trained using a large number of CG images as the source domain, and a target model is generated by transfer learning from the source model using domain adaptation by using images captured by a camera as the target domain. Here, the classes included in the domain are, for example, people, cars, bicycles, dogs, and motorbikes, and the task is classification.

[0018] The machine learning device 100 is a device that generates a target model from a trained source model and a target domain by transfer learning.

[0019] The source domain acquisition unit 10 acquires a large number of CG images of people, cars, bicycles, dogs, and motorcycles as source domains, which can be used to train a source model so that people, cars, bicycles, dogs, and motorcycles can be classified with high accuracy.

[0020] The learning unit 20 performs machine learning of a neural network model using the source domain to generate a source model, and stores the generated source model in the source model storage unit 30. The source model is capable of classifying the source domain with high accuracy.

[0021] The source model stored in the source model storage unit 30 is a trained model used as a transfer source model in transfer learning. The source model is a neural network model.

[0022] The target domain acquisition unit 40 acquires images of people, cars, bicycles, dogs, and motorcycles taken by a camera as the target domain. The target domain generally has less data than the source domain.

[0023] The transfer learning unit 50 uses the target domain acquired by the target domain acquisition unit 40 to transfer learn the source model stored in the source model storage unit 30 to generate a target model, and stores the generated target model in the target model storage unit 60.

[0024] The target model stored in the target model storage unit 60 is a trained model generated by transfer learning. The target model is a neural network model. The target model is a part of the source model that has been copied and retrained using the target domain.

[0025] The inference device 200 is a device that performs inference and classifies images using the target model generated by the machine learning device 100. The inference device 200, for example, includes an imaging unit, performs inference on images acquired by the imaging unit, and outputs the inference results.

[0026] The target domain acquisition unit 70 acquires the target domain to be inferred and supplies it to the inference unit 80. The inference unit 80 infers the target domain based on the target model stored in the target model storage unit 60, and supplies the inference result to the inference result output unit 90. The inference result output unit 90 outputs the classification that is the inference result.

[0027] 2 is a detailed configuration diagram of the transfer learning unit 50 of the machine learning device 100. The transfer learning unit 50 includes a domain adaptation data richness determination unit 52, a learning layer determination unit 54, and a transfer learning execution unit 56.

[0028] The domain adaptation data richness determining unit 52 determines the domain adaptation data richness based on the number of training data of the target domain.

[0029] More specifically, the domain adaptation data richness is the ratio between the number of training data for the class with the smallest number of training data per class in the target domain and the predetermined number of training data TDNUM. Here, the predetermined number of training data is the number of training data considered sufficient for accurate learning. The predetermined number of training data depends on the neural network model. Here, assuming the neural network model is VGG16, it is set to 3000. The learning accuracy according to the number of training data may be measured in advance using training data from the source domain, and the number of training data exceeding the desired accuracy may be used as the predetermined number of training data. The domain adaptation data richness is preferably calculated using the following formula: MIN(n,m) is a function that selects the minimum value of n or m. In this case, the domain adaptation data richness is a value between 0 and 1. Domain adaptation data richness = MIN(1, (number of training data in the class with the smallest number of training data in the target domain) / TDNUM)

[0030] The learning layer determination unit 54 determines a layer to be learned in the target model, which is a neural network model, based on the domain adaptation data richness.

[0031] Specifically, the learning layer determination unit 54 may increase the number of learning layers of the target model as the domain adaptation data richness increases, and decrease the number of learning layers of the target model as the domain adaptation data richness decreases.

[0032] The higher the domain adaptation data richness, the more layers the learning layer determination unit 54 includes in the learning target, from upper layers (layers closest to the input) to lower layers (layers closest to the output), and the lower the domain adaptation data richness, the fewer layers on the lower layer (layers closer to the output) side the learning target. In other words, the higher the domain adaptation data richness, the more layers the learning layer determination unit 54 includes in the learning target layers that are closer to the input layer.

[0033] The learning layer determination unit 54 may select only the fully connected layer of the target model as the layer to be learned when the domain adaptation data richness is equal to or less than a predetermined value.

[0034] In this way, when the degree of domain adaptation data richness is low, i.e., when the amount of training data is small, by training only the lower layers of the target model, the source model can be adapted to the target domain while maintaining the generalization performance of detailed feature extraction in the upper layers of the source model, and the accuracy of the target model after transfer learning can be maintained high. Conversely, when the degree of domain adaptation data richness is high, i.e., when the amount of training data is large, by training more layers of the target model, the accuracy of the target model after transfer learning can be increased.

[0035] The transfer learning execution unit 56 uses images of the target domain as training data to perform transfer learning on the layers to be learned of the target model determined by the learning layer determination unit 54. Layers other than the layers to be learned are not to be learned, and the layers of the copied source model are used as they are without being newly learned.

[0036] FIG. 3 is a diagram showing the structure of a neural network model used as a source model and a target model in the machine learning device 100.

[0037] In this embodiment, the source model and target model are VGG16, a neural network model. VGG16 is composed of 13 convolutional layers (CONV), 3 fully connected layers (Dense), and 5 pooling layers. The layers to be trained are the convolutional layers and the fully connected layers. The pooling layer is a layer that subsamples the feature map, which is the output of the convolutional layer. The source model and target model are not limited to VGG16, and the number of layers is not limited to this embodiment.

[0038] FIG. 4 is a diagram illustrating the layers to be trained for the target model according to the degree of domain adaptation data richness.

[0039] If the domain adaptation data richness is 1.00, all layers are used as training targets. If the domain adaptation data richness is less than 1.00 and greater than or equal to 0.95, layers excluding CONV1-1 are used as training targets. CONV1-1 uses a direct copy of the source model's layers. If the domain adaptation data richness is less than 0.95 and greater than or equal to 0.90, layers excluding CONV1-1 and CONV1-2 are used as training targets. CONV1-1 and CONV1-2 use a direct copy of the source model's layers. Similarly, if the domain adaptation data richness is less than 0.10 and greater than or equal to 0.00, no layers are used as training targets. In this case, all layers use a direct copy of the source model's layers.

[0040] In this way, the greater the domain adaptation data richness, the more convolutional layers closer to the input layer will be trained, and the smaller the domain adaptation data richness, the fewer convolutional layers closer to the output layer will be trained.

[0041] The relationship between domain adaptation data richness and the layers to be learned is not limited to this. The higher the domain adaptation data richness, the more layers should be included in the learning target, from the upper layers (layers closest to the input) to the lower layers (layers closest to the output); the lower the domain adaptation data richness, the fewer layers should be included in the learning target on the lower layer (layers closest to the output). Note that "Dense" refers to three fully connected layers. Here, three fully connected layers are controlled together, but they may also be controlled individually, as with convolutional layers. Also, while the domain adaptation data richness is defined as the ratio between the number of training data for the class with the smallest number of training data per class in the target domain and the predetermined number of training data TDNUM, the domain adaptation data richness may also be defined as the number of training data for the class with the smallest number of training data per class in the target domain.

[0042] FIG. 5 is a diagram illustrating another example of layers to be learned depending on the domain adaptation data richness.

[0043] In the example shown in Figure 5, multiple convolutional layers are bundled and controlled in pooling layer units. For example, two convolutional layers, CONV1-1 and CONV1-2, are bundled together and are candidates for training if their domain adaptation data richness is 1.00 or less and 0.95 or greater. Two convolutional layers, CONV2-1 and CONV2-2, are bundled together and are candidates for training if their domain adaptation data richness is 1.00 or less and 0.85 or greater. Similarly, two adjacent convolutional layers are bundled together and are candidates for training. This allows the feature extraction of the source model to be maintained for each feature map resolution.

[0044] FIG. 6 is a diagram illustrating yet another example of layers to be learned according to the domain adaptation data richness.

[0045] In the example shown in Figure 6, layers close to the input layer are never targeted for training. Here, the four convolutional layers CONV1-1, CONV1-2, CONV2-1, and CONV2-2 that are close to the input layer are not targeted for training. As for the other convolutional layers, the greater the domain adaptation data richness, the fewer convolutional layers that are closer to the output layer are targeted for training.

[0046] In this way, we can directly use the detailed edge-level feature extraction of the source domain and adapt it to the target domain from somewhat abstracted features, thereby improving accuracy and reducing the amount of calculation.

[0047] FIG. 7 is a flowchart illustrating the machine learning procedure performed by the machine learning device 100 of FIG.

[0048] The domain adaptation data adequacy determination unit 52 of the transfer learning unit 50 of the machine learning device 100 determines the domain adaptation data adequacy based on the number of training data of the target domain (S10).

[0049] The learning layer determination unit 54 determines the layer to be used as the learning target for the target model obtained by duplicating the source model, based on the domain adaptation data richness (S20).

[0050] The transfer learning execution unit 56 performs transfer learning on the learning target layer of the target model using images of the target domain as training data (S30).

[0051] The various processes of machine learning device 100 and inference device 200 described above can naturally be realized as devices using hardware such as a CPU and memory, but can also be realized by firmware stored in a ROM (read-only memory) or flash memory, or by software for a computer, etc. The firmware program or software program can be provided by being recorded on a computer-readable recording medium, or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting via terrestrial or satellite digital broadcasting.

[0052] As described above, according to the machine learning device 100 of this embodiment, by changing the layers of the neural network of the target model that undergoes transfer learning in accordance with the degree of domain adaptation data richness based on the amount of training data in the target domain, it is possible to generate a target model that has high processing efficiency in accordance with the characteristics of the domain, as well as high inference accuracy and generalization performance.

[0053] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0054] 10 source domain acquisition unit, 20 learning unit, 30 source model memory unit, 40 target domain acquisition unit, 50 transfer learning unit, 52 domain adaptation data richness determination unit, 54 learning layer determination unit, 56 transfer learning execution unit, 60 target model memory unit, 70 target domain acquisition unit, 80 inference unit, 90 inference result output unit, 100 machine learning device, 200 inference device.

Claims

1. a domain adaptation data adequacy determination unit that determines a domain adaptation data adequacy based on the number of teacher data of the second domain when a first model of the neural network trained using teacher data of the first domain is subjected to transfer learning using teacher data of the second domain; a learning layer determination unit that determines a learning target layer of a second model obtained by duplicating the first model based on the domain adaptation data richness; a transfer learning unit that performs transfer learning on a learning target layer of the second model using training data of the second domain, The machine learning device is characterized in that the learning layer determination unit controls whether to bundle multiple convolutional layers in units of pooling layers and set them as layers to be learned.

2. 2. The machine learning device according to claim 1, wherein the learning layer determination unit increases the number of layers to be learned as the domain adaptation data richness increases, and decreases the number of layers to be learned as the domain adaptation data richness decreases.

3. The machine learning device according to claim 1 or 2, wherein the learning layer determination unit is configured to include layers closer to the input layer as the domain adaptation data richness increases.

4. The machine learning device according to claim 1 , wherein the learning layer determination unit selects only fully connected layers as layers to be learned when the domain adaptation data richness is equal to or less than a predetermined value.

5. When a first model of a neural network trained using training data of a first domain is subjected to transfer learning using training data of a second domain, a step of determining a degree of domain adaptation data richness based on the number of training data of the second domain; determining a layer to be trained for a second model that replicates the first model based on the domain adaptation data richness; and performing transfer learning on a learning target layer of the second model using training data of the second domain; The machine learning method is characterized in that the step of determining the layer to be learned controls whether to bundle multiple convolutional layers together as the layer to be learned in units of pooling layers.

6. When a first model of a neural network trained using training data of a first domain is subjected to transfer learning using training data of a second domain, a step of determining a degree of domain adaptation data richness based on the number of training data of the second domain; determining a layer to be trained for a second model that replicates the first model based on the domain adaptation data richness; and performing transfer learning on a learning target layer of the second model using training data of the second domain; The machine learning program is characterized in that the step of determining the layer to be learned controls whether to bundle multiple convolutional layers together as the layer to be learned in units of pooling layers.

Images