Learning device, learning method, and program

Abstract

To learn a model while maintaining the feature quantity of a learned model in a learning initial stage.SOLUTION: When causing a user to learn a model by using a parameter of a learned model, a learning device adds a task, and mixes a value for the added task and training data prepared in advance at a predetermined mixing ratio and causes the user to learn the model. The learning device increases an intermediate layer to be learned toward a low order layer according to the progress of learning with a layer for solving the added task as a starting point. The learning device may use output in a predetermined learned model in place of the training data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to a learning device, a recognition device, a learning method, and a program. [Background technology]

[0002] In recent years, the practical application of models based on machine learning technology has been progressing. Among these, the practical application of models using neural networks has been particularly advanced. While many methods have been proposed to achieve highly accurate models, there is no established method for predicting what kind of structure and combination of recognition problems should be used to solve a highly accurate model before training. To achieve a highly accurate model suitable for practical use, it is necessary to check for discrepancies between the estimation results obtained by inputting data into a trained model and the ideal output, which is the ground truth data. If a discrepancy exists, the input data and the output of the trained model are analyzed, and countermeasures are taken. If the analysis reveals that the discrepancy between the estimation results and the ground truth data is due to the trained model not acquiring features effective for recognition, it is possible to revise the configuration of the trained model to enable it to learn features effective for recognition, thereby reducing the discrepancy and improving accuracy.

[0003] Non-patent document 1 discloses a method for acquiring general-purpose features for each task by simultaneously learning multiple tasks, thereby improving generalization performance without getting stuck in local minima. However, in order to efficiently learn a model, it is necessary to pre-set the tasks that the model will solve at the initial stage of learning. In this method, if you try to change the model configuration and add a new task, the parameters related to the added task will be initialized with random numbers or the like before learning. In that case, the parameters related to the added task will be significantly updated in the initial stage of learning, so the features of the previously learned tasks cannot be retained. In contrast, Non-Patent Document 2 discloses a method for integrating two trained models using the distillation method described in Non-Patent Document 3, which has been reported to have the effect of adding flexible recognition tasks and improving generalization performance. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] R. Collobert and J. Weston, A unified architecture for natural language processing, Proceedings of the 25th international conference on Machine learning, 20(1):150-167, 2008. [Non-Patent Document 2] J. Zhang et al. Class-incremental Learning via Deep Model Consolidation, arXiv:1903.07864, 2019. [Non-Patent Document 3] G. Hinton, O. Vinyals, J. Dean, Distillation the Knowledge in a Neural Network, Neural Information Processing Systems, 2014. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The method described in Non-Patent Document 2 trains the entire model to improve the recognition accuracy of all tasks solved by the two pre-trained models. Therefore, it may not always be optimal when the additional tasks are used only as supplementary information to the pre-trained tasks to improve the accuracy of the pre-trained tasks. For example, if you want to train a model specialized for a specific task, the effectiveness of the method described in Non-Patent Document 2 will be diminished if the recognition accuracy of the pre-trained model is low for that specific task you want to improve the accuracy of.

[0006] In view of the aforementioned problems, this invention aims to accurately update the parameters of a trained model. [Means for solving the problem]

[0007] The learning device according to the present invention includes an additional means for adding a task to a trained model having a hierarchical structure, an estimated value for the added task, and training data for the added task, or A pre-trained model different from the aforementioned pre-trained model, which is a model for solving the added task. The system is characterized by comprising: a mixing means that mixes predetermined values ​​obtained from data generated based on with a predetermined mixing ratio that changes according to the progress of learning; an output means that outputs an estimated value for the main task of the trained model using the estimated value obtained by mixing the predetermined values ​​by the mixing means; and an update means that updates the parameters of the trained model based on the estimation error between the estimated value for the added task and the estimated value for the main task of the trained model. [Effects of the Invention]

[0008] According to the present invention, the parameters of a trained model can be updated with high accuracy. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example of the hardware configuration of a learning device according to the first embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of the learning device according to the first embodiment. [Figure 3]It is a diagram showing the learning data of the first embodiment. [Figure 4] It is a flowchart showing an example of the processing procedure of the learning device according to the first embodiment. [Figure 5] It is a flowchart showing an example of the processing procedure during learning according to the first embodiment. [Figure 6] It is a diagram showing the neural network of the first embodiment. [Figure 7] It is a diagram showing the transition of the learning settings of the first embodiment. [Figure 8] It is a diagram showing the input and output of the inference unit according to the second embodiment. [Figure 9] It is a block diagram showing an example of the functional configuration of the learning device according to the third embodiment. [Figure 10] It is a flowchart showing an example of the processing procedure for selecting the recognition problem to be added according to the fourth embodiment. [Figure 11] It is a diagram for explaining the misrecognition case of the first embodiment.

Mode for Carrying Out the Invention

[0010] (First Embodiment) The present invention is an invention regarding a method of adding and learning recognition problems solved by a learned model in order to improve the accuracy of a neural network having a hierarchical structure. Hereinafter, embodiments to which the present invention is applied will be described. In the following description, the recognition problem finally solved by the model is expressed as the main problem (main task), and the recognition problem solved during the learning process of the model is expressed as the auxiliary problem (subtask). In the present embodiment, the main problem is set as the problem of extracting the "face" region. Also, it is assumed that a learned model for estimating the "person" region including the "face" is prepared in advance.

[0011] In the present embodiment, in order to estimate the "face" region, when using a learned model for estimating the "person" region, for the purpose of suppressing over-detection in which regions other than the "face" are estimated as the "face", a method of adding and learning recognition problems of extracting the "doll" region and the "animal" region to the auxiliary problem will be described. However, the recognition problems to which the present invention can be applied are not limited to these combinations. For example, if a trained model that estimates the "face" region is used and there are many cases of undetected "faces" where the "face" region wearing a mask is not estimated as a "face," a recognition problem to extract the "face wearing a mask" region may be added as an auxiliary problem. Hereafter, over-detection and under-detection will be collectively referred to as misrecognition.

[0012] Here, we will use Figure 11 to illustrate specific examples of misrecognition. Figure 11(a) shows an example of over-detection, and Figure 11(b) shows an example of under-detection. First, let's explain Figure 11(a). Input data 101 is the data to be recognized, and training data 102 is the data that represents the ideal correct answer to the problem of extracting "face" regions from input data 101. Estimation result 103 is the estimation result of the "face" region when input data 101 is input to the trained model. The white areas are the estimation results of the "face" region (the same applies in the following explanation). Estimation result 103 estimates areas that are not "face" regions in training data 102 as "face" regions, resulting in over-detection.

[0013] Next, let's explain Figure 11(b). Input data 104 is the data to be recognized, and training data 105 is the data that represents the ideal correct answer to the problem of extracting the "face" region from input data 104. Estimation result 106 is the estimation result of the "face" region when input data 104 is input to the trained model. Estimation result 106 estimates the "face" region of training data 105 as a region other than a "face", and is therefore undetected.

[0014] Next, the hardware configuration of the learning device 200 according to this embodiment will be described using Figure 1. The learning device 200 has a hardware configuration that includes a CPU 11, ROM 12, RAM 13, HDD 14, display unit 15, operation unit 16, and network I / F unit 17. The CPU 11 reads the control program stored in the ROM 12 and executes various processes. The RAM 13 is used as the CPU 11's main memory and temporary storage area such as the work area. The HDD 14 stores various data and programs. The display unit 15 displays various information. The operation unit 16 has a keyboard and mouse and accepts various operations from the user. The network interface unit 17 performs communication processing with external devices via the network. Alternatively, the network interface unit 17 may communicate with external devices wirelessly.

[0015] The functions and processing of the learning device 200, described later, are realized by the CPU 11 reading a program stored in the ROM 12 or HDD 14 and executing this program. Alternatively, the CPU 11 may read a program stored in a recording medium such as an SD card instead of the ROM 12.

[0016] Next, an example of the functional configuration of the learning device 200 in this embodiment will be described using Figure 2. Figure 2(a) is a block diagram showing an example of the functional configuration of the learning device 200, and Figure 2(b) is a block diagram showing an example of the configuration of the inference unit 208 of the learning device 200. As shown in Figure 2(a), the learning device 200 includes a parameter storage unit 201, a selection unit 202, an addition unit 203, a data storage unit 204, and a learning unit 205. The parameter storage unit 201 stores parameters necessary for recognition, such as filters, weight coefficients, and constant terms for the convolutional layers of the convolutional neural network. The selection unit 202 selects the additional supplementary questions. The addition section 203 adds the auxiliary problems selected by the selection section 202. The data storage unit 204 stores the training data.

[0017] Here, we will explain the structure of the training data used in this embodiment using Figure 3. The training data 300 consists of input data 301 to be recognized and training data 310 that shows the correct answers to the recognition problem for the input data 301. The training data 302 to 305 in Figure 3 are data that show the ideal correct answers for the "person," "animal," "costume," and "face" regions for the input data 301, respectively. In this embodiment, the model is trained using a dataset consisting of a predetermined number of training data prepared in advance by the user.

[0018] Returning to the explanation of Figure 2, the learning unit 205 updates the parameters stored in the parameter storage unit 201 using the learning data stored in the data storage unit 204. The learning unit 205 further includes an update unit 206, a decision unit 207, an inference unit 208, and a mixing unit 209. The determination unit 207 determines the parameter to be updated from the parameters stored in the parameter storage unit 201. The update unit 206 updates the parameters determined by the determination unit 207 and stores them in the parameter storage unit 201. The mixing unit 209 mixes predetermined values ​​with the estimated value of the auxiliary problem at an arbitrary mixing ratio. The inference unit 208 outputs an estimation result for the input based on the value in the parameter storage unit 201.

[0019] The inference unit 208 further includes an acquisition unit 211, a recognition unit 212, and an output unit 213 that outputs the estimation results of the recognition unit 212, as shown in Figure 2(b). The inference unit 208 can also function as a recognition device independent of the learning device 200, provided that it can access the parameter storage unit 201. For example, the parameters stored in the parameter storage unit 201 can be copied to a device independent of the learning device, and the processing of the inference unit 208 can be executed.

[0020] The acquisition unit 211 acquires training data from the data storage unit 204 and outputs the input data to the recognition unit 212. Alternatively, the acquisition unit 211 may acquire only the data to be recognized, which corresponds to the input data, from the data storage unit 204 and output it to the recognition unit 212. The recognition unit 212 retrieves parameters from the parameter storage unit 201 and solves the main problem. Here, we will explain the neural network used in the processing of the recognition unit 212 using Figure 6(a).

[0021] The neural network 601 is a neural network having multiple convolutional layers. Convolutional layer 605 is an example of a single convolutional layer that performs convolution, pooling, and normalization operations. The multiple convolutional layers that make up the neural network 601 are divided into the following configurations according to their function: an intermediate representation unit 702 that receives input data 301 from the acquisition unit 211 and acquires an intermediate representation, an auxiliary problem acquisition unit 703 that estimates an auxiliary problem from the intermediate representation, and an integration unit 704 that estimates the estimated value (posterior probability) of the main problem based on the estimation results of the auxiliary problem. Here, it is assumed that the neural network 601 is initially set to a recognition problem that extracts the "person" region as the auxiliary problem. The auxiliary problem acquisition unit 703 acquires the estimated value of each auxiliary problem based on the intermediate representation acquired by each layer of the intermediate representation unit 702.

[0022] The neural network 602 in Figure 6(b) is the neural network after the processing in the additional unit 203 has added a layer to the neural network 601 that estimates the recognition problem of extracting the "animal" region and the "costume" region. The processing in the additional unit 203 will be explained in detail later. The recognition unit 212 obtains the neural network parameters set as described above from the parameter storage unit 201. Here, using Figure 7(a), an example of the configuration of the recognition unit 212, which has been configured by reading the parameters of the neural network 602, will be explained.

[0023] The recognition unit 212 includes an input unit 701, an intermediate representation unit 702, an auxiliary problem acquisition unit 703, an integration unit 704, and an estimated value conversion unit 705. The configuration of the intermediate representation unit 702, the auxiliary problem acquisition unit 703, and the integration unit 704 is determined by the parameters read from the parameter storage unit 201. The acquisition unit 211 acquires the input data to be recognized for the main problem from the data storage unit 204. The input data acquired by the acquisition unit 211 is sent to the intermediate representation unit 702 via the input unit 701 of the recognition unit 212. The posterior probability, which is an estimate of the main problem obtained through the intermediate representation unit 702, the auxiliary problem acquisition unit 703, and the integration unit 704, is output to the estimate conversion unit 705. The estimated value conversion unit 705 converts the estimated value of the main problem into a predetermined format and outputs it to the output unit 213 as the final estimated result. For example, the estimated value conversion unit 705 binarizes the posterior probability, which is the estimated value of the main problem, using a threshold to determine whether it is a recognized object or not, converts it into a label, and outputs it to the output unit 213 as the final estimated result of the main problem. Alternatively, the estimated value of the main problem obtained by the integration unit 704 may be used as the final estimated result.

[0024] Next, an example of the processing procedure of the learning device 200 in this embodiment will be described using the flowchart in Figure 4(a). The processing in the flowchart in Figure 4(a) starts when an instruction to start learning is received. First, in S401, the CPU 11 acquires the trained parameters necessary to solve a predetermined recognition problem and stores them in the parameter storage unit 201. In this embodiment, we assume that these are the parameters of a neural network, as described in Figure 6(a), where the primary problem is "face" region extraction and the auxiliary problem is "person" region extraction. Trained parameters are acquired by training a neural network with such a configuration, or by downloading trained parameters that are publicly available on the Web.

[0025] In S402, the inference unit 208 performs estimation processing using the learned parameters acquired by the CPU 11. Subsequently, the selection unit 202 selects an additional auxiliary problem based on the estimation results of the inference unit 208, according to the user's input. In this embodiment, we will describe a case where the objective is to reduce over-detection cases in the "human" region detection problem, where the "costume" region or "animal" region is detected as an estimation result. In such a case, the detection problems of the "costume" region or "animal" region are selected as additional auxiliary problems. Furthermore, the additional auxiliary problems are not limited to those designed to address over-detection cases, as described above. This invention is equally effective for undetected cases. For example, if there are many undetected cases of "faces" wearing masks, additional problems for extracting "faces wearing masks" regions may be added to the training process.

[0026] Furthermore, the effects of the present invention can be expected even if the auxiliary problem is not an example of misrecognition of the main problem. For example, there are cases where the main problem and the auxiliary problem are in a relationship of inclusion. If the main problem is a problem of extracting the "automobile" region, adding a problem of extracting the "vehicle" region, which is a category that includes automobiles, as an auxiliary problem allows for the recognition of the "automobile" region using the global characteristics of "vehicles," and thus a high level of effectiveness can be expected. For the same reason, if the main problem is a problem of extracting the "vehicle" region, a high level of effectiveness can also be expected when the additional auxiliary problem is set to extract the "automobile" region.

[0027] In S404, the addition unit 203 adds the parameters necessary to solve the auxiliary problem selected by the selection unit 202 to the learned parameters stored in the parameter storage unit 201, and stores them in the parameter storage unit 201. An example of this process flow is described below. First, trained parameters are obtained from the parameter storage unit 201. Here, as mentioned earlier in the example of S401, we will explain using a trained neural network with a configuration like the neural network 601 in Figure 6(a) as an example. The auxiliary problem acquisition unit 703 of the trained neural network 601 is assigned the problem of extracting the "human" region as an auxiliary problem. By assigning the auxiliary problems selected by the selection unit 202, the problem of extracting the "costume" region and the problem of extracting the "animal" region, to this auxiliary problem acquisition unit 703, a neural network 602 with a different configuration of the auxiliary problem acquisition unit 703 can be obtained.

[0028] Next, parameters necessary to solve the additional auxiliary problem are added, specifically parameters that connect the predetermined intermediate layer between the auxiliary problem acquisition unit 703 and the integration unit 704, and the predetermined intermediate layer between the auxiliary problem acquisition unit 703 and the intermediate representation unit 702. These added parameters are then stored in the parameter storage unit 201. The added parameters are initialized with random numbers. This process allows the neural network 601 to be transformed into the neural network 602. In this embodiment, an example is described in which the unit for solving the additional recognition problem is added in parallel with the unit for solving the learned auxiliary problem. However, the unit for solving the newly added auxiliary problem may be added to any intermediate layer as shown in Figure 6(e).

[0029] Next, in S405, the learning unit 205 updates the parameters stored in the parameter storage unit 201. An example of this process flow will be explained below using the flowchart in Figure 5 and Figure 7. Figure 5 is a flowchart showing an example of a detailed processing procedure for S405. Figures 7(b) to 7(d) show an example of the progression of operations performed by the recognition unit 212, the determination unit 207, and the mixing unit 209.

[0030] First, the series of processes in L501 to L504 is a loop process based on the number of learning iterations. The learning unit 205 increments the count by 1 (the number of learning iterations increases by 1) when it has learned all the learning data held by the data storage unit 204 once, and then repeats the processes in L501 to L504 a predetermined number of times. Hereafter, learning will also be referred to as parameter updating. In S501, the determination unit 207 determines the parameters of the parameter storage unit 201 that the update unit 206 will update. The user must set the criteria for determining the parameters to be updated.

[0031] In this embodiment, a parameter β that changes in conjunction with the number of training iterations is used to determine the parameter to be updated. The value of parameter β is set to an initial value of 1.0 and decreases by 0.1 for every 1000 increases in the number of training iterations. The parameter to be updated is obtained by equation (1) below, starting from the layer that solves the auxiliary problem and moving in the input direction. train These parameters relate to layers separated by an integer value, and to all layers from the layer that solves the auxiliary problem onward. In equation (1), N represents the number of convolutional layers in the intermediate representation unit 702. N train =N × (1.0 - β) ···(1)

[0032] Through the above process, as the number of training iterations increases, it becomes possible to increase the number of layers that are updated on the lower order, starting from the layer that solves the newly added recognition problem. Below, using Figure 7, we will explain a specific example of the parameter determination process for updating the parameters. Figure 7(b) is a diagram illustrating the parameter determination process immediately after the addition unit 203 adds an auxiliary problem. Figure 7(c) is a diagram illustrating the parameter determination process immediately after each training data has been trained 5000 times from the state in Figure 7(b), and Figure 7(d) is a diagram illustrating the parameter determination process after each training data has been trained 10000 times or more.

[0033] In the state shown in Figure 7(b), where the number of training iterations is 0, the value of parameter β becomes 1.0, and N is determined by equation (1). train The value is 0. Therefore, starting from the layer that solves the auxiliary problem in S501, the parameters for 0 layers in the input direction, and the parameters for the layers after the layer that solves the auxiliary problem are updated by learning. In other words, the decision unit 207 determines that only the parameters of the layers after the layer that solves the auxiliary problem are to be updated. The parameters of the layers in the integration unit 704 become the parameters to be updated by learning, and the parameters of the layers in the intermediate representation unit 702 are fixed.

[0034] In the state shown in Figure 7(c), where the number of training iterations is 5000, the value of parameter β becomes 0.5, and N is determined by equation (1). train The value is N / 2. Therefore, the parameters to be updated are those related to N / 2 layers in the input direction, starting from the layer that solves the auxiliary problem in S501, and the parameters related to the layers after the layer that solves the auxiliary problem. Furthermore, in the state shown in Figure 7(d), where the number of training iterations exceeds 10,000, the value of parameter β becomes 0, and therefore N is determined by equation (1). train The value is N. Therefore, starting from the layer that solves the auxiliary problem in S501, the parameters for N layers on the lower side (input direction) and the parameters for the layers after the layer that solves the auxiliary problem are updated. In other words, the determination unit 207 determines that all layers are to be updated as parameters.

[0035] By sequentially expanding the range of layers to be updated, starting from the layer that estimates the added auxiliary problem, and using the pre-trained intermediate representations from the lower layers for training the higher layers, training can be performed without significantly updating the pre-trained intermediate representations compared to training all layers from the beginning. The range of layers to be updated may be expanded according to the progress of learning, as described above, or according to the change in the error calculated by a predetermined method. Specifically, for example, the change in the error between the estimated value of the main problem calculated in S507 (described later) and the training data up to the previous learning iteration n-1 is calculated, and the change in the error between the estimated value of the auxiliary problem calculated in S504 (described later) and its training data is calculated. Then, the prediction errors of the main problem and auxiliary problem are calculated to be 1 / β from the previous β update. 2 You could also set it so that β decreases by 0.1 each time it decreases.

[0036] Returning to the explanation of Figure 5, the series of processes in L502-L503 is a loop process that is executed for each training data. The learning unit 205 repeatedly executes the processes in L502-L503 for the number of training data. Note that processing may be performed on multiple training data at once, or on all training data at once. In S502, the input unit 701 acquires the input data of the i-th learning determined by L502 from the dataset and outputs it to the intermediate representation unit 702. In S503, the intermediate representation unit 702 obtains an intermediate representation from the input data acquired by the input unit 701 as described above with reference to FIG. 6(a).

[0037] In S504, the auxiliary problem acquisition unit 703 obtains an estimated value of each auxiliary problem based on the intermediate representation acquired by the intermediate representation unit 702. Here, let the estimated value of the auxiliary problem obtained for the training data be x est . Note that the intermediate representation has a two-dimensional map shape corresponding to the input image, and for each position corresponding to the input image, whether it is an object to be estimated by the auxiliary problem is estimated as a likelihood. It may be converted into a map with a lower resolution than the input image by pooling or the like. For example, if the value range is 0 to 1, when it is 0, it means that the possibility of being an estimation target of the auxiliary problem is low, and when it is 1, it means that the possibility of being an estimation target of the auxiliary problem is high. In S505, the mixing unit 209 obtains a value x est obtained by mixing the estimated value x gt of the auxiliary problem acquired by the auxiliary problem acquisition unit 703 with a predetermined value x mix at an arbitrary mixing rate α. Further, the mixing unit 209 outputs the obtained value x mix to the integration unit 704 as the estimated value of the auxiliary problem. x mix =(1 - α)x est +α×x gt ···(2)

[0038] Here, for the predetermined value x gt , teacher data for the added auxiliary problems (problems of extracting "doll" and "animal" regions) is used. Alternatively, by separately training or downloading a model specialized for solving the auxiliary problems, the estimated value of the auxiliary problem obtained from that model is x gtIt may also be used as such. The conditions for changing the mixing ratio α can be set by the user performing the learning via the operation unit 16. The mixing ratio α may be decreased as the number of learning iterations increases.

[0039] Here, we will explain the change in the mixing ratio α according to the number of training iterations using Figure 7. In this embodiment, the initial value of the mixing ratio α is set to 1.0, and the mixing ratio α is set to decrease by 0.1 each time each training data is trained 1000 times. Immediately after the start of learning, as shown in Figure 7(b), the mixing ratio α is 1.0, so the estimated values ​​of the added recognition problem and the training data are mixed in a ratio of 0:100. In other words, in the initial state of learning, the output of S505 becomes the training data itself. In the initial stages of learning, the discrepancy between the estimated values ​​and the training data is large, so regardless of what the estimated values ​​are, the layers after the intermediate representation are made to output the ideal training data itself. As a result, the layers after the intermediate representation are trained in the ideal state of the intermediate representation. As learning progresses, it is expected that the discrepancy between the estimated values ​​and the training data will decrease, so the mixing ratio of the estimated values ​​is gradually increased, and ultimately, learning is performed using the estimated values. In the state shown in Figure 7(c), where the number of training iterations is 5000, the mixing ratio α is 0.5, so the estimated values ​​of the added recognition problem and the training data are mixed in a 50:50 ratio. In the state shown in Figure 7(d), where the number of training iterations is 10000 or more, the mixing ratio α is 0, so the estimated values ​​of the added recognition problem and the training data are mixed in a 100:0 ratio. As shown in Figure 7(b) to Figure 7(c) and Figure 7(c) to Figure 7(d), the ratio of estimated values ​​for the added auxiliary problems to the training data changes depending on the number of training iterations. Through this process, the degree of dependence on the estimated values ​​for the added auxiliary problems during training can be gradually increased, and ultimately, the desired estimated values ​​corresponding to all auxiliary problems can be obtained.

[0040] The mixing ratio α may be changed according to the progress of learning, as described above, or it may be changed based on other criteria. For example, if the prediction errors for the main problem and auxiliary problems are 1 / α using the training data or validation data prepared separately by the user,2 The mixing ratio α may be set to decrease by 0.1 each time it decreases. Alternatively, the mixing ratio α may be the same value as the parameter β mentioned above in S501.

[0041] Returning to the explanation of Figure 5, in S506, the integration unit 704 calculates an estimated value of the main problem based on the value obtained in S505 and outputs it to the estimated value conversion unit 705. In S507, the estimated value conversion unit 705 converts the estimated value of the main problem calculated by the integration unit 704 into a predetermined format and outputs it to the output unit 213 as the estimated value of the main problem. In S508, the update unit 206 updates the parameters to be updated, as determined by the determination unit 207, using the backpropagation method based on the estimated error between the auxiliary problem and the main problem. Note that the parameter update method is not limited to the backpropagation method; other methods may also be used.

[0042] Returning to the explanation of Figure 4, in S406, the update unit 206 evaluates the performance of the trained model. Specifically, first, the inference unit 208 outputs an estimation result for the primal problem based on the parameters of the trained model stored in the parameter storage unit 201. The update unit 206 evaluates the performance of the trained model in a predetermined way based on the estimation result of the inference unit 208 and the training data stored in the data storage unit 204. The prescribed method may be one that uses a confusion matrix, one that uses the mean squared error, or one that is independently determined by the evaluator.

[0043] In S407, the update unit 206 determines whether to continue learning. Specifically, the update unit 206 determines whether the evaluation result in S406 meets the pre-set criteria. If the evaluation result does not meet the pre-set criteria, the update unit 206 decides to continue learning and returns to S402. On the other hand, if the evaluation result does meet the pre-set criteria, the update unit 206 decides not to continue learning and terminates the process shown in the flowchart in Figure 4(a).

[0044] In this embodiment, the recognition problem to be solved by the model and the additional recognition problem are set to segmentation problems. However, the present invention is not limited to segmentation problems and can be applied to recognition problems solved using machine learning in general. For example, it could be a classification problem, a regression problem, or a detection problem. Furthermore, the combination of the recognition problem to be solved by the model and the additional recognition problem can be the same or different. For example, the recognition problem to be solved by the model could be set to "scene recognition," and the additional recognition problem could be set to "object detection." Furthermore, in this embodiment, both the recognition problem solved by the model midway and the main problem are domain partitioning problems, but the present invention can be applied whether these recognition problems are the same or different. Also, in this embodiment, an example was described in which a recognition problem different from the main problem is added as an additional auxiliary problem, but the present invention may use the same recognition problem as the main problem as the additional auxiliary problem.In addition, in this embodiment, the input data to the model was described as an image, but the format of the input data to which the present invention can be applied is not limited to images.

[0045] In this embodiment, a model is trained that outputs only the estimation results of a pre-defined recognition problem in the intermediate layer. However, the present invention can also be applied to models in which units that learn intermediate representations without explicitly providing training data exist in parallel. As described above, according to this embodiment, when adding a task to be solved by the model using the parameters of a pre-trained model, the mixing ratio α is changed according to the number of training iterations, and the degree of dependence on the estimated value of the added auxiliary problem during training is gradually increased. This makes it possible to train the model without significantly updating the learned features of the pre-trained model in the early stages of training.

[0046] (Second embodiment) In the first embodiment, a method for learning a model was described in which the estimated values ​​of one or more auxiliary problems are integrated within the learning device and the estimated result of the main problem is output. However, the configuration of the learning device to which the present invention can be applied is not limited to a configuration that integrates the estimated values ​​of one or more auxiliary problems within the learning device. In this embodiment, we describe a model that outputs a value obtained by performing a simple calculation on the estimated values ​​of the primal problem and the auxiliary problems for solving the primal problem as the final estimated result of the primal problem.

[0047] In this embodiment, we will describe a process to add a problem to extract regions other than "faces" as an auxiliary problem in order to suppress the trained model that estimates the "face" region from estimating regions other than "faces" as "face" regions. Furthermore, in this embodiment, if the estimated value of the problem to extract regions other than "faces" is higher than a threshold, the estimated value of the "face" region is changed, and that value is output as the estimation result to train the model. Furthermore, in the following embodiments, the explanation of parts that overlap with the first embodiment will be omitted, and only the parts that differ in content from the first embodiment will be explained.

[0048] First, the details of the recognition unit 212 of the learning device 200 in this embodiment will be explained using Figures 6(c) and 6(d). Figure 6(c) is a diagram showing an example of a neural network defined by the parameters acquired by the CPU 11 in S401, and Figure 6(d) is a diagram showing an example of a neural network after the addition unit 203 adds an auxiliary problem in S404. First, the recognition unit 212 receives the data acquired by the acquisition unit 211 as input data and solves the recognition problem of extracting the "face" region, which is the primal problem, using an N-layer convolutional layer 605. The convolutional layer 605 performs convolution, pooling, and normalization processes. The auxiliary problem acquisition unit 703 receives the data acquired by the acquisition unit 211 as input data and solves the recognition problems of extracting the auxiliary "animal" region and the "costume" region, as well as the recognition problem of extracting the "face" region, which is the primal problem, using an N-layer convolutional layer 605. Subsequently, if the estimated value of the recognition problem for extracting regions other than "face" is higher than the threshold, the estimated value of the "face" region is lowered and output, and the recognition problem of extracting the "face" region, which is the primal problem, is solved.

[0049] Next, an example of the processing procedure of the learning device 200 in this embodiment will be described using the flowchart in Figure 4(a). Regarding S401 and S402, the process is the same as in the first embodiment, so we will omit the explanation. In S404, the addition unit 203 adds parameters for solving the extraction problems of the "costume" and "animal" regions as parameters related to the auxiliary problems, in parallel with the parameters necessary to solve the main problem, the "face" region extraction problem, and stores them in the parameter storage unit 201. The addition unit 203 also adds parameters for connecting the units that solve the main problem and the auxiliary problems with the integration unit 704 (hereinafter also referred to as the parameters of the integration unit 704) and stores them in the parameter storage unit 201. In this embodiment, in the processing described later, the integration unit 704 compares the estimated value of the auxiliary problem with the threshold θ, so the addition unit 203 adds the threshold θ as a parameter of the integration unit 704 and stores it in the parameter storage unit 201.

[0050] In S405, the learning unit 205 updates the parameters stored in the parameter storage unit 201. Here, the details of the process in S405 will be explained using the flowchart in Figure 5. Regarding steps S501 to S505, the process is the same as in the first embodiment, so we will omit the explanation. In S506, the integration unit 704 obtains an estimated value for the main problem based on the estimated value of the auxiliary problem obtained in processing S505 and the parameters stored in the parameter storage unit 201 in S404. For example, if the posterior probability (estimated value) of the auxiliary problem for extracting regions other than "faces" is greater than or equal to a threshold θ, the integration unit 704 replaces the posterior probability of the main problem with 0.0.

[0051] Here, we will explain a specific example of the processing in S506 using Figure 8. Figure 8(a) shows an example of an input image acquired by the acquisition unit 211, and Figure 8(b) shows an example of the estimated values ​​for each auxiliary problem obtained in the processing of S505. Figure 8(c) shows an example of the estimated value for the "face" region extraction problem (main problem) obtained in S506. The input image 801 in Figure 8(a) is the image to be recognized. Image 802 in Figure 8(b) is an image showing the "face" region and its posterior probability as the estimation result of the "face" region extraction problem. Images 803 and 804 are images showing the "animal" region, the "costume" region and its posterior probability as the estimated values ​​for the "animal" region and the "costume" region extraction problem, respectively.

[0052] In image 802, the posterior probability of the "face" region on the left is 0.8, and the posterior probability of the "face" region on the right is 0.2. Also, in image 803, the posterior probability of the "animal" region on the right is 0.9. Here, if the threshold θ is set to 0.5, the posterior probability of 0.9 for the "animal" region in image 803 is greater than or equal to the threshold θ. Therefore, the integration unit 704 replaces the posterior probability of the "face" region on the right side of image 802 with 0.0. In other words, image 805 is obtained assuming that no face was detected in that region. Generalizing this, the posterior probability x of the main problem obtained by the integration unit 704 is out The posterior probability x of the auxiliary problem sub Using x and the threshold θ, it can be expressed as shown in equation (3) below. Here, x main This is the posterior probability of the main problem, obtained based on the posterior probability of the auxiliary problem.

[0053]

number

[0054] In S507, the estimated value conversion unit 705 outputs the calculation result from S506 as the estimation result of the main problem. In the above example, the output shown in image 805 of Figure 8(c) is obtained as the estimation result of the "face" region extraction problem. The following steps S508, S406, and S407 are the same processes as in the first embodiment, so their explanation will be omitted. As described above, according to this embodiment, the present invention can be applied without being heavily dependent on the configuration of the trained model.

[0055] In this embodiment, we have described cases of over-detection for the main problem, but it can also be applied to cases of non-detection. In that case, the integration unit 704 can be set to use as the final estimation result the sum of the estimated value of the auxiliary problem and the estimated value of the main problem in the non-detection case. Furthermore, the parameters of the integration unit 704 that the addition unit 203 stores in the parameter storage unit 201 in S404 are not limited to the threshold θ for comparison with the estimated value of the auxiliary problem. For example, the weight w that combines the unit that solves the auxiliary problem and the unit that solves the main problem that is ultimately output. c The bias b may also be stored as a parameter of the integration unit 704. In that case, in S506, the integration unit 704 calculates the posterior probability x of the auxiliary problem using the following equation (4). c From the posterior probability x of the main problem out You may obtain it.

[0056]

number

[0057] (Third embodiment) In the first embodiment, an example was described in which the user prepares training data corresponding to each recognition problem to be solved by the model when creating a dataset for training. However, it is difficult to prepare a large number of images with training data corresponding to the auxiliary problems added by the extension 203 described in the first embodiment. Therefore, in this embodiment, we will describe a method using a so-called distillation method, which utilizes the output results of a trained model as training data for the additional recognition problem. By implementing the method described in this embodiment, the human resources required to provide training data can be reduced.

[0058] The following example illustrates how to generate training data to be used for training based on the output of a predetermined trained model, using Figure 9 as an example. In this embodiment, the additional auxiliary problem is the extraction problem of the "animal" region. Figure 9(a) shows an example of a functional configuration for dynamically acquiring training data within the learning device 1000. Figure 9(b) is a diagram illustrating the process of acquiring training data in advance. Furthermore, in this embodiment, the difference between the learning device 200 in the first embodiment and the training data storage unit 903 will be explained using Figure 9(b), and the parts that overlap with the first embodiment will not be explained.

[0059] First, to generate training data, the acquisition unit 211 outputs input data 301 from the data storage unit 204 to the recognition unit 212. The recognition unit 212 obtains parameters related to the neural network, which has been trained to solve the recognition problem added as an auxiliary problem by the addition unit 203 as the main problem, from the parameter storage unit 201, and performs recognition processing on the input data 301. Then, the recognition unit 212 outputs the estimation result 902 for the auxiliary problem to the output unit 213. The output unit 213 then obtains the estimation result 902 related to the auxiliary problem added by the addition unit 203. Finally, the training data storage unit 903 stores the estimation result 902 of the auxiliary problem obtained from the recognition device in the data storage unit 204 as training data corresponding to the input data 301.

[0060] The values ​​stored as training data may be the posterior probabilities (so-called soft targets) from the inference stage, as described in Non-Patent Document 3, or they may be binarized values ​​obtained from the recognition device based on pre-set thresholds for the estimation results of the auxiliary problem. By performing this process on all input data held by the data storage unit 204, a dataset for training can be generated using the output results of the trained recognition device. The process of acquiring training data for the auxiliary problems to be added by the addition unit 203 based on the learned parameters may be performed on all training data in advance before the learning device 200 executes the learning process, or it may be performed sequentially during the learning process. As described above, according to this embodiment, the human resources required to acquire training data can be reduced when performing the method described in the first embodiment.

[0061] (Fourth embodiment) In the first embodiment, an example was described in which the user pre-configures the recognition problem to be added. However, the recognition problem to be added can be selected and configured by the learning device. In this embodiment, a method for selecting the recognition problem to be added based on the output results of a trained model or manually assigned training data is described. In this embodiment, when solving the main problem using a pre-trained model, it is assumed that the model is trained with the aim of suppressing over-detection when there are many instances of over-detection in the estimation results.

[0062] Figure 4(b) is a flowchart showing an example of the processing procedure of the learning device 200 in this embodiment. The difference from the procedure described in the first embodiment is that instead of manually selecting the auxiliary problem to be added in S402, S403 is executed to automatically select the recognition problem to be added. Note that the processing of S401, S404 to S407 other than S403 is the same as in the first embodiment, so the explanation is omitted.

[0063] In S403, the selection unit 202 acquires predetermined values ​​such as the recognition result of the main problem based on the learned parameters for the input data, the recognition result of the auxiliary problems of the additional candidates based on the learned parameters, and the training data for the auxiliary problems of the additional candidates. The selection unit 202 then compares the acquired predetermined values ​​and selects the recognition problem to be added based on the comparison result.

[0064] Specifically, the selection unit 202 calculates the common area between the estimation results of the model trained on the candidate recognition problems to be added as auxiliary problems and the over-detection region of the main problem, and selects the recognition problem with the largest area of ​​the common area as the auxiliary problem to be added. In this embodiment, when acquiring the training data for the recognition problem to be added using the method of the third embodiment, the recognition problem to be added is set based on the output of the trained model, but the present invention can also be applied when the training data includes training data for recognition problems other than the recognition problem of the trained model.

[0065] The details of the process in S403 will be explained below using the flowchart in Figure 10. In S1001, the selection unit 202 creates a table to store the area of ​​the overlapping region (common region) between the training data for the recognition problem of the additional candidates and the over-detection region of the trained model, and performs an initialization process. The series of processes L1001 to L1002 is a loop process that is executed for each training data. The learning unit 205 repeatedly executes the processes L1001 to L1002 for the number of training data in the dataset. The data used in this embodiment may be training data, or it may be data other than training data that the user has prepared in advance.

[0066] In S1002, the inference unit 208 performs inference processing using the parameters of the trained model stored in the parameter storage unit 201. Subsequently, the selection unit 202 obtains the over-detection region in the output of the trained model based on the estimation results of the inference unit 208. In S1003, the selection unit 202 obtains training data for the recognition problem of the additional candidates through processing by the inference unit 208 of the model that has learned the recognition problem of the additional candidates. In S1004, the selection unit 202 adds the area of ​​the common region from S1002 and S1003 to the values ​​in the table created in S1001. Finally, in S1005, the selection unit 202 determines that any number of recognition problems are to be added in descending order of the area of ​​the common region to determine the recognition problem.

[0067] In this embodiment, the primary problem and the recognition problem for additional candidates are set as domain partitioning problems, and therefore, a method for determining the additional recognition problem by the area of ​​the domain has been described. However, the problems to which the present invention can be applied are not limited to domain partitioning problems. For example, in the case of a classification problem, the similarity between the estimation result of the trained model and the training data may be used. Furthermore, although this embodiment describes an example where both the primary problem and the additional recognition problem are domain partitioning problems, the present invention is also applicable when the trained model solves the classification problem and the additional recognition problem solves domain partitioning. For example, the additional recognition problem may be determined based on the product of the posterior probability of misrecognized cases by the trained model and the area ratio of the training data.

[0068] Furthermore, in this embodiment, an example was described in which a recognition problem is added to a trained model to suppress false positives for the primal problem. However, as described above in the first embodiment, the present invention can be expected to be similarly effective when adding recognition problems for undetected cases or inclusion relationships.

[0069] The present invention works effectively when the accuracy of the model that solves the recognition problem of the additional candidate is high. This is because the training data for the additional recognition problem becomes more consistent, and the model can learn features specific to the additional recognition problem. Note that a model that solves many recognition problems may be used as the model that solves the recognition problem of the additional candidate, or multiple models that solve a single recognition problem may be used. As described above, in this embodiment, it is possible to train the model quickly and efficiently by selecting additional recognition problems.

[0070] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0071] This embodiment includes the following configurations, methods, and programs.

[0072] (Composition 1) An additional means for adding tasks to a pre-trained model having a hierarchical structure, A mixing means for mixing an estimated value for the added task with a predetermined value obtained from the training data for the added task or data generated based on the trained model at a predetermined mixing ratio, An update means that updates the parameters of the trained model using the estimated value obtained by mixing the predetermined values ​​by the mixing means, A learning device characterized by having the following features.

[0073] (Configuration 2) The learning device according to configuration 1, characterized in that the mixing means changes the predetermined mixing ratio according to the progress of learning. (Composition 3) The learning device according to configuration 1 or 2, further comprising a determination means for determining the layer on which to update parameters using the update means.

[0074] (Composition 4) The learning device according to configuration 3, characterized in that the determination means determines the layer to be updated according to the progress of learning. (Composition 5) The learning device according to configuration 4, characterized in that the decision means increases the number of layers to be updated from the layer that solves the added task downwards, in accordance with the progress of learning.

[0075] (Composition 6) The learning device according to any one of configurations 1 to 5, wherein the additional means is characterized by adding a task selected in response to user operation. (Composition 7) The learning device according to any one of configurations 1 to 5, wherein the additional means is characterized by adding a task selected based on a comparison between the output of the trained model and the predetermined value. (Composition 8) The learning device according to any one of configurations 1 to 5, characterized in that the additional task is a task different from the main task of the trained model. (Composition 9) A learning device according to any one of configurations 1 to 5, characterized in that the additional task includes the same task as the main task of the trained model.

[0076] (Composition 10) A recognition device that performs a recognition task using a trained model whose parameters have been updated by a learning device, The learning device is An additional means for adding tasks to a pre-trained model having a hierarchical structure, A mixing means for mixing an estimated value for the added task with a predetermined value obtained from the training data for the added task or data generated based on the trained model at a predetermined mixing ratio, An update means that updates the parameters of the trained model using the estimated value obtained by mixing the predetermined values ​​by the mixing means, It has, The aforementioned recognition device is A recognition device characterized by having recognition means that performs a recognition task using a trained model with updated parameters.

[0077] (Method 1) An additional step of adding tasks to a pre-trained model that has a hierarchical structure, A mixing step of mixing the estimated value for the added task and a predetermined value obtained from the training data for the added task or data generated based on the trained model at a predetermined mixing ratio, An update step in which the parameters of the trained model are updated using the estimated values ​​obtained by mixing the predetermined values ​​in the mixing step, A learning method characterized by having the following features. [Explanation of Symbols]

[0078] 203 additional parts, 209 mixed parts, 206 updated parts

Claims

1. An additional means for adding tasks to a pre-trained model having a hierarchical structure, A mixing means that mixes the estimated value for the added task with the training data for the added task, or a predetermined value obtained from data generated based on a model different from the previously trained model for solving the added task, at a predetermined mixing ratio that changes according to the progress of learning. An output means that outputs an estimated value for the main task of the trained model using an estimated value obtained by mixing the predetermined values ​​by the mixing means, An update means for updating the parameters of the trained model based on the estimation error between the estimated value for the added task and the estimated value for the main task of the trained model, A learning device characterized by having the following features.

2. The learning device according to claim 1, further comprising a determination means for determining the layer on which to update parameters by the update means.

3. The learning device according to claim 2, characterized in that the determination means determines the layer to be updated according to the progress of learning.

4. The learning device according to claim 3, characterized in that the decision means increases the number of layers to be updated from the layer that solves the added task downwards, in accordance with the progress of learning.

5. The learning device according to any one of claims 1 to 4, wherein the additional means adds a task selected in response to user operation.

6. The learning device according to any one of claims 1 to 4, wherein the additional means adds a task selected based on a comparison between the output of the trained model and the predetermined value.

7. The learning device according to any one of claims 1 to 4, characterized in that the additional task is a task different from the main task of the trained model.

8. The learning device according to any one of claims 1 to 4, characterized in that the additional tasks include the same tasks as the main tasks of the trained model.

9. A recognition device that performs a recognition task using a trained model whose parameters have been updated by a learning device, The learning device is An additional means for adding tasks to a pre-trained model having a hierarchical structure, A mixing means that mixes the estimated value for the added task with the training data for the added task, or a predetermined value obtained from data generated based on a model different from the previously trained model for solving the added task, at a predetermined mixing ratio that changes according to the progress of learning. An output means that outputs an estimated value for the main task of the trained model using an estimated value obtained by mixing the predetermined values ​​by the mixing means, An update means for updating the parameters of the trained model based on the estimation error between the estimated value for the added task and the estimated value for the main task of the trained model, It has, The aforementioned recognition device is A recognition device characterized by having recognition means that performs a recognition task using a trained model with updated parameters.

10. A learning method performed by a learning device, An additional step of adding tasks to a pre-trained model that has a hierarchical structure, A mixing step in which the estimated value for the added task and the training data for the added task, or a predetermined value obtained from data generated based on a model for solving the added task that is different from the previously trained model, are mixed at a predetermined mixing ratio that changes according to the progress of learning. An output step which outputs an estimated value for the main task of the trained model using the estimated value obtained by mixing the predetermined values ​​in the mixing step, An update step to update the parameters of the trained model based on the estimation error between the estimated value for the added task and the estimated value for the main task of the trained model, A learning method characterized by having the following features.

11. An additional step of adding tasks to a pre-trained model that has a hierarchical structure, A mixing step in which the estimated value for the added task and the training data for the added task, or a predetermined value obtained from data generated based on a model for solving the added task that is different from the previously trained model, are mixed at a predetermined mixing ratio that changes according to the progress of learning. An output step which outputs an estimated value for the main task of the trained model using the estimated value obtained by mixing the predetermined values ​​in the mixing step, An update step to update the parameters of the trained model based on the estimation error between the estimated value for the added task and the estimated value for the main task of the trained model, A program that causes a computer to execute something.

Images