Information processor, control method for information processor, and program

Abstract

To enable selection, in a more suitable manner, of data which can be expected to improve an evaluation score of a specific evaluation index, under a situation in which a plurality of evaluation indices are evaluated as targets.SOLUTION: A search part 120 searches, from a series of learning data, for a data group that is capable of yielding a higher evaluation score, by being used for training of the model, with respect to an evaluation index corresponding to at least one evaluation score that is lower than a threshold, among evaluation scores for a plurality of evaluation indices each targeting a model to be learned, in learning of a task in which the plurality of evaluation indices exist. An evaluation score calculation part 110 calculates evaluation scores for each of the plurality of evaluation indices, with a model being learned as a target. A data group selection part 130 selects, from the series of learning data based on the search result, a data group capable of improving at least one evaluation score that is lower than the threshold, among the evaluation scores for the respective evaluation indices calculated with the model being learned as a target.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, various services using so-called AI have been provided. Further, as an example of a method for constructing a model (trained model) that realizes an AI for predicting an arbitrary event, a method using machine learning is known. One of the algorithms of such a machine learning model is supervised learning. Supervised learning is learning using training data composed of inputs and correct labels. When constructing a model by supervised learning, it is possible to expect an effect of suppressing overfitting and improving prediction accuracy by performing learning using high-quality training data. Here, the high-quality training data corresponds to training data having a high effect of improving the prediction accuracy of the model. Further, in order to tune a model to be adapted to a specific situation or application, it is desirable to perform learning using training data considering the situation or application. Therefore, in supervised learning, it may be important to appropriately select the training data to be used. Patent Document 1 discloses a technique of extracting feature amounts of training data, projecting them into a feature space, and selecting training data according to the distances between the feature amounts in the feature space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, consider the case where a machine learning model is used to evaluate multiple evaluation metrics. For example, when evaluating the degree of image degradation (PSNR) in a noise reduction task, the PSNR may be evaluated for each of a plurality of regions in the image. As another example, in an object detection task, cases where the accuracy rate for each class is evaluated may also be included. Thus, when a machine learning model evaluates multiple evaluation metrics, a certain level or higher of evaluation score may be required for any of the multiple evaluation metrics. In such a situation, for example, when the evaluation score is low for a specific evaluation metric, improvement of the evaluation score will be required. On the other hand, in the technique disclosed in Patent Document 1, since data selection is made according to the distance between each feature amount in the feature space, it is not known whether the specific estimation accuracy actually improves without passing through the machine learning model. Further, after feature extraction is performed by the machine learning model, mapping to the feature space and calculation of the distance between each feature amount are performed, so that a certain amount of time or more is required for the processing until data selection is performed.

[0005] In view of the above problems, an object of the present invention is to enable selection of data for which an effect of improving the evaluation score of a specific evaluation metric can be expected in a more suitable manner in a situation where evaluation is performed on multiple evaluation metrics.

Means for Solving the Problems

[0006] The information processing apparatus according to the present invention, in learning a task in which there are a plurality of evaluation metrics, for at least an evaluation metric corresponding to an evaluation score that is less than a threshold among the evaluation scores of each of the plurality of evaluation metrics for a model that is learned based on machine learning, a search means for searching for a data group capable of obtaining a higher evaluation score by using the data group in the learning of the model, a calculation means for calculating the evaluation score for each of the plurality of evaluation metrics for a model being learned based on machine learning, and a selection means for selecting, from among the series of learning data, a data group that can improve at least an evaluation score that is less than a threshold among the evaluation scores of each of the plurality of evaluation metrics calculated by the calculation means for the model being learned, based on the search result by the search means.

Effect of the Invention

[0007] According to the present invention, it becomes possible to select data that can be expected to have an effect of improving the evaluation score of a specific evaluation metric in a more suitable manner in a situation where evaluation is performed for a plurality of evaluation metrics.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] <Hardware Configuration> With reference to FIG. 8, an example of the hardware configuration of an information processing apparatus according to an embodiment of the present disclosure will be described. The processor 2001 is realized by, for example, a CPU (Central Processing Unit) or the like, and controls the operation of the entire information processing apparatus. The memory 2002 is realized by, for example, a RAM (Random Access Memory), and is a storage area for temporarily storing various programs and data. The storage medium 2003 is realized by, for example, a storage device readable by the information processing apparatus such as a hard disk or a CD-ROM, and is a storage area for storing various programs and data in the long term. In the present embodiment, the program stored in the storage medium 2003 is expanded in the memory 2002, and then the processor 2001 executes the program expanded in the memory 2002, whereby the functions of each part of the information processing apparatus are realized. The input interface 2004 is an interface for the information processing apparatus to acquire information from an external device. The output interface 2005 is an interface for the information processing apparatus to output information to an external device. Each component of the above-described information processing apparatus is connected to be able to transmit and receive data to and from each other via the bus 2006.

[0011] <First Embodiment> As a first embodiment of the present disclosure, taking a noise reduction task as an example, an example of an information processing apparatus (hereinafter, also referred to as a learning data selection apparatus) that selects learning data applied to model learning when evaluation indices are known in advance and pre-learning is performed will be described. The noise reduction task corresponds to a task of estimating a noise-free image (image before degradation) before degradation by noise from a noisy image (degraded image) degraded by noise.

[0012] With reference to FIG. 1(a), an example of the functional configuration of a learning data selection apparatus 1000 according to the present embodiment will be described. The learning data selection apparatus 1000 includes a learning data storage unit 150, an evaluation data storage unit 160, an evaluation score calculation unit 110, a search unit 120, a data group selection unit 130, and a data classification unit 140. Further, the search unit 120 according to the present embodiment includes a pre-learning unit 121. Further, the data classification unit 140 includes a random classification unit 141.

[0013] The learning data storage unit 150 stores a series of images (hereinafter, also referred to as learning images) used for model learning and correct data (correct data in supervised learning) corresponding to the images in an associated manner. In the present embodiment, since the noise reduction task is the target for model learning, the learning images are noisy images and the correct data are noise-free images. Regarding the noisy images, for example, it is possible to generate them by adding noise based on a noise model of an image sensor to a noise-free image. As another example, it is also possible to obtain a noisy image corresponding to the noise-free image by imaging the same scene as the noise-free image with a shorter exposure time and a higher ISO sensitivity than the noise-free image.

[0014] The evaluation data storage unit 160 stores evaluation data for calculating an evaluation score by the evaluation score calculation unit 110. The evaluation score calculation unit 110 calculates evaluation scores of a plurality of evaluation indices during model learning for the evaluation data stored in the evaluation data storage unit 160. The exploration unit 120 searches for a data group capable of improving an evaluation index whose evaluation score calculated by at least the evaluation score calculation unit 110 is less than a threshold value from among a series of learning data. The exploration unit 120 according to the present embodiment has a pre-learning unit 121, and searches for a data group capable of improving the evaluation score of each evaluation index based on the results of pre-learning. The data group selection unit 130 selects a data group to be used for learning the model from among a series of learning data based on the search result by the exploration unit 120. The data classification unit 140 classifies the learning data into a plurality of data groups. The data classification unit according to the present embodiment has a random classification unit 141, and randomly classifies the learning data into a plurality of data groups.

[0015] With reference to FIG. 2, an example of the processing of the learning data selection apparatus 1000 according to the present embodiment will be described. In the example shown in FIG. 2, the processes of S101 to S104 are processes executed by the pre-learning unit 121 before learning the target model. Further, S105 to S109 are processes repeatedly executed by the evaluation score calculation unit 110 and the data group selection unit 130 during the learning of the target model. Hereinafter, each process will be specifically described.

[0016] In S101, the data classification unit 140 classifies a series of learning data (for example, a series of learning data stored in the learning data storage unit 150) used for learning the model into a plurality of data groups. In the present embodiment, as described above, the data classification unit 140 has a random classification unit 141, and the random classification unit 141 classifies a series of learning data. Specifically, the random classification unit 141 randomly classifies a series of target learning data into a plurality of data groups. For example, FIG. 3(a) shows an example of the classification result of a series of learning data by the data classification unit 140. As shown in FIG. 3(a), a series of learning images are classified into a plurality of data groups. In the present embodiment, for the sake of simplicity in understanding the features, a series of learning images are assumed to be classified into three data groups, namely, data group A, data group B, and data group C.

[0017] In S102, the pre-training unit 121 individually trains the model for each data group, using each of the plurality of classified data groups in S101 as training data. For example, FIG. 3(a) schematically shows a situation where a model is generated by individually training each of the plurality of classified data groups as training data. In the case of the example shown in FIG. 3(a), since a series of training data is classified into three data groups, three models are trained. In this embodiment, for convenience, the model trained using data group A is referred to as model A, the model trained using data group B is referred to as model B, and the model trained using data group C is referred to as model C. Also, for the model to be trained, for example, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), Transformer, etc. can be applied. Of course, these models are merely examples and are not limited thereto, and other models may be applied.

[0018] In S103, the pre-training unit 121 evaluates the data for each evaluation metric using the trained model obtained as a result of the training in S102 for each of the plurality of evaluation metrics corresponding to the target task. As shown in FIG. 3(a), it is assumed that a series of evaluation images are divided into data suitable for evaluating each of the plurality of evaluation metrics. In this embodiment, for the sake of clarity of features, it is assumed that evaluations are performed for each of the three evaluation metrics A to C. Also, the data for evaluating evaluation metric A is also referred to as evaluation metric A data, the data for evaluating evaluation metric B is also referred to as evaluation metric B data, and the data for evaluating evaluation metric C is also referred to as evaluation metric C data. Also, the evaluation images A, B, and C shown in FIG. 3(a) respectively correspond to examples of evaluation metric A data, evaluation metric B data, and evaluation metric C data. For the evaluation metric, for example, PSNR in each of a plurality of regions in the image can be used. Figure 3(b) schematically shows a situation where evaluation is performed for each of evaluation index A, evaluation index B, and evaluation index C for each of model A, model B, and model C illustrated in Figure 3(a). As shown in Figure 3(b), the pre-training unit 121 performs evaluation of each evaluation image for each evaluation index using each of the generated model A, model B, and model C. In this case, since the data of three types of evaluation indexes are evaluated by three models, a total of nine evaluation results can be obtained.

[0019] In S104, the pre-training unit 121 determines a set of each of the plurality of data groups classified in S101 and evaluation data corresponding to each of the plurality of evaluation indexes based on the evaluation results of each of the plurality of evaluation indexes using each of the plurality of models in S103. As shown in Figure 3(c), the pre-training unit 121 summarizes the evaluation results for each combination of each trained model and each evaluation index data, and identifies a model having a higher evaluation score for the evaluation index data. For example, when focusing on evaluation index A, among the three models of model A to C, model C has the highest evaluation score, and a circle is marked in the table shown in Figure 3(c). On the other hand, for model B, the evaluation score is the lowest, and a cross is marked in the table shown in Figure 3(c). For model A, the evaluation score is between the scores of model C and model B, and a triangle is marked in the table shown in Figure 3(c). That is, in the example shown in Figure 3(c), in order to improve the evaluation score for evaluation index A, data group C may be used as the training data to be applied to the learning of the model. Similarly, in order to improve the evaluation score for evaluation index B, data group A may be used as the training data, and in order to improve the evaluation score for evaluation index C, data group B may be used as the training data. As described above, the combination of the data group used for the learning of each trained model and the evaluation image corresponding to the evaluation index in which the improvement of the evaluation score is incorporated by applying the data group as the training data will be found.

[0020] S105 is a process in which a target model is trained using a series of training data. Note that the subject of the S105 process is not particularly limited. As a specific example, the learning data selection device 1000 itself may train the target model. Also, as another example, another device different from the learning data selection device 1000 may train the target model. Further, at the first training, the learning of the target model is performed by applying a data group randomly selected from the series of training data as the training data. In contrast, at the second and subsequent trainings, the learning of the target model is performed by applying the data group selected by the processes of S106 to S109 as the training data. Hereinafter, the processes of S106 to S109 will be described in more detail.

[0021] In S106, the evaluation score calculation unit 110 calculates an evaluation score for each of a plurality of regions in the target image at a determined timing using the machine learning model being trained. In this embodiment, it is assumed that PSNR is used as the evaluation score. Also, regarding the calculation timing of the evaluation score, an index such as time or epoch is used, and the used index and the calculation timing are arbitrarily specified by the user.

[0022] In S107, the evaluation score calculation unit 110 determines whether there is a PSNR less than a predetermined threshold among the PSNRs calculated for each of the plurality of regions in the image in S106. Note that, regarding the predetermined threshold applied in S107, for example, when there is a target machine learning model, the threshold applied to the evaluation of the PSNR of each of the plurality of regions in the image in that machine learning model may be similarly applied. Also, as another example, an arbitrary value may be set by the user as the predetermined threshold. FIG. 4 shows an example of a result in which the PSNR is calculated for each of a plurality of regions in an image, and it is determined whether or not the calculated result of the PSNR is less than a threshold value. In the example shown in FIG. 4, only for the evaluation index B, the evaluation score exceeds the threshold value, and for the evaluation index A and the evaluation index C, the evaluation scores are less than the threshold value. When the evaluation score calculation unit 110 determines in S107 that there is a PSNR less than a predetermined threshold value, the process proceeds to S108. On the other hand, when the evaluation score calculation unit 110 determines in S107 that there is no PSNR less than a predetermined threshold value, the series of processes shown in FIG. 2 is terminated. At this time, the device that is learning the target model may determine whether to end or continue the learning of the model based on an instruction from the user.

[0023] In S108, based on the determination result of the combination of each of the plurality of classified data groups in S104 and each evaluation data, the data group selection unit 130 selects a data group capable of improving the PSNR of the corresponding region from the plurality of data groups. For example, in the case of the example shown in FIG. 4, since the evaluation scores of the evaluation index A and the evaluation index C are less than the threshold value, based on the example shown in FIG. 3(c), at least one of the data group C capable of improving the evaluation index A and the data group B capable of improving the evaluation index C will be selected. In S109, the data group selection unit 130 sets the data group selected in S108 as learning data to be applied to the learning of the model that is the learning target in S105. That is, in the process of S105 that is subsequently executed again, the learning of the target model is performed based on the learning data set in S109.

[0024] In addition, when there are multiple evaluation metrics whose evaluation scores are less than the threshold, for example, evaluation metric data corresponding to any one of the evaluation metrics that are less than the threshold may be selected, and control may be applied such that the processes of S105 to S109 are repeatedly executed until the evaluation metric is improved. In this case, for example, in S108, the data group selection unit 130 may select a data group that can improve the evaluation metric corresponding to the selected evaluation metric data. Also, in this case, after the target evaluation metric is improved, evaluation metric data corresponding to other evaluation metrics whose evaluation scores are less than the threshold is newly selected, and control may be applied such that the processes of S105 to S109 are repeatedly executed until the other evaluation metrics are improved. Of course, the above control is merely an example and does not limit the model learning method when there are multiple evaluation metrics whose evaluation scores are less than the threshold. As a specific example, when there are multiple evaluation metrics whose evaluation scores are less than the threshold, after evaluation metric data corresponding to two or more of the multiple evaluation metrics is selected, model learning may be performed. Also, in this case, in S108, the data group selection unit 130 may select a data group that can improve each of the two or more evaluation metrics.

[0025] By repeatedly executing the processes of S105 to S109 as described above, the model is learned such that the evaluation score of each of the series of evaluation metrics becomes equal to or greater than the threshold, or a value closer to the threshold. Thus, according to this embodiment, based on the result of pre-learning, by selecting a data group used for model learning from among a series of learning data, it becomes possible to improve the PSNR in the region of the image where the PSNR is less than a predetermined threshold.

[0026] Note that in this embodiment, since the noise reduction task is taken as an example for explanation, a noisy image is used as the learning image, a noise-free image is used as the correct answer image, and the PSNR for each region in the image is used as the evaluation metric. On the one hand, the application target of the technology according to this embodiment is not limited to the noise reduction task. For example, it is also possible to apply the technology according to the present disclosure to tasks such as a classification task or a BB (Bounding Box) detection task. In that case, it is desirable that learning images, correct answer images, evaluation metrics, etc. be appropriately selected according to the type of task. As a specific example, in the case of a classification task, the correct answer rate for each class is set as an evaluation metric, and learning data may be selected such that classes with a correct answer rate less than a predetermined threshold are improved. As another example, in the case of a BB detection task, the AP (Average Precision) for each class is set as an evaluation metric, and learning data may be selected such that classes with an AP less than a predetermined threshold are improved. Moreover, not limited to the above examples, for any task having an evaluation metric that can be represented numerically, it is possible to be the application target of the technology according to this embodiment.

[0027] <Second Embodiment> As a second embodiment of the present disclosure, taking the noise reduction task as an example, an example of a learning data selection device that selects learning data applied to the learning of a model in a case where a small amount of learning is performed to cope with a situation where the evaluation metric is not known in advance will be described.

[0028] With reference to FIG. 1(b), an example of the functional configuration of the learning data selection device 1001 according to this embodiment will be described. In FIG. 1(b), components having the same reference numerals as those shown in FIG. 1(a) are substantially the same as the components having the corresponding reference numerals in FIG. 1(a), and thus detailed descriptions thereof are omitted. The learning data selection device 1001 according to this embodiment is different from the learning data selection device 1000 shown in FIG. 1(a) in that it includes a small amount of learning unit 122 included in the search unit 120 and a feature classification unit 142 included in the data classification unit 140. The feature classification unit 142 classifies a series of learning data into a plurality of data groups based on a plurality of types of features of the target learning data. An example of the method of classifying a series of learning data by the feature classification unit 142 will be described in detail separately in conjunction with the description of the processing of the learning data selection device 1001. The small amount learning unit 122 uses the data groups classified from a series of learning data by the feature classification unit 142 to perform learning with a smaller amount of learning compared to the learning of a normal model. As a specific example, when the small amount learning unit 122 is learned by a data set of a series of learning data, the small amount learning unit 122 performs learning of the target model with a smaller amount of learning than the amount of learning (1 epoch). Note that the amount of learning by the small amount learning unit 122 may be appropriately adjusted according to the characteristics of the target task. As a specific example, the amount of learning by the small amount learning unit 122 may be determined by experimentally confirming the learning effect while appropriately adjusting the amount of learning. Depending on the characteristics of the target task, for example, even if the number of iterations is 1 / 2 or less of the number of iterations when learning is performed by a series of learning data sets, an effect may be expected. Moreover, based on the result of the small amount of learning, the small amount learning unit 122 searches for a data group that can improve the evaluation score of each evaluation index from among a plurality of data groups classified from a series of learning data.

[0029] With reference to FIG. 5, an example of the processing of 1001 of the learning data selection device according to the present embodiment will be described. In S201, the data classification unit 140 classifies a series of learning data used for model learning into a plurality of data groups. In the present embodiment, as described above, the data classification unit 140 has the feature classification unit 142, and the feature classification unit 142 classifies a series of learning data. Specifically, the feature classification unit 142 classifies a series of learning data into a plurality of data groups based on a plurality of types of features of the learning data.

[0030] Here, with reference to FIGS. 6 and 7, an example of a method for classifying a series of learning data by the feature classification unit 142 will be described. As shown in FIG. 7(a), the learning images are classified into a plurality of data groups according to features. In the present embodiment, for the sake of simplicity in understanding the features, a series of learning images (a series of learning data) are assumed to be classified into three data groups: a first feature data group A, a second feature data group B, and a third feature data group C. For example, Fig. 6(a) is a diagram showing an example of a method for classifying learning data. In Fig. 6(a), the black dots indicate samples obtained by converting each of a series of learning data into a three-dimensional vector based on the features of the learning data and plotting them in a three-dimensional space. In reality, each learning data is converted into a vector of a higher dimension and plotted in a multi-dimensional space. However, for simplicity of explanation in this embodiment, the case where each learning data is converted into a three-dimensional vector and plotted in a three-dimensional space will be taken as an example for explanation. For a series of samples (black dots) in which each of a series of learning data is plotted, for example, by using a clustering method called the k-means method, it is possible to classify the series of samples into a plurality of groups. Further, Fig. 6(b) is a diagram showing another example of a method for classifying learning data. In the example shown in Fig. 6(b), a series of learning data (learning images) are classified according to two types of features, frequency and contrast. Each of the series of learning data is plotted in a two-dimensional space with the vertical axis representing frequency and the horizontal axis representing contrast. That is, in the example shown in Fig. 6(b), a series of learning data are classified as data groups located in each of the four quadrants of a two-dimensional space with the vertical axis representing frequency and the horizontal axis representing contrast. Specifically, the data group located in the first quadrant corresponds to a high-frequency high-contrast image group, and the data group located in the second quadrant corresponds to a high-frequency low-contrast image group. Also, the data group located in the third quadrant corresponds to a low-frequency low-contrast image group, and the data group located in the fourth quadrant corresponds to a low-frequency high-contrast image group. Note that the methods shown in Fig. 6(a) and Fig. 6(b) are merely examples, and the method therefor is not particularly limited as long as it is possible to classify a series of learning data into a plurality of data groups based on the features of the learning data.

[0031] Here, refer to Fig. 5 again. S202 is a process in which a target model is trained using a series of training data. Note that the subject of the S202 process is not particularly limited. As a specific example, the learning data selection device 1001 itself may train the target model. Also, as another example, another device different from the learning data selection device 1001 may train the target model. Also, at the time of the first training, the learning of the target model is performed by applying, as training data, a data group randomly selected from a series of training data. On the other hand, at the time of the second and subsequent trainings, the learning of the target model is performed by applying, as training data, the data group selected by the processes of S203 to S209. Hereinafter, each of the processes of S203 to S209 will be described in more detail.

[0032] In S203, the evaluation score calculation unit 110 calculates an evaluation score for each of a plurality of regions in the target image at a determined timing using the machine learning model being trained. In this embodiment, it is assumed that PSNR is used as the evaluation score. Also, as for the calculation timing of the evaluation score, an index such as time or epoch is used, and the index used and the calculation timing are arbitrarily specified by the user.

[0033] In S204, the evaluation score calculation unit 110 determines whether there is a PSNR less than a predetermined threshold among the PSNRs calculated for each of the plurality of regions in the image in S106. Note that, for the predetermined threshold applied in S204, for example, when there is a target machine learning model, the threshold applied to the evaluation of the PSNR of each of the plurality of regions in the image in that machine learning model may be similarly applied. Also, as another example, an arbitrary value may be set by the user as the above-mentioned predetermined threshold. When the evaluation score calculation unit 110 determines in S204 that there is a PSNR less than the predetermined threshold, the process proceeds to S205. On the other hand, when the evaluation score calculation unit 110 determines in S204 that there is no PSNR less than a predetermined threshold, it ends the series of processes shown in FIG. 5. At this time, the device that is training the target model may determine whether to end or continue the training of the model based on an instruction from the user.

[0034] In S205, the small-scale learning unit 122 uses the data group classified from the series of training data in S201 to train the target model with a smaller amount of training than when training a normal model. As a specific example, in a situation where a series of training data sets are used for training by about 500 iterations, the small-scale learning unit 122 may train the target model with a learning amount of about 100 iterations. In this way, even when the target model is trained with a small amount of learning as exemplified above, it is possible to confirm the influence on the PSNR of the target area when the selected data group is applied as training data. Hereinafter, for convenience, the training with a smaller amount of learning than when training a normal model as described above is also referred to as small-scale learning.

[0035] For example, as shown in FIG. 7(a), for the model to be trained (the model to be trained in S202), small-scale learning is performed using each of the classified data groups, generating a plurality of trained models. In the example shown in FIG. 7(a), since the series of training data is classified into three data groups, three trained models are generated by performing small-scale learning using each of the three data groups. In this embodiment, for convenience, the model trained using data group A is called model A, the model trained using data group B is called model B, and the model trained using data group C is called model C.

[0036] In S206, the few-shot learning unit 122 uses the learned model obtained as a result of few-shot learning in S205 to evaluate the data for each of a plurality of evaluation metrics corresponding to the target task. As shown in Fig. 7(a), it is assumed that a series of evaluation images are divided into data suitable for evaluating each of a plurality of evaluation metrics. In this embodiment, for the sake of clarity of features, for the sake of convenience, it is assumed that evaluation is performed for each of the three evaluation metrics A to C. Also, the data for evaluating evaluation metric A is also referred to as evaluation metric A data, the data for evaluating evaluation metric B is also referred to as evaluation metric B data, and the data for evaluating evaluation metric C is also referred to as evaluation metric C data. Further, the evaluation images A, B, and C shown in Fig. 7(a) respectively correspond to examples of evaluation metric A data, evaluation metric B data, and evaluation metric C data. Note that for the evaluation metric, for example, it is possible to use the PSNR in each of a plurality of regions in the image. Fig. 7(b) schematically shows a situation in which evaluation is performed for each of evaluation metric A, evaluation metric B, and evaluation metric C for each of the models A, B, and C illustrated in Fig. 7(a). As shown in Fig. 7(b), the few-shot learning unit 122 uses each of the generated models A, B, and C to evaluate each of the evaluation images for each evaluation metric. In this case, since the data of the three types of evaluation metrics are evaluated by the three models, a total of nine evaluation results can be obtained.

[0037] In S207, the few-shot learning unit 122 determines a set of each of the plurality of data groups classified in S201 and evaluation data corresponding to each of the plurality of evaluation metrics based on the evaluation results of each of the plurality of evaluation metrics using each of the plurality of models in S206. As shown in FIG. 7(c), the small-scale learning unit 122 summarizes the evaluation results for each combination of each learned model and each evaluation index data, and identifies a model with a higher evaluation score for the evaluation index data. For example, when focusing on evaluation index A, among the three models A to C, model B has the highest evaluation score, and is marked with 〇 in the table shown in FIG. 7(c). On the other hand, for model A, the evaluation score is the lowest, and is marked with × in the table shown in FIG. 7(c). For model C, the evaluation score is between that of model B and model A, and is marked with △ in the table shown in FIG. 7(c). That is, in the example shown in FIG. 7(c), in order to improve the evaluation score for evaluation index A, data group B may be used as the learning data to be applied to the learning of the model. Similarly, in order to improve the evaluation score for evaluation index B, data group C may be used as the learning data, and in order to improve the evaluation score for evaluation index C, data group A may be used as the learning data. That is, it is found that there is a combination of the data group used for the learning of each learned model and the evaluation image corresponding to the evaluation index for which the improvement of the evaluation score is incorporated by applying the data group as the learning data.

[0038] In S208, based on the determination results of the combinations of each of the plurality of classified data groups in S207 and each evaluation data, the data group selection unit 130 selects a data group capable of improving the PSNR of the corresponding region from the plurality of data groups. In S209, the data group selection unit 130 sets the data group selected in S208 as the learning data to be applied to the learning of the model that is the learning target in S202.

[0039] In addition, when there are multiple evaluation indicators with evaluation scores below the threshold, for example, evaluation indicator data corresponding to any one of the evaluation indicators with scores below the threshold may be selected, and control may be applied such that the processes of S202 to S209 are repeatedly executed until the evaluation indicator is improved. In this case, for example, in S208, the data group selection unit 130 may select a data group capable of improving the evaluation indicator corresponding to the selected evaluation indicator data. Also, in this case, after the target evaluation indicator is improved, evaluation indicator data corresponding to other evaluation indicators with evaluation scores below the threshold may be newly selected, and control may be applied such that the processes of S202 to S209 are repeatedly executed until the other evaluation indicators are improved. Of course, the above control is merely an example and does not limit the model learning method when there are multiple evaluation indicators with evaluation scores below the threshold. As a specific example, when there are multiple evaluation indicators with evaluation scores below the threshold, after evaluation indicator data corresponding to two or more of the multiple evaluation indicators are selected, model learning may be performed. Also, in this case, in S208, the data group selection unit 130 may select a data group capable of improving each of the two or more evaluation indicators.

[0040] By repeatedly executing the processes of S202 to S209 as described above, the model is learned such that the evaluation scores of each of the series of evaluation indicators become equal to or closer to the threshold value. As described above, according to this embodiment, based on the results of few-shot learning, by selecting a data group used for model learning from a series of learning data, it becomes possible to improve the PSNR in regions of an image where the PSNR is less than a predetermined threshold. Also, for this embodiment as well, similar to the first embodiment described above, the application target is not limited to the noise reduction task. That is, for tasks having evaluation indicators that can be represented numerically, such as a classification task or a BB detection task, etc., the technology according to this embodiment can be applied.

[0041] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0042] Also, the disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) In learning of a task having a plurality of evaluation metrics, for each of the plurality of evaluation metrics of the evaluation scores of at least less than a threshold value among the evaluation scores of each of the plurality of evaluation metrics for a model that is learned based on machine learning, a search means for searching for a data group that can obtain a higher evaluation score by using it in the learning of the model from a series of learning data, a calculation means for calculating the evaluation score for each of the plurality of evaluation metrics for the model being learned based on machine learning, and for the model being learned, a selection means for selecting, from the series of learning data, a data group that can improve at least an evaluation score less than a threshold value among the evaluation scores of each of the plurality of evaluation metrics calculated by the calculation means, an information processing apparatus characterized by comprising. (Configuration 2) having a classification means for classifying the series of learning data into a plurality of data groups, and the search means searches for, from the plurality of data groups classified by the classification means, a data group that can obtain a higher evaluation score by using it in the learning of the model for each of the plurality of evaluation metrics of the evaluation scores of at least less than a threshold value among the evaluation scores of each of the plurality of evaluation metrics for a model that is learned based on machine learning, the information processing apparatus according to Configuration 1. (Configuration 3) Before the learning of the model during the learning, for each of the plurality of models learned based on each of the plurality of data groups classified by the classification means, for each of the plurality of evaluation metrics, a data group that can obtain a higher evaluation score by being used for the learning of the model is searched from the plurality of data groups. The selection means, based on the search result by the search means, selects, from among the plurality of data groups classified by the classification means, a data group for which a higher evaluation score can be obtained for at least an evaluation metric corresponding to an evaluation score less than a threshold among the evaluation scores of each of the plurality of evaluation metrics calculated by the calculation means for the model during the learning. The information processing apparatus according to Configuration 2 is characterized by this. (Configuration 4) The classification means randomly classifies the series of learning data into the plurality of data groups. The information processing apparatus according to Configuration 3 is characterized by this. (Configuration 5) For the model during the learning, for each of the plurality of data groups classified by the classification means, for each of the plurality of evaluation metrics calculated by the calculation means, for an evaluation metric corresponding to an evaluation score less than a threshold among the evaluation scores of each of the plurality of evaluation metrics obtained by performing learning with a learning amount smaller than the learning amount when learning is performed with the series of learning data, a data group that can obtain a higher evaluation score by being used for the learning of the model is searched from the plurality of data groups classified by the classification means. The selection means selects the data group searched by the search means from among the plurality of data groups classified by the classification means. The information processing apparatus according to Configuration 2 is characterized by this. (Configuration 6) The classification means classifies the series of learning data into the plurality of data groups based on a plurality of types of features of the learning data. The information processing apparatus according to Configuration 5 is characterized by this. (Method 1) A control method for an information processing apparatus, in learning of a task with a plurality of evaluation metrics, for each of a plurality of evaluation metrics of a model that is learned based on machine learning, for an evaluation metric corresponding to an evaluation score that is at least less than a threshold among the evaluation scores of each of the plurality of evaluation metrics, a search step of searching for a data group that can obtain a higher evaluation score by using it for learning of the model from a series of learning data, a calculation step of calculating the evaluation score for each of the plurality of evaluation metrics for a model being learned based on machine learning, and a selection step of selecting, from the series of learning data, a data group that can improve at least an evaluation score that is less than the threshold among the evaluation scores of each of the plurality of metrics calculated in the calculation step for the model being learned. A control method for an information processing apparatus, characterized by including these steps. (Program 1) A program that causes a computer, in learning of a task with a plurality of evaluation metrics, for each of a plurality of evaluation metrics of a model that is learned based on machine learning, for an evaluation metric corresponding to an evaluation score that is at least less than a threshold among the evaluation scores of each of the plurality of evaluation metrics, a search means for searching for a data group that can obtain a higher evaluation score by using it for learning of the model from a series of learning data, a calculation means for calculating the evaluation score for each of the plurality of evaluation metrics for a model being learned based on machine learning, and a selection means for selecting, from the series of learning data, a data group that can improve at least an evaluation score that is less than the threshold among the evaluation scores of each of the plurality of metrics calculated by the calculation means for the model being learned, and causing the computer to function as an information processing apparatus having these means.

Explanation of Signs

[0043] 1000 Learning Data Selection Device 110 Evaluation Score Calculation Unit 120 Search Unit 130 Data Group Selection Unit

Claims

1. In the learning of a task with a plurality of evaluation metrics, For at least one evaluation metric whose evaluation score is less than a threshold among the evaluation scores of each of the plurality of evaluation metrics for a model that is learned based on machine learning, a search means for searching for a data group that can obtain a higher evaluation score by using it in the learning of the model from a series of learning data; A calculation means for calculating the evaluation score for each of the plurality of evaluation metrics for a model being learned based on machine learning; A selection means for selecting, from the series of learning data, a data group that can improve at least one evaluation score that is less than a threshold among the evaluation scores of each of the plurality of evaluation metrics calculated by the calculation means for the model being learned, based on the search result by the search means; An information processing apparatus, characterized by comprising the above.

2. It has a classification means for classifying the series of learning data into a plurality of data groups, The search means searches for a data group that can obtain a higher evaluation score by using it in the learning of the model, for at least one evaluation metric whose evaluation score is less than a threshold among the evaluation scores of each of the plurality of evaluation metrics for a model that is learned based on machine learning, from the plurality of data groups classified by the classification means. The information processing apparatus according to claim 1, characterized by the above.

3. The search means, prior to the learning of the model being learned, for each of a plurality of models learned based on each of the plurality of data groups classified by the classification means, searches for a data group that can obtain a higher evaluation score by using it in the learning of the model, for each of the plurality of evaluation metrics, from the plurality of data groups. The selection means selects, based on the search result by the search means, from among the plurality of data groups classified by the classification means, a data group for which a higher evaluation score is obtained for at least one evaluation metric whose evaluation score is less than a threshold among the evaluation scores of each of the plurality of evaluation metrics calculated by the calculation means for the model being learned. The information processing apparatus according to claim 2, characterized by the above.

4. The information processing apparatus according to claim 3, characterized in that the classification means randomly classifies the series of learning data into the plurality of data groups.

5. The search means, for each of the plurality of data groups classified by the classification means, for the model being learned, performs learning with an amount of learning smaller than the amount of learning when learning is performed using the series of learning data, and for each of the plurality of models obtained thereby, for each of the evaluation indicators among the plurality of evaluation indicators calculated by the calculation means, for the evaluation indicator corresponding to an evaluation score less than the threshold value among the evaluation scores, searches for a data group that can obtain a higher evaluation score by using it for learning of the model from the plurality of data groups classified by the classification means, The selection means selects the data group searched by the search means from among the plurality of data groups classified by the classification means The information processing apparatus according to claim 2, characterized in that.

6. The classification means classifies the series of learning data into the plurality of data groups based on a plurality of types of features of the learning data, The information processing apparatus according to claim 5, characterized in that.

7. A control method for an information processing apparatus, In learning of a task having a plurality of evaluation indicators, A search step of searching for a data group that can obtain a higher evaluation score by using it for learning of a model, from among a series of learning data, for an evaluation indicator corresponding to an evaluation score that is at least less than a threshold value among the evaluation scores of each of the plurality of evaluation indicators for a model learned based on machine learning, A calculation step of calculating the evaluation score for each of the plurality of evaluation indicators for a model being learned based on machine learning, A selection step of selecting, from among the series of learning data, a data group that can improve at least an evaluation score less than the threshold value among the evaluation scores of each of the plurality of indicators calculated in the calculation step for the model being learned, based on the search result in the search step, A control method for an information processing apparatus, characterized by including.

8. A computer, In learning of a task having a plurality of evaluation indicators, Search means for searching for a data group that can obtain a higher evaluation score by using it for learning of a model, from among a series of learning data, for an evaluation indicator corresponding to an evaluation score that is at least less than a threshold value among the evaluation scores of each of the plurality of evaluation indicators for a model learned based on machine learning, Calculation means for calculating the evaluation score for each of the plurality of evaluation indicators for the model being learned based on machine learning; Selection means for selecting, from among the series of learning data, a data group capable of improving an evaluation score that is at least less than a threshold among the evaluation scores of each of the plurality of indicators calculated by the calculation means for the model being learned, based on the search result by the search means; A program for causing an information processing apparatus to function as an information processing apparatus characterized by including the above.

Images