Prediction device, prediction method, and prediction program

Abstract

A prediction device (10) has a feature extraction unit (12), a mapping unit (14), and an accuracy prediction unit (15). The feature extraction unit (12) extracts features of a data set. The mapping unit (14) maps the features extracted by the feature extraction unit (12) to parameters of a regression function for predicting the accuracy of a model. The accuracy prediction unit (15) predicts the accuracy of the model by using the regression function constructed on the basis of the parameters mapped by the mapping unit (14).

Description

Prediction device, prediction method, and prediction program

[0001] The present invention relates to a prediction device, a prediction method, and a prediction program.

[0002] A technique known as Neural Scaling Laws (NSLs) has been known in the field of machine learning. NSLs aim to express the relationship between the amount of training data and the accuracy of a machine learning model (hereinafter simply referred to as a model) as a regression function in machine learning tasks.

[0003] For example, the Power Law shown in equation (1) is known as a regression function that expresses the relationship between the amount of training data and the accuracy of a model.

[0004]

[0005] n in equation (1) is the amount of training data (e.g., the number of images in an image recognition task). In this case, the goal of NSLs is to find α in equation (2), i.e., the three parameters in equation (1), so that equation (1) best matches the measured accuracy of the model.

[0006]

[0007] The Rosenfeld method is known as one type of NSL (see, for example, Non-Patent Document 1). For example, in a person matching task using images, the Rosenfeld method is performed in the following procedure. The application location is the location where the images to be used in the person matching task are collected. (Step 1) A model is trained using training images collected from the application location. (Step 2) Verification is performed using the trained model to measure accuracy. (Step 3) Steps 1 and 2 are repeated a certain number of times with different numbers of training images. (Step 4) The parameters of the regression function are updated using the number of training images and the measured accuracy (the problem of minimizing squared error is solved using the steepest descent method, etc.).

[0008] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020.

[0009] However, conventional techniques have the problem that they require a large amount of resources to predict the accuracy of the model.

[0010] For example, procedures 1 and 2 of the Rosenfeld method require a large amount of calculation, and a graphics processing unit (GPU) is often used.

[0011] In order to solve the above-mentioned problems and achieve the objectives, the prediction device is characterized by having a feature extraction unit that extracts features of a dataset, a parameter acquisition unit that acquires parameters of a regression function that predicts the accuracy of a machine learning model based on the features extracted by the feature extraction unit, and an accuracy prediction unit that predicts the accuracy of the machine learning model using the regression function constructed based on the parameters acquired by the parameter acquisition unit.

[0012] According to the present invention, it is possible to reduce resources required to predict the accuracy of a model.

[0013] FIG. 1 is a diagram illustrating an example of the configuration of a prediction device according to a first embodiment. FIG. 2 is a diagram illustrating a method for determining a regression function of parameters. FIG. 3 is a flowchart illustrating the flow of adjustment processing according to the first embodiment. FIG. 4 is a flowchart illustrating the flow of prediction processing according to the first embodiment. FIG. 5 is a diagram illustrating an example of the configuration of a prediction device according to a second embodiment. FIG. 6 is a flowchart illustrating the flow of adjustment processing according to the second embodiment. FIG. 7 is a flowchart illustrating the flow of prediction processing according to the second embodiment. FIG. 8 is a diagram illustrating an example of a computer that executes a program.

[0014] Hereinafter, embodiments of a prediction device, a prediction method, and a prediction program according to the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0015] [First embodiment] [Configuration of first embodiment] The configuration of a prediction device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a prediction device according to the first embodiment.

[0016] The prediction device 10 predicts the accuracy of a model. In this case, the prediction device 10 constructs a regression function for predicting the accuracy of the model from the amount of training data. The prediction device 10 also calculates parameters for constructing the regression function.

[0017] Let's consider a case where the machine learning task is person identification (Person Re-ID) using image recognition. In this case, the model outputs information to identify the person in the input image. The training data used to train the model are images taken at a specified location. The model is, for example, a neural network.

[0018] The images to be recognized and the images used for training may be taken in different locations, and the characteristics of the images may differ depending on the location where they were taken.

[0019] For example, at a ski resort, it is difficult to identify a person based on their face because their face is hidden by a helmet and goggles, whereas at a ski resort, a person can be identified by, for example, the color of their ski wear.

[0020] Therefore, if images taken at a ski resort are used as training data, the model will tend to identify people by focusing on their clothing.

[0021] If such a model were used without retraining to match people in images taken in a company office, the accuracy would likely decrease because many people in the office wear similar clothing (e.g., dark suits), making it difficult to identify people by their clothing.

[0022] In this case, it is important to understand how much training data is needed to efficiently retrain the model to suit the company's office location.

[0023] In order to obtain the relationship between the amount of training data and accuracy, it is possible to use the Rosenfeld method described in Non-Patent Document 1. However, as mentioned above, the Rosenfeld method has the problem of requiring a large amount of resources.

[0024] In contrast, the prediction device 10 can reduce resource consumption by making adjustments in advance, and can determine the relationship between the amount of training data and accuracy according to the characteristics of the training data.

[0025] For example, the advance adjustment may be performed in an environment that allows long-term processing, such as an overnight batch process, etc. This allows the prediction device 10 to determine the relationship between the amount of training data and accuracy according to the characteristics of the training data, even in an environment with limited available resources (e.g., an environment in which a GPU is not used).

[0026] When the training data is images, the characteristics of the training data vary depending on the location where the images were taken (for example, a ski resort and an office), the number of cameras, the camera placement, the camera settings, the shooting environment, and so on.

[0027] In this embodiment, the features of the training data are referred to as location features for convenience. However, location features are not limited to location-dependent features, but include all features found in the training data itself and the environment in which the training data is collected.

[0028] Furthermore, although an example in which the machine learning task is person matching will be described here, this embodiment can also be applied to tasks other than person matching as long as the machine learning task involves collecting training data.

[0029] This embodiment can be applied to any object matching, including vehicle matching, in addition to person matching. Object matching is a task of determining whether objects (people, vehicles, etc.) appearing in multiple images are the same. In particular, vehicle matching may be used by police to find getaway vehicles.

[0030] Furthermore, this embodiment can be applied to image recognition tasks such as image classification and object detection in addition to object matching. Image classification is a task of identifying objects (dogs, cats, etc.) in an image and classifying the image. Object detection is a task of detecting specific objects from an image. Object detection can find vehicles, pedestrians, obstacles, etc. from frames of a video captured by an in-vehicle camera, for example.

[0031] Furthermore, this embodiment is applicable to speech recognition in which speech is collected as training data, and natural language recognition in which text is collected as training data.

[0032] 1 , the prediction device 10 includes a parameter update unit 11, a feature extraction unit 12, an adjustment unit 13, a mapping unit 14, and an accuracy prediction unit 15. The parameter update unit 11 also includes a training unit 111 and a verification unit 112.

[0033] The adjustment dataset and the prediction dataset are collections of data, and have different location characteristics. For example, the adjustment dataset is a collection of images of people taken at a ski resort. For example, the prediction dataset is a collection of images of people taken in an office. The adjustment dataset may also include images obtained by synthesis.

[0034] The training data is extracted from the training data set or the prediction data set. The training data set includes data that has been previously labeled (e.g., a person's identification ID).

[0035] The parameter update unit 11 measures the accuracy of the model according to the amount of training data and updates the parameters of the regression function. The regression function and parameters are as shown in Equations (1) and (2). The parameter update unit 11 may update the parameters using the Rosenfeld method.

[0036] The training unit 111 trains the model using a predetermined amount of training data extracted from the training dataset, using a known method such as backpropagation.

[0037] The verification unit 112 verifies the accuracy of the model trained by the training unit 111. For example, the verification unit 112 verifies the accuracy of the model in identifying whether a query and a gallery included in the training dataset are data with the same label.

[0038] The adjustment unit 13 adjusts the slope and intercept for mapping by the mapping unit 14 based on the features of the adjustment dataset and the actual measurement value of the accuracy of the model trained using the adjustment dataset. The adjustment dataset is an example of a first dataset.

[0039] The adjustment unit 13 can obtain the actual measurement value of the accuracy of the model from the parameter update unit 11. The adjustment unit 13 also obtains a mapper that performs mapping according to the slope and intercept. Details of the adjustment unit 13 and the mapping unit 14 will be described later.

[0040] The feature extraction unit 12 extracts location features from the dataset (the adjustment dataset or the prediction dataset) (location embedding). Here, location features when the training data is an image will be described.

[0041] Each piece of data included in the dataset is represented by an image x, a person label y that identifies a person, and a camera label z that identifies the camera that captured the image, where x∈[0, 1]. ｃ×ｈ×ｗ , y∈{1,...,Y}, z∈{1,...,Z}.

[0042] h and w are the number of vertical and horizontal pixels of the image, respectively. c is the pixel value (RGB value) of three channels. Y is the total number of people. Z is the total number of cameras.

[0043] If the location is l, the data at the lth location is given by the probability distribution p ｌ (x, y, z). Differences in this probability distribution cause differences in location characteristics.

[0044] The feature extraction unit 12 extracts one or more location features from the dataset based on x, y, and z. ｌ is expressed as equation (3).

[0045]

[0046] The feature extraction unit 12 extracts at least one of the five location features. The size of the vector in equation (3) changes depending on the number of extracted location features. Each location feature will be explained below. It is assumed that the camera that captured the images included in the dataset and the people in the images are known.

[0047] [Location Feature 1: ACN (Actual Camera Number)] ｌ ＡＣＮ is a feature that grows with the actual number of cameras (ACN) that captured the image. ｌ ＡＣＮ is different from the total number of cameras installed at the location. ｌ ＡＣＮ The larger the , the more training data is likely to be needed to improve the accuracy of the model.

[0048] The feature extraction unit 12 extracts s ​​as shown in equation (4). ｌ ＡＣＮ where |S| in equation (4) means that elements included in S are counted without allowing duplication. Also, ^m (^ immediately above m) is a hyperparameter indicating the number of Monte Carlo samples.

[0049]

[0050] For example, let us assume that, among images taken at a location where Z=10, i.e., 10 cameras are arranged, the third and seventh cameras take fewer than 10 images, and the remaining eight cameras take several hundred images. In such a case, the actual number of cameras that took the images is considered to be about 8, not 10. For example, the influence of the third and seventh cameras in equation (4) becomes negligible.

[0051] The feature extraction unit 12 ｌ ＡＣＮ Instead of or ｌ ＡＣＮ In addition to the location feature s, which increases with the number of cameras (CN) that captured the image, ｌ ＣＮmay be extracted. ｌ ＣＮ is equal to Z.

[0052] [Location feature 2: PIV (Person-wise Image Volume)] ｌ ＰＩＶ is a feature that increases with the number of people in the image (PIV). ｌ ＰＩＶ The larger the , the more training data is likely to be needed to improve the accuracy of the model.

[0053] The feature extraction unit 12 extracts s ​​as shown in equation (5). ｌ ＰＩＶ However, as in equation (4), |S| in equation (5) means that elements included in S are counted without allowing duplication.

[0054]

[0055] [Location Feature 3: LM (Luminance Mean)] s ｌ ＬＭ is a feature that increases with the brightness (LM) of the image. ｌ ＬＭ It is believed that the larger the , the more accurate the model can be with less training data.

[0056] The feature extraction unit 12 extracts s ​​as shown in equation (6). ｌ ＬＭ Calculate.

[0057]

[0058] Here, Lum is a function that converts RGB values ​​into brightness, and is expressed as in equation (7).

[0059]

[0060] [Location Feature 4: LD (Luminance Deviation)] s ｌ ＬＤ is a feature that increases with the brightness deviation (LD) of the image. ｌ ＬＤIt is believed that the larger LD is, the more unique the clothing of people in the image is, and the easier it is to identify each person.

[0061] The feature extraction unit 12 extracts s ​​as shown in equation (8). ｌ ＬＤ Calculate.

[0062]

[0063] [Location Feature 5: GPN (Gallery Person Number)] ｌ ＧＰＮ is a feature that grows with the number of people (GPN) in the images included in the gallery dataset. ｌ ＧＰＮ The larger the , the more training data is likely to be needed to improve the accuracy of the model.

[0064] The feature extraction unit 12 extracts s ​​as in equation (9) or (10). ｌ ＰＩＶ However, as in equation (4), |S| in equations (9) and (10) means that elements included in S are counted without allowing duplication.

[0065]

[0066]

[0067] The base of the logarithm in equation (10) is not limited to 10, but may be, for example, the base of a natural logarithm. ｌ Unlike the hyperparameter ^m, is the number of images included in the gallery dataset (labeled images) collected at that location.

[0068] The feature extraction unit 12 can further correct the calculated location feature by a scaling parameter λ or a curvature v. For example, the feature extraction unit 12 can further correct the location feature ^s calculated by the method described above. ｌ λ(^s ｌ ) ν Correct it as follows.

[0069] The scales of the location features may differ from each other. λ is a value that keeps the scales of the location features roughly within the range of [0, 1]. Furthermore, the curvature ν can improve the accuracy of the linear mapping.

[0070] The mapping unit 14 maps the features extracted by the feature extraction unit 12 to parameters of a regression function that predicts the accuracy of the model. The mapping unit 14 is an example of a parameter acquisition unit. That is, the mapping unit 14 acquires the parameters by mapping.

[0071] The mapping unit 14 uses the slope and intercept adjusted by the adjustment unit 13 to map the characteristics of the prediction dataset, which has different characteristics from the adjustment dataset, onto the parameters of the regression function.

[0072] The mapper is W ｋ ・s+b ｋ s is as shown in equation (3). The adjustment unit 13 determines the gradient matrix W such that equation (11) is minimized. ｋ and intercept vector b ｋ The gradient matrix W ｋ The size of the intercept vector b is A × I. ｋ The size of the vector is A. A is the number of parameters of the regression function. I is the number of location features extracted by the feature extraction unit 12 (1 to 5 in this embodiment). k is the number of top data used to verify accuracy (rank-k accuracy).

[0073]

[0074] The regression function is not limited to the Power Law shown in Equation (1). The regression function may be a TV (Terminal Velocity) function. That is, the mapping unit 14 maps the location features to parameters of the regression function, which is a TV function.

[0075] The original TV function is the initial acceleration a ０ , terminal velocity v Ｔ The free fall is expressed as in equation (12).

[0076]

[0077] In this embodiment, a TV function such as that shown in equation (13) is used, in which the initial acceleration of the original TV function is replaced with a parameter obtained by mapping, and the terminal velocity is set to 100. Here, ^v is the predicted accuracy, and n is the amount of training data.

[0078]

[0079] The TV function has one parameter, while the Power Law has three. A small number of parameters prevents overfitting. The TV function is also effective in predicting the accuracy of a model when there is a small amount of training data.

[0080] The mapper 14 uses the adjusted mapper (W ｋ , b ｋ ) to map the location feature s to the parameter α as shown in equation (14).

[0081]

[0082] The accuracy prediction unit 15 predicts the accuracy of the model using a regression function constructed based on the parameters mapped by the mapping unit 14. If the regression function is a TV function, the accuracy prediction unit 15 predicts the accuracy of the model using the TV function constructed based on the parameters.

[0083] In this way, the accuracy prediction unit 15 can obtain a regression function suitable for the prediction data set using the parameters. Fig. 2 is a diagram for explaining how to obtain the regression function.

[0084] 2, during tuning, a parameter update unit 11 updates the parameters of the regression function, and an adjustment unit 13 adjusts a mapper for mapping location features to the parameters.

[0085] Then, at the time of prediction, the accuracy prediction unit 15 can obtain a regression function using the parameters mapped from the features of the prediction dataset, without updating the parameters of the regression function.

[0086] [Processing of First Embodiment] The flow of adjustment processing of the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of adjustment processing of the first embodiment.

[0087] 3, the prediction device 10 first measures the accuracy of the machine learning model using an adjustment dataset (step S101). Next, the prediction device 10 updates the parameters of the regression function for predicting accuracy based on the measured accuracy (step S102).

[0088] Then, the prediction device 10 adjusts a mapper for obtaining parameters of a regression function from the location features extracted from the adjustment dataset (step S103).

[0089] The flow of the prediction process according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the prediction process according to the first embodiment.

[0090] 4, the prediction device 10 first extracts location features from the prediction dataset (step S201), and then calculates parameters of a regression function from the extracted location features using a mapper (step S202).

[0091] Then, the prediction device 10 predicts the accuracy of the machine learning model for the prediction dataset using the regression function constructed using the calculated parameters (step S203).

[0092] [Effects of the First Embodiment] As described above, the prediction device 10 includes the feature extraction unit 12, the mapping unit 14, and the accuracy prediction unit 15. The feature extraction unit 12 extracts features of a dataset. The mapping unit 14 maps the features extracted by the feature extraction unit 12 to parameters of a regression function that predicts the accuracy of the model. The accuracy prediction unit 15 predicts the accuracy of the model using the regression function constructed based on the parameters mapped by the mapping unit 14.

[0093] The prediction device 10 can predict the accuracy of a model without actually training the model or measuring the accuracy, thereby reducing the resources required for predicting the accuracy of the model.

[0094] Machine learning tasks are sometimes performed by renting a GPU server. The rental cost of the GPU server varies depending on the usage time. This embodiment can predict in advance how much training data is needed to produce a model with the desired accuracy, thereby assisting in estimating the rental cost.

[0095] The adjustment unit 13 adjusts the slope and intercept for mapping by the mapping unit 14 based on the features of the first dataset and the actual measurement value of the accuracy of the model trained using the first dataset. The mapping unit 14 maps the features of the second dataset, which have different features from the first dataset, to the parameters of the regression function using the slope and intercept adjusted by the adjustment unit 13. This allows the prediction device 10 to easily perform mapping by linear mapping.

[0096] The mapping unit 14 maps the features to parameters of a regression function, which is a TV function. The accuracy prediction unit 15 predicts the accuracy of the model using the TV function constructed based on the parameters. This allows the prediction device 10 to prevent the mapper from overfitting the location features and effectively predict the accuracy of the model when there is little training data.

[0097] The feature extraction unit 12 corrects the extracted features using a scaling parameter and curvature, thereby enabling the prediction device 10 to align the scales of multiple location features and improve the accuracy of the linear mapping using the curvature.

[0098] The feature extraction unit 12 extracts features corresponding to at least one of the actual number of cameras that captured the image, the number of people in the image, the brightness of the image, and the deviation of the brightness of the image from a dataset of images in which the cameras that captured the image and the people in the image are known. This enables the prediction device 10 to extract location features that have a significant impact on accuracy from the image.

[0099] Second Embodiment As in the first embodiment, estimating the parameter α of a prediction feature (a feature of a prediction data set) when the prediction feature is not surrounded by adjustment features (a feature of an adjustment data set) is called extrapolation. That is, the prediction device of the first embodiment can obtain the parameters of a regression function by extrapolation.

[0100] In contrast, estimating the parameter α of a prediction feature (a feature of the prediction dataset) when the prediction feature is surrounded by adjustment features (a feature of the adjustment dataset) is called interpolation. Generally, interpolation is achieved by simpler processing than extrapolation. When the adjustment dataset and the prediction dataset are the same, the prediction device of the second embodiment can determine the parameters of the regression function by interpolation without having to perform extrapolation.

[0101] The configuration of a prediction device according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of a prediction device according to the second embodiment. Note that differences between the second embodiment and the first embodiment will be described here, and descriptions of common parts will be omitted as appropriate.

[0102] 5 , the prediction device 10a of the second embodiment differs from the prediction device 10 of the first embodiment in that it includes a search unit 14a. The search unit 14a is an example of a parameter acquisition unit. That is, the search unit 14a acquires parameters through a search.

[0103] In the second embodiment, the extended dataset has the same functions as the adjustment dataset and the prediction dataset. For example, a part of the extended dataset (adjustment subset) is the adjustment dataset. Also, a part of the extended dataset (prediction subset) is the prediction dataset. In this case, there is no overlap between the adjustment subset and the prediction subset. In other words, the intersection between the adjustment subset and the prediction subset is an empty set.

[0104] In the second embodiment, the parameter update unit 11 measures the accuracy of the model according to the amount of the extended data set and updates the parameters of the regression function. The processing of the parameter update unit 11 is the same as that of the first embodiment. However, the parameter update unit 11 may update parameters according to each of multiple subsets (adjustment subsets) of the extended data set. In this case, parameters according to each of the multiple adjustment subsets are obtained.

[0105] The feature extraction unit 12 extracts location features of a part (prediction subset) of the extended data set. The feature extraction unit 12 also extracts location features of each of the adjustment subsets used by the parameter update unit 11. Note that the location features of the prediction subset are surrounded by the location features of the adjustment subset.

[0106] The search unit 14a then identifies an adjustment subset from among the adjustment subsets whose location features are closest to those of the prediction subset (for example, whose vectors have the largest Euclidean distance). The search unit 14a obtains parameters corresponding to the identified adjustment subset as a search result.

[0107] The accuracy prediction unit 15 predicts the accuracy of the model using a regression function constructed based on the parameters of the search results.

[0108] [Processing of Second Embodiment] The flow of adjustment processing of the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of adjustment processing of the second embodiment.

[0109] 6, the prediction device 10a first measures the accuracy of the machine learning model using the expanded data set (step S101a), and then updates the parameters of the regression function for predicting the accuracy based on the measured accuracy (step S102a).

[0110] The flow of the prediction process according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of the prediction process according to the second embodiment.

[0111] 7, the prediction device 10a first extracts location features from a portion of the extended data set (step S201a), and then searches for parameters of a regression function based on the extracted location features (step S202a).

[0112] Then, the prediction device 10a predicts the accuracy of the machine learning model for a part of the extended data set using a regression function constructed using the parameters obtained by the search (step S203a).

[0113] [System Configuration, etc.] The components of each device shown in the figures are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0114] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0115] [Program] In one embodiment, the prediction device 10 can be implemented by installing a program that executes the above-described learning process as package software or online software on a desired computer. For example, by executing the above-described program on an information processing device, the information processing device can function as the prediction device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handy-phone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0116] The prediction device 10 may also be implemented as a server device that provides services related to the above-described processing to a client terminal device used by a user. For example, the server device may be implemented as a server device that receives a prediction dataset as input and outputs the accuracy of a predicted model or a regression function for predicting the accuracy. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above-described processing through outsourcing.

[0117] 8 is a diagram showing an example of a computer that executes a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0118] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0119] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the prediction device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the prediction device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0120] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0121] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0122] 10, 10a Prediction device 11 Parameter update unit 12 Feature extraction unit 13 Adjustment unit 14 Mapping unit 14a Search unit 15 Accuracy prediction unit 111 Training unit 112 Verification unit

Claims

1. A prediction device comprising: a feature extraction unit that extracts features of a dataset; a parameter acquisition unit that acquires parameters of a regression function that predicts the accuracy of a machine learning model based on the features extracted by the feature extraction unit; and an accuracy prediction unit that predicts the accuracy of the machine learning model using the regression function constructed based on the parameters acquired by the parameter acquisition unit.

2. The prediction device according to claim 1, characterized in that the parameter acquisition unit maps the features extracted by the feature extraction unit to the parameters.

3. The prediction device of claim 2, further comprising an adjustment unit that adjusts a slope and an intercept for mapping by the parameter acquisition unit based on the features of a first dataset and an actual measured value of the accuracy of the machine learning model trained using the first dataset, and the parameter acquisition unit uses the slope and the intercept adjusted by the adjustment unit to map the features of a second dataset, the features of which differ from those of the first dataset, onto the parameters of the regression function.

4. The prediction device described in claim 2, characterized in that the parameter acquisition unit maps the features to parameters of the regression function, which is a TV function, and the accuracy prediction unit predicts the accuracy of the machine learning model using a TV function constructed based on the parameters.

5. The prediction device according to claim 2, wherein said feature extraction unit corrects the extracted features using a scaling parameter and a curvature.

6. The prediction device according to claim 1, characterized in that the feature extraction unit extracts features corresponding to at least one of the actual number of cameras that captured the images, the number of people appearing in the images, the brightness of the images, and the deviation of the brightness of the images from a dataset of images in which the cameras that captured the images and the people appearing in the images are known.

7. A prediction method executed by a prediction device, comprising: a feature extraction step of extracting features of a dataset; a parameter acquisition step of acquiring parameters of a regression function that predicts the accuracy of a machine learning model based on the features extracted by the feature extraction step; and an accuracy prediction step of predicting the accuracy of the machine learning model using the regression function constructed based on the parameters acquired by the parameter acquisition step.

8. A prediction program that causes a computer to execute the following steps: a feature extraction step for extracting features of a dataset; a parameter acquisition step for acquiring parameters of a regression function for predicting the accuracy of a machine learning model based on the features extracted by the feature extraction step; and an accuracy prediction step for predicting the accuracy of the machine learning model using the regression function constructed based on the parameters acquired by the parameter acquisition step.

Images