Learning device, learning method, and learning program

Abstract

To allow for improving accuracy of a model.SOLUTION: A learning device disclosed herein comprises: a division unit configured to divide predetermined learning data with features to be learned by a model into a plurality of sets in chronological order; and a training unit configured to randomly select multiple sets from among the sets obtained by the division by the division unit and make the model learn the features of the learning data included in the selected sets.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a learning device, a learning method, and a learning program.

Background Art

[0002] In recent years, techniques have been proposed for causing various models such as SVM (Support Vector Machine) and DNN (Deep Neural Network) to perform various predictions and classifications by learning the features possessed by learning data. As an example of such a learning method, a technique has been proposed for dynamically changing the learning mode of learning data according to the value of hyperparameters and the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above prior art, the accuracy of the model cannot always be improved.

[0005] For example, in the above prior art, the learning data to be the learning target of features is only dynamically changed according to the value of hyperparameters and the like. Therefore, when the value of the hyperparameter is not appropriate, the accuracy of the model may not be improved.

[0006] The present application has been made in view of the above, and an object thereof is to provide a learning device, a learning method, and a learning program capable of improving the accuracy of a model.

Means for Solving the Problems

[0007] The learning device according to the present invention is characterized by having a division unit that divides predetermined learning data for which a model learns features into a plurality of sets in chronological order, and a learning unit that randomly selects a plurality of sets from the sets obtained by the division unit and causes the model to learn the features of the learning data contained in the selected sets. [Effects of the Invention]

[0008] According to one embodiment, it is possible to improve the accuracy of the model. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the processing performed by the information providing device according to this embodiment. [Figure 2] Figure 2 shows an example of an information processing system according to the present invention. [Figure 3] Figure 3 shows an overview of the processing performed by the information processing device according to this embodiment. [Figure 4] Figure 4 shows an example of how the dataset is divided for each trial when it is split according to its intended use. [Figure 5] Figure 5 shows an example of the configuration of an information processing device according to the embodiment. [Figure 6] Figure 6 is an explanatory diagram that conceptually illustrates the partitioning of the dataset. [Figure 7] Figure 7 is Figure (1) showing the change in model performance when the first and fourth optimization algorithms are executed. [Figure 8] Figure 8 is Figure (2) showing the change in model performance when the first and fourth optimization algorithms are executed. [Figure 9] Figure 9 shows a comparative example comparing the performance of models depending on the combination of the first and fourth optimization algorithms. [Figure 10] Figure 10 shows an example of the second optimization algorithm. [Figure 11]FIG. 11 is a diagram showing an example of a third optimization algorithm. [Figure 12] FIG. 12 is a diagram showing a comparative example in which the performance of the model is compared for each shuffle buffer size. [Figure 13] FIG. 13 is a diagram showing an example of condition information regarding a fifth optimization algorithm. [Figure 14] FIG. 14 is a diagram showing an example of a fifth optimization algorithm. [Figure 15] FIG. 15 is a diagram showing an example of an optimization algorithm for optimizing a mask target. [Figure 16] FIG. 16 is a diagram showing a comparative example in which the accuracy of the model is compared between the case where optimization of the mask target is executed and the case where optimization of the mask target is not executed. [Figure 17] FIG. 17 is a diagram showing a configuration example of an execution control device according to an embodiment. [Figure 18] FIG. 18 shows an example of a model architecture storage unit according to an embodiment. [Figure 19] FIG. 19 is a diagram showing an example of a model architecture in which information indicating an arithmetic unit to be executed is associated. [Figure 20] FIG. 20 is a diagram showing the improvement status of performance by an experiment targeting a model for multi-class classification. [Figure 21] FIG. 21 is a diagram showing an example of the experimental content of an experiment conducted on a model corresponding to service SV1. [Figure 22] FIG. 22 is a diagram showing the improvement status of performance by an experiment targeting a model for two-class classification. [Figure 23] FIG. 23 is a diagram showing an example of the experimental content of an experiment conducted on a model corresponding to service SV6. [Figure 24] FIG. 24 is a flowchart showing an example of the flow of fine-tuning according to an embodiment. [Figure 25A]FIG. 25A is a diagram showing Comparative Example (1) in which the accuracy of a model is compared between the case where fine-tuning according to an embodiment is performed and the case where fine-tuning according to the embodiment is not performed. [Figure 25B] FIG. 25B is a diagram showing Comparative Example (2) in which the accuracy of a model is compared between the case where fine-tuning according to an embodiment is performed and the case where fine-tuning according to the embodiment is not performed. [Figure 25C] FIG. 25C is a diagram showing Comparative Example (3) in which the accuracy of a model is compared between the case where fine-tuning according to an embodiment is performed and the case where fine-tuning according to the embodiment is not performed. [Figure 26] FIG. 26 is a hardware configuration diagram showing an example of a computer.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments for carrying out the apparatus, method, and program according to the present application (specifically, a learning apparatus, a learning method, a learning program / a classification apparatus, a classification method, a classification program / an execution control apparatus, an execution control method, an execution control program) (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the learning apparatus, learning method, and learning program according to the present application are not limited by this embodiment. Also, each embodiment can be appropriately combined within a range that does not conflict with the processing content. In addition, in the following embodiments, the same parts are denoted by the same reference numerals, and duplicate explanations are omitted.

[0011] 〔1. About Embodiments〕 In the following embodiments, the information processing executed by the information processing apparatus 100, which is an example of the learning apparatus and the classification apparatus, and the information processing executed by the execution control apparatus 200 will be mainly focused on and described. On the other hand, the processing executed by the information providing apparatus 10 included in the system having the information processing apparatus 100 and the execution control apparatus 200 will be first described as a premise of the information processing according to the embodiment.

[0012] [2. Configuration of the Information Provision System] Figure 1 shows an example of processing performed by the information providing device 10 according to the embodiment. In the example of Figure 1, the information processing device 100 and the execution control device 200 are not shown, but an information providing system 1 is shown as an example of a system having these devices.

[0013] As shown in Figure 1, the information provision system 1 comprises an information provision device 10, a model generation server 2, and a terminal device 3. The information provision system 1 may also have multiple model generation servers 2 and multiple terminal devices 3. Furthermore, the information provision device 10 and the model generation server 2 may be implemented using the same server device or cloud system. Here, the information provision device 10, the model generation server 2, and the terminal device 3 are connected via a network N, enabling communication by wired or wireless means.

[0014] The information providing device 10 is an information processing device that performs an indicator generation process to generate generation indicators, which are indicators (i.e., recipes for the model) in model generation, and a model generation process to generate a model according to the generation indicators, and provides the generated generation indicators and model. It can be implemented, for example, by a server device or a cloud system.

[0015] Model generation server 2 is a generation device that generates a model trained on the features of the training data, and can be implemented as a server device or a cloud system, for example. For example, model generation server 2 accepts a configuration file as a model generation indicator, specifying the type and behavior of the model to be generated and how to train the features of the training data, and then automatically generates a model according to the accepted configuration file. Model generation server 2 may train the model using any model training method. Alternatively, model generation server 2 may be one of various existing services such as AutoML.

[0016] Terminal device 3 is a terminal device used by user U, and can be implemented as, for example, a PC (Personal Computer) or a server device. For example, terminal device 3 generates model generation indicators through interaction with information provision device 10, and retrieves the model generated by model generation server 2 according to the generated generation indicators.

[0017] [3. Overview of the processes performed by the information provision device] Next, an overview of the processing performed by the information providing device 10 will be described. First, the information providing device 10 receives a request from the terminal device 3 for training data to be used to train the model (step S1). For example, the information providing device 10 stores various types of training data used for training in a predetermined storage device and receives a request from the user U for training data to be specified. The information providing device 10 may also obtain training data to be used for training from, for example, the terminal device 3 or various external servers.

[0018] Here, the training data can be any data. For example, the information provider 10 may use various information about the user as training data, such as the location history of each user, the history of web content viewed by each user, the purchase history and search query history of each user. The information provider 10 may also use the demographic attributes and psychographic attributes of the user as training data. Furthermore, the information provider 10 may use metadata such as the type, content, and creator of the various web content to be distributed as training data.

[0019] In such cases, the information provider 10 generates candidate generative metrics based on statistical information of the training data used for learning (step S2). For example, the information provider 10 generates candidate generative metrics that indicate what kind of model and what kind of learning method should be used for learning, based on the characteristics of the values ​​included in the training data. In other words, the information provider 10 generates generative metrics that indicate a model capable of accurately learning the characteristics of the training data and a learning method for accurately learning the characteristics of the model. That is, the information provider 10 optimizes the learning method. The content of the generative metrics generated when certain training data is selected will be described later.

[0020] Next, the information provider 10 provides candidate generation indicators to the terminal device 3 (step S3). In this case, user U modifies the candidate generation indicators according to their preferences, rules of thumb, etc. (step S4). Then, the information provider 10 provides each candidate generation indicator and the training data to the model generation server 2 (step S5).

[0021] Meanwhile, the model generation server 2 generates a model for each generation metric (step S6). For example, the model generation server 2 trains a model having the structure indicated by the generation metric to learn the features of the training data using the learning method indicated by the generation metric. Then, the model generation server 2 provides the generated model to the information providing device 10 (step S7).

[0022] Here, it is assumed that each model generated by the model generation server 2 will have differences in accuracy due to differences in the generation metrics. Therefore, the information providing device 10 generates new generation metrics using a genetic algorithm based on the accuracy of each model (step S8), and repeatedly generates models using the newly generated generation metrics (step S9).

[0023] For example, the information provider 10 divides the training data into evaluation data and training data, and obtains multiple models that have been trained on the features of the training data, each generated according to a different generative index. For example, the information provider 10 generates 10 generative indexes, and uses the 10 generated generative indexes and the training data to generate 10 models. In such a case, the information provider 10 measures the accuracy of each of the 10 models using the evaluation data.

[0024] Next, the information provider 10 selects a predetermined number of models (for example, 5) from the 10 models, in order of highest accuracy. Then, the information provider 10 generates new generative indicators from the generative indicators used when generating the 5 selected models. For example, the information provider 10 considers each generative indicator as an individual in a genetic algorithm, and considers the type of model, the structure of the model, and various learning methods (i.e., the various indicators shown by the generative indicators) that each generative indicator represents as genes in a genetic algorithm. Then, the information provider 10 generates 10 new generative indicators for the next generation by selecting individuals to crossover and performing gene crossover. The information provider 10 may also consider mutation when performing gene crossover. Furthermore, the information provider 10 may perform two-point crossover, multi-point crossover, uniform crossover, or random selection of genes to be crossovered. Furthermore, the information-providing device 10 may adjust the crossover rate during crossover so that, for example, the genes of individuals with higher model accuracy are passed on to the next generation of individuals.

[0025] Furthermore, the information provider 10 generates 10 new models again using the next generation generative index. Then, based on the accuracy of the 10 new models, the information provider 10 generates new generative indexes using the genetic algorithm described above. By repeatedly performing this process, the information provider 10 can bring the generative indexes closer to generative indexes that are appropriate for the characteristics of the training data, that is, optimized generative indexes.

[0026] Furthermore, the information provider 10 selects the model with the highest accuracy as the target for provision when predetermined conditions are met, such as when it has generated a predetermined number of new generation metrics, or when the maximum, average, or minimum accuracy of the model exceeds a predetermined threshold. The information provider 10 then provides the terminal device 3 with the selected model along with the corresponding generation metrics (step S10). As a result of this process, the information provider 10 can generate appropriate model generation metrics and provide a model that conforms to the generated generation metrics simply by selecting training data from the user.

[0027] In the example described above, the information provider 10 achieved stepwise optimization of the generation metric using a genetic algorithm, but the embodiments are not limited to this. As will become clear in the explanation below, the accuracy of a model varies greatly depending not only on the characteristics of the model itself, such as the type and structure of the model, but also on the metrics used when generating the model (i.e., when learning the characteristics of the training data), such as what kind of training data is input into the model and how, and what kind of hyperparameters are used to train the model.

[0028] Therefore, if the information provider 10 generates a generative index that is estimated to be optimal according to the training data, it does not need to perform optimization using a genetic algorithm. For example, the information provider 10 may present the user with a generative index that it has generated according to whether or not the training data satisfies various conditions generated according to empirical rules, and may also generate a model according to the presented generative index. Furthermore, if the information provider 10 accepts a modification of the presented generative index, it may generate a model according to the modified generative index, present the accuracy of the generated model to the user, and accept further modifications of the generative index. In other words, the information provider 10 may allow user U to experiment to find the optimal generative index.

[0029] [4. Regarding the generation of generative indicators] The following describes an example of what kind of generative metrics to generate for what kind of training data. Note that this is just one example; any processing can be used to generate generative metrics according to the characteristics of the training data.

[0030] [4-1. Regarding generation indicators] First, let's explain an example of the information that a generative index represents. For example, when training a model to learn the features of training data, the manner in which the training data is input to the model, the manner of the model, and the manner in which the model is trained (i.e., the features represented by the hyperparameters) are thought to contribute to the accuracy of the final model. Therefore, the information provider 10 improves the accuracy of the model by generating generative indices that optimize each manner according to the features of the training data.

[0031] For example, training data can be thought to contain data with various labels, that is, data exhibiting various features. However, if data exhibiting features that are not useful for classifying data is used as training data, the accuracy of the final model may deteriorate. Therefore, the information provider 10 determines the features of the training data to be input to the model. For example, the information provider 10 decides which labeled data (i.e., which features) from the training data to input. In other words, the information provider 10 optimizes the combination of input features.

[0032] Furthermore, the training data is thought to contain columns of various formats, such as data consisting only of numbers or data containing strings. When inputting such training data into the model, it is thought that the accuracy of the model will change depending on whether the data is input as is or converted to another format. For example, when inputting multiple types of training data (training data that each exhibits different features), such as string training data and numerical training data, the accuracy of the model is thought to change depending on whether the strings and numbers are input as is, whether the strings are converted to numbers and only the numbers are input, or whether the numbers are treated as strings and input. Therefore, the information providing device 10 determines the format of the training data to be input to the model. For example, the information providing device 10 decides whether the training data to be input to the model should be numerical or string. In other words, the information providing device 10 optimizes the column type of the features to be input.

[0033] Furthermore, if there are training data sets that each exhibit different characteristics, the accuracy of the model is expected to change depending on which combination of characteristics is input simultaneously. In other words, if there are training data sets that each exhibit different characteristics, the accuracy of the model is expected to change depending on which combination of characteristics (i.e., the relationships between combinations of multiple characteristics) is trained. For example, if there is training data that exhibits a first characteristic (e.g., gender), training data that exhibits a second characteristic (e.g., address), and training data that exhibits a third characteristic (e.g., purchase history), the accuracy of the model is expected to change depending on whether the training data that exhibits the first characteristic and the training data that exhibits the second characteristic are input simultaneously or whether the training data that exhibits the first characteristic and the training data that exhibits the third characteristic are input simultaneously. Therefore, the information providing device 10 optimizes the combination of characteristics (cross-features) that the model learns to recognize relationships.

[0034] Here, various models project the input data into a predetermined dimensional space divided by a predetermined hyperplane, and classify the input data according to which of the divided spaces the projected position belongs to. Therefore, if the number of dimensions of the space onto which the input data is projected is lower than the optimal number of dimensions, the classification ability of the input data deteriorates, resulting in a decrease in the accuracy of the model. Also, if the number of dimensions of the space onto which the input data is projected is higher than the optimal number of dimensions, the dot product value with the hyperplane changes, which may result in the inability to properly classify data different from the data used during training. Therefore, the information providing device 10 optimizes the number of dimensions of the input data input to the model. For example, the information providing device 10 optimizes the number of dimensions of the input data by controlling the number of nodes in the input layer of the model. In other words, the information providing device 10 optimizes the number of dimensions of the space into which the input data is embedded.

[0035] In addition to SVMs, the models also include neural networks with multiple hidden layers. Various types of neural networks are known, such as feedforward DNNs where information is transmitted unidirectionally from the input layer to the output layer, convolutional neural networks (CNNs) that perform convolution in the hidden layers, recurrent neural networks (RNNs) with directed cycles, and Boltzmann machines. Furthermore, these various neural networks include LSTMs (Long Short-Term Memory) and other types of neural networks.

[0036] Thus, the accuracy of a model is expected to change depending on the type of model used to learn various features of the training data. Therefore, the information provider 10 selects the type of model that is estimated to learn the features of the training data with good accuracy. For example, the information provider 10 selects the type of model depending on what labels are assigned to the training data. To give a more specific example, if there is data with terms related to "history" as labels, the information provider 10 selects an RNN that is thought to be able to learn the features of history better, and if there is data with terms related to "images" as labels, it selects a CNN that is thought to be able to learn the features of images better. In addition to these, the information provider 10 can also determine whether a label is a pre-specified term or a term similar to a specified term, and select a model of a type that is pre-associated with the term that is determined to be identical or similar.

[0037] Furthermore, it is thought that the model's learning accuracy changes when the number of hidden layers in the model or the number of nodes in a single hidden layer changes. For example, if the number of hidden layers in the model is large (the model is deep), it is thought that classification according to more abstract features can be achieved, but on the other hand, local errors in backpropagation may not propagate easily to the input layer, which may result in improper learning. Also, if the number of nodes in the hidden layers is small, a higher level of abstraction can be performed, but if the number of nodes is too small, there is a high possibility that information necessary for classification will be lost. Therefore, the information provider 10 optimizes the number of hidden layers and the number of nodes in the hidden layers. In other words, the information provider 10 optimizes the model's architecture.

[0038] Furthermore, the accuracy of the nodes is thought to change depending on whether or not attention is present, whether or not the nodes included in the model have autoregression, and which nodes are connected to each other. Therefore, the information providing device 10 optimizes the network by determining whether or not it has autoregression and which nodes are connected to each other.

[0039] Furthermore, when training a model, the model's optimization method (the algorithm used during training), the dropout rate, the node's activation function, and the number of units are set as hyperparameters. It is thought that the model's accuracy will also change if these hyperparameters change. Therefore, the information provider 10 optimizes the training method, i.e., the hyperparameters, when training the model.

[0040] Furthermore, the accuracy of the model changes if the model size (number of input layers, hidden layers, and output layers, or the number of nodes) changes. Therefore, the information providing device 10 also optimizes the model size.

[0041] In this way, the information provider 10 optimizes the indicators used when generating the various models described above. For example, the information provider 10 stores conditions corresponding to each indicator in advance. These conditions are set based on empirical rules, such as the accuracy of various models generated from past learning models. The information provider 10 then determines whether the learning data satisfies each condition and adopts an indicator pre-associated with the conditions that the learning data satisfies or does not satisfy as a generation indicator (or a candidate thereof). As a result, the information provider 10 can generate a generation indicator that can accurately learn the features of the learning data.

[0042] Furthermore, as mentioned above, if the process of automatically generating generative metrics from training data and creating a model according to these generative metrics is performed automatically, users do not need to refer to the training data itself or make judgments about what kind of data distribution exists. As a result, the information providing device 10 can, for example, reduce the effort required of data scientists and others to recognize the training data when creating a model, and prevent the infringement of privacy associated with the recognition of training data.

[0043] [4-2. Generative metrics according to data type] The following describes an example of the conditions for generating generative metrics. First, we will describe an example of the conditions depending on the type of data used as training data.

[0044] For example, the training data used for learning may include integers, floating-point numbers, or strings. Therefore, it is estimated that selecting an appropriate model for the format of the input data will result in higher model learning accuracy. Accordingly, the information providing device 10 generates a generation index based on whether the training data is an integer, a floating-point number, or a string.

[0045] For example, if the training data is integers, the information provider 10 generates a generation index based on the continuity of the training data. For example, if the density of the training data exceeds a predetermined first threshold, the information provider 10 considers the training data to be continuous data and generates a generation index based on whether the maximum value of the training data exceeds a predetermined second threshold. Also, if the density of the training data falls below a predetermined first threshold, the information provider 10 considers the training data to be sparse training data and generates a generation index based on whether the number of unique values ​​included in the training data exceeds a predetermined third threshold.

[0046] Let's explain a more specific example. In the following example, we will describe an example of the process of selecting a feature function from the configuration file sent to the model generation server 2, which automatically generates a model using AutoML, as the generation metric. For example, if the training data is integers, the information provider 10 determines whether the density exceeds a predetermined first threshold. For example, the information provider 10 calculates the density by dividing the number of unique values ​​among the values ​​included in the training data by the maximum value of the training data plus 1.

[0047] Next, if the density exceeds a predetermined first threshold, the information provider 10 determines that the training data is continuous training data, and determines whether the value obtained by adding 1 to the maximum value of the training data exceeds a second threshold. If the value obtained by adding 1 to the maximum value of the training data exceeds the second threshold, the information provider 10 selects "Categorical_colum_with_identity & embedding_column" as the feature function. On the other hand, if the value obtained by adding 1 to the maximum value of the training data falls below the second threshold, the information provider 10 selects "Categorical_column_with_identity" as the feature function.

[0048] On the other hand, if the density falls below a predetermined first threshold, the information provider 10 determines that the training data is sparse and determines whether the number of unique values ​​in the training data exceeds a predetermined third threshold. If the number of unique values ​​in the training data exceeds the predetermined third threshold, the information provider 10 selects "Categorical_column_with_hash_bucket & embedding_column" as the feature function, and if the number of unique values ​​in the training data falls below the predetermined third threshold, it selects "Categorical_column_with_hash_bucket" as the feature function.

[0049] Furthermore, if the training data is a string, the information provider 10 generates a generation index based on the number of string types included in the training data. For example, the information provider 10 counts the number of unique strings (number of unique data) included in the training data, and if the counted number is below a predetermined fourth threshold, it selects "categorical_column_with_vocabulary_list" and / or "categorical_column_with_vocabulary_file" as the feature function. Also, if the counted number is below a predetermined fifth threshold that is greater than the fourth threshold, the information provider 10 selects "categorical_column_with_vocabulary_file & embedding_column" as the feature function. Also, if the counted number is above a predetermined fifth threshold that is greater than the fourth threshold, the information provider 10 selects "categorical_column_with_hash_bucket & embedding_column" as the feature function.

[0050] Furthermore, if the training data is floating-point, the information provider 10 generates a conversion metric for the training data into input data for the model, as a model generation metric. For example, the information provider 10 selects either "bucketized_column" or "numeric_column" as the feature function. That is, the information provider 10 bucketizes (groups) the training data and selects whether to input the bucket number or the numerical value directly. The information provider 10 may, for example, bucketize the training data so that the range of numerical values ​​associated with each bucket is roughly the same, or it may associate a range of numerical values ​​with each bucket so that the number of training data classified into each bucket is roughly the same. The information provider 10 may also select the number of buckets or the range of numerical values ​​associated with each bucket as a generation metric.

[0051] Furthermore, the information provider 10 acquires training data exhibiting multiple features and generates a generation index for the model, which indicates the features of the training data that should be learned by the model. For example, the information provider 10 decides which labels of training data to input into the model and generates a generation index indicating the decided labels. The information provider 10 also generates a generation index for the model, which indicates multiple types of training data whose correlations should be learned by the model. For example, the information provider 10 decides which combinations of labels to input to the model simultaneously and generates a generation index indicating the decided combinations.

[0052] Furthermore, the information provider 10 generates a generation index that indicates the number of dimensions of the training data input to the model, as a model generation index. For example, the information provider 10 may determine the number of nodes in the input layer of the model according to the number of unique data included in the training data, the number of labels to input to the model, the combination of the number of labels to input to the model, the number of buckets, etc.

[0053] Furthermore, the information provider 10 generates a generation index that indicates the type of model used to learn the features of the training data, as a model generation index. For example, the information provider 10 determines the type of model to be generated based on the density and sparseness of the training data previously used for learning, the content of the labels, the number of labels, the number of label combinations, etc., and generates a generation index that indicates the determined type. For example, the information provider 10 generates a generation index that indicates models such as "BaselineClassifier", "LinearClassifier", "DNNClassifier", "DNNLinearCombinedClassifier", "BoostedTreesClassifier", "AdaNetClassifier", "RNNClassifier", "DNNResNetClassifier", and "AutoIntClassifier" as classes of models in AutoML.

[0054] The information provider 10 may also generate generative metrics that show various independent variables for each class of model. For example, the information provider 10 may generate generative metrics that show the number of intermediate layers in the model or the number of nodes included in each layer as generative metrics for the model. The information provider 10 may also generate generative metrics that show the connection patterns between nodes in the model or generative metrics that show the size of the model as generative metrics for the model. These independent variables will be appropriately selected depending on whether the various statistical features of the training data satisfy predetermined conditions.

[0055] Furthermore, the information provider 10 may generate a generation index that indicates the learning mode when training the model to learn the features of the training data, i.e., the hyperparameters, as a model generation index. For example, the information provider 10 may generate a generation index that indicates "stop_if_no_decrease_hook", "stop_if_no_increase_hook", "stop_if_higher_hook", or "stop_if_lower_hook" when setting the learning mode in AutoML.

[0056] In other words, the information provider 10 generates a generation index that indicates the characteristics of the training data to be trained on the model, the form of the model to be generated, and the training form when training the model to learn the characteristics of the training data, based on the labels of the training data used for training and the characteristics of the data itself. More specifically, the information provider 10 generates a configuration file for controlling the generation of the model in AutoML.

[0057] [4-3. Regarding the order in which to determine the generation index] Here, the information provider 10 may optimize the various indicators described above simultaneously or in an appropriate order. Furthermore, the information provider 10 may allow the user to change the order in which each indicator is optimized. That is, the information provider 10 may receive from the user the characteristics of the training data to be trained on the model, the form of the model to be generated, and the order in which the training form is determined when training the model with the characteristics of the training data, and then determine each indicator in the order in which it was received.

[0058] For example, when the information provider 10 starts generating a generative index, it optimizes the input features, such as the characteristics of the input training data and the manner in which the training data is input. Next, it optimizes the input cross-features, such as which combinations of features to train. Subsequently, the information provider 10 selects a model and optimizes the model structure. After that, the information provider 10 optimizes the hyperparameters and finishes generating the generative index.

[0059] Here, the information provider 10 may repeatedly optimize the input features in input feature optimization by selecting and modifying various input features such as the characteristics and input patterns of the input training data, and by selecting new input features using a genetic algorithm. Similarly, the information provider 10 may repeatedly optimize the input cross-feature in input cross-feature optimization, and may repeatedly perform model selection and model structure optimization. The information provider 10 may also repeatedly perform hyperparameter optimization. Furthermore, the information provider 10 may repeatedly perform a series of processes including input feature optimization, input cross-feature optimization, model selection, model structure optimization, and hyperparameter optimization to optimize each indicator.

[0060] Furthermore, the information provider 10 may, for example, perform hyperparameter optimization before model selection and model structure optimization, or perform input feature optimization and input cross-feature optimization after model selection and model structure optimization. Alternatively, the information provider 10 may, for example, repeatedly perform input feature optimization, and then repeatedly perform input cross-feature optimization. After that, the information provider 10 may repeatedly perform input feature optimization and input cross-feature optimization. In this way, any setting can be adopted regarding which indicators to optimize, in what order, and which optimization processes to repeatedly execute during the optimization process.

[0061] [5. Information processing according to the embodiment] Up to this point, we have explained the various processes performed by the information providing device 10 using Figure 1. From here on, we will explain the information processing performed by the information processing device 100 and the information processing performed by the execution control device 200.

[0062] [5-1. Configuration of the Information Processing System] First, before describing the information processing according to the embodiment, we will explain the information processing system Sy, which is a part of the information provision system 1, using Figure 2. Figure 2 is a diagram showing an example of the information processing system Sy according to the embodiment. The information processing system Sy corresponds to a partial system of the information provision system 1, which includes only the information processing device 100 and the execution control device 200.

[0063] As shown in Figure 2, the information processing system Sy includes an information processing device 100 and an execution control device 200. In this embodiment, the information processing device 100 is described as a server device, but it may be implemented by a cloud system or the like. Similarly, in this embodiment, the execution control device 200 is described as a server device, but it may be implemented by a cloud system or the like.

[0064] As explained in Figure 1, the information provision device 10 optimizes the model architecture according to the characteristics of the data and automatically generates the model in order to facilitate model creation.

[0065] In contrast, the information processing device 100 primarily performs information processing that optimizes learning and generation methods, such as how to learn or generate a model. The information processing device 100 can also operate as the information providing device 10 by possessing some or all of the functions of the information providing device 10. Furthermore, the information processing device 100 can also possess some or all of the functions of the model generation server 2. In addition to the processing described in Figure 1 as performed by the information providing device 10, the information processing device 100 will also perform various processing as shown in the following embodiments.

[0066] Furthermore, the execution control unit 200 primarily performs information processing that optimizes the execution entity that performs processing using a model (for example, processing that predicts a specific target).

[0067] The optimization processes performed by the information processing device 100 can be broadly divided into two categories: optimization processes that optimize the learning method, which determines how to learn or generate a model, and optimization processes that optimize the data input to the learned model when the learned model is actually used. Therefore, in the following embodiments, the optimization processes for optimizing the learning method and the data input to the learned model will be explained first, targeting the information processing device 100, followed by an explanation of the optimization processes performed by the execution control device 200.

[0068] Furthermore, the optimization process for optimizing the learning method can be further classified into five optimization processes, such as the first to fifth optimizations, which will be described later. Therefore, regarding the optimization process for optimizing the learning method, we will first explain the overview of each of the first to fifth optimizations, and an example of the execution order in which the first to fifth optimizations are performed, using Figure 3 below. After that, we will explain a detailed example of each of the first to fifth optimizations based on the functional configuration diagram shown in Figure 5.

[0069] [5-2. An example of processing performed by an information processing device] From here, we will explain an example of the processing performed by the information processing device 100 using Figure 3. Figure 3 is a diagram showing the overall picture of the processing performed by the information processing device 100 according to the embodiment. For example, in the actual operation of a model, there are motivations such as wanting to make the model size as small as possible, reducing unnecessary calculations and increasing the inference speed. Therefore, Figure 3 shows a scene in which the computation graph is optimized to improve the size of the model and the performance in the serving environment when providing (serving) inference by the model as an API. A computation graph is a directed graph that represents the calculation process, where the vertices (nodes) of the graph represent the calculation content to be performed, and the edges represent the input and output of each node. For this reason, the model is defined as, for example, a graph of tensor calculations.

[0070] Furthermore, as described above, the information processing device 100 tunes the model to serve a higher-performance model by optimizing the learning method. Therefore, Figure 3 illustrates a series of tuning algorithms (fine-tuning according to the embodiment) including various optimizations according to the embodiment.

[0071] Furthermore, as shown in Figure 3, the fine-tuning according to this embodiment is divided into an optimization process that optimizes the learning method, and a tuning process that further fine-tunes the model for service use by modifying a part of the trained model obtained through the optimization process and retraining it. The optimization process is performed, for example, by the optimizer function (referred to as "Optimizer OP") of the information processing device 100. The tuning process is performed by the data select function (referred to as "Selector SE") of the information processing device 100.

[0072] First, the information processing device 100 generates multiple initial values ​​for model parameters (e.g., weights and biases) based on random numbers (pseudo-random numbers) (step S11). At this time, the information processing device 100 controls the initialization of the model parameters more appropriately by performing a first optimization to optimize the seed for obtaining random numbers (i.e., the random number seed). Thus, the first optimization is the optimization of the random number seed in the computation graph.

[0073] In deep learning, initial values ​​for model parameters are determined based on pseudo-random numbers, and the model is trained to learn the features of the training data. As a result of this process, the values ​​of the model parameters gradually change (converge) to values ​​that correspond to the features of the training data. Therefore, if the initial values ​​of the model parameters deviate significantly from the values ​​corresponding to the features of the training data, the time required for training will increase, and the learning speed will slow down. From this perspective, one approach is to generate multiple models, each with different initial values, and then select the model that shows the best accuracy as the training result.

[0074] On the other hand, considering the relationship between model parameters and the accuracy achieved by a set of model parameters, it is estimated that, due to the model's structure, the accuracy does not change discontinuously for each model parameter, but rather exhibits a nearly continuous relationship where model parameters closer to the optimal value result in higher accuracy. Furthermore, if the initial values ​​of the model parameters are close to local minimums rather than optimal values ​​corresponding to the training data, the model parameters may remain at local minimums, potentially preventing further improvement in accuracy. For this reason, when generating multiple models with different initial values, it is desirable to generate a set of initial values ​​for model parameters that have a certain degree of spread (i.e., a distribution).

[0075] Therefore, the information processing device 100 performs a first optimization so that it can generate multiple models in which the set of model parameters has a predetermined distribution. For example, when the information processing device 100 generates model parameters for each model, it generates the model parameters using a predetermined random function from predetermined initial values. Such a random function can be configured in various ways, such as what kind of distribution the generated random numbers have (e.g., uniform distribution or normal distribution), what kind of mean value the generated random numbers have from the input seed value, and what range of random numbers are generated. Therefore, the information processing device 100 optimizes the value of the random number seed, such as the seed value input to the random function and various settings.

[0076] More specifically, the information processing device 100 sets multiple random number seeds that satisfy a predetermined distribution through a first optimization. Then, the information processing device 100 inputs each of the set random number seeds into a random function to generate a random number corresponding to that random number seed. Furthermore, the random numbers generated in this way will exhibit a predetermined distribution. Therefore, by using these random numbers, the information processing device 100 can generate a set of initial values ​​for model parameters having a predetermined distribution in step S11.

[0077] Next, the information processing device 100 generates a model for each of the initial values ​​of the model parameters generated in step S11 (step S12). Specifically, the information processing device 100 generates a model having each set of model parameters from a group of initial values ​​of model parameters that fall within a predetermined distribution, for each set of model parameters with a different combination.

[0078] Next, the information processing device 100 randomly extracts data for the current iterative learning (i.e., the learning data to be learned) from the training data and stores the extracted data in a buffer. Then, when the learning of the features of the data stored in the buffer is complete, the information providing device 10 extracts new data and stores it in the buffer, and controls the system to perform iterative learning according to the shuffling by running the learning of the data stored in the buffer (step S13).

[0079] Here, if the training dataset is divided into several subsets, the best-performing model will not necessarily be trained if all subsets are used for model training. On the other hand, if the model is trained by the iterative training described above, it is thought that the accuracy of the model can be further improved by optimizing the combination of data included in one subset. Therefore, when the information processing device 100 performs step S13, it performs a second optimization to optimize the training data, which is selected from the dataset to be used for training, and a third optimization to optimize the size of the buffer in which shuffling takes place. Thus, the second optimization optimizes the data used for training, and the third optimization optimizes the size of the shuffle buffer.

[0080] For example, in step S13, the information processing device 100 performs a second optimization and a third optimization to generate training data for the target of learning (training data corresponding to the optimized buffer size), which is the training data used in this iterative learning, and stores it in the buffer.

[0081] Furthermore, the information processing device 100 causes each model generated in step S12 to learn the features of the training data stored in the buffer in step S13 (step S14).

[0082] For example, the information processing device 100 learns the features of each piece of training data stored in the buffer, one by one, but shuffles the learning order (the order of the training data) within the buffer. Specifically, the information processing device 100 shuffles the learning order to a random order each time an epoch is reached.

[0083] Here, it is considered important that the data is shuffled well when training a model. However, simply shuffling the data may not be sufficient, as it could lead to biases in the training order or the data distribution per batch, potentially hindering effective training. For example, when training a model, the model is trained (model parameter adjustments) using one set of training data, and then trained again using different training data, sequentially learning the features of the training data. Therefore, if the training data has a time series, it is considered better to distribute the time series of the training data to some extent in order to broadly and generally learn the features of the training data. On the other hand, if there are large gaps in the time series of the training data input to the model consecutively, the range of adjustments to the model parameters may become too large, potentially preventing proper training. In other words, when training a model to learn the features of time series training data, it is necessary to use the training data sequentially so that the time series is somewhat scattered in order to learn features that are not constrained by the time series. However, if the time series is too scattered, it may become impossible to train the model properly. In such cases, it is not possible to improve the accuracy of the model.

[0084] Therefore, when performing step S14, the information processing device 100 optimizes the seed value used to generate the random order so that there is no bias in the random order between epochs (i.e., a uniform distribution). Specifically, the information processing device 100 generates an optimal random order by performing a fourth optimization that optimizes the seed for random order generation (i.e., the random number seed), so that specific training data are not learned in the same order every time. Thus, the fourth optimization is the optimization of the random number seed in data shuffling.

[0085] For example, as a fourth optimization, the information processing device 100 generates a random number seed for the current learning process to ensure that there is no bias in the random order associated with each training data between epochs. The information processing device 100 then inputs each generated random number seed into a random function to generate a random order. The information processing device 100 also associates the generated random order with the training data for each target of learning to generate the final training data for the target of learning in a buffer. As a result, in actual learning, learning is performed for each pair of model and training data obtained by multiplying a model having model parameters generated to exhibit a predetermined distribution by the first optimization with training data whose order has been randomly determined by the fourth optimization.

[0086] The information processing device 100 then sequentially trains each model on the features of the final training data in the randomly generated order. Specifically, once the information processing device 100 has finished training the features of the training data in the randomly generated order (after one epoch), it generates a new random order and moves on to the next epoch, training each model on the features of the training data again in the generated random order. In this way, the information processing device 100 repeats the training loop for the specified number of epochs.

[0087] Once the Loop, which involves repeatedly training for the specified number of epochs, is complete, the buffer becomes empty. Therefore, the information processing device 100 stores the unprocessed training data from the training data obtained in step S13 into the now empty buffer, and then repeats step S14 again using this stored training data to train all of the training data obtained in step S13.

[0088] Detailed examples of the second to fourth optimizations, as well as detailed examples of the iterative learning in steps S13 and S14, will be described later.

[0089] Furthermore, in the actual learning process in step S14, the attempts to explore hyperparameters are repeated, but in order to achieve efficient exploration, the information processing device 100 performs a fifth optimization as an optimization of the attempts by pruning. Thus, the fifth optimization is an optimization related to early stopping, in which attempts that are not expected to yield good results are terminated early without being carried to completion.

[0090] For example, the information processing device 100 allows the user to specify constraints that define which trials are subject to early stopping (to be terminated early), from the perspective of evaluation values ​​used to assess the accuracy of the model. The information processing device 100 then monitors whether the constraints are met for each trial, and terminates the trial when it determines that the constraints are met, continuing only the remaining trials. In other words, the information processing device 100 selects only trials where the evaluation value used to assess the accuracy of the model satisfies a predetermined condition (for example, the inverse of the constraints) (trials that are not selected are pruned), and continues learning for the selected trials. A detailed example of the fifth optimization will be described later.

[0091] Furthermore, the information processing device 100 selects the best model from among the generated models based on the accuracy of each trained model obtained through the training process to which the optimization process has been applied (step S15). For example, the information processing device 100 calculates the accuracy of each model using evaluation data, and calculates a higher evaluation value the greater the variation in accuracy (amount of improvement in accuracy). Then, the information processing device 100 selects the model for which the highest evaluation value was calculated as the best model.

[0092] Up to this point, we have explained the learning method to which optimization processing is applied by the optimizer OP. From here on, we will explain the tuning process using the selector SE.

[0093] For example, the information processing device 100 performs a tuning process to fine-tune the best model by executing a selector SE, which involves modifying a part of the best model and retraining it. The information processing device 100 can also use the training data used in the training process to which the optimization process was applied as a single dataset in the tuning process.

[0094] Here, the above datasets were divided as shown in Figure 4 according to their intended use, so that each tuning process using training data with different ranges (time ranges corresponding to the time series) could be treated as a single trial, and the tuning results (accuracy of the best model) could be effectively evaluated. Figure 4 shows examples of how the datasets are divided for each trial when they are divided according to their intended use.

[0095] The data included in the dataset corresponds to the purchase history of goods purchased using a specified service (e.g., a specified shopping service) and has a time-series concept. Therefore, the data included in the dataset is arranged in chronological order. In the example in Figure 4, the dataset has a time range from "June 11, 0:00" to "June 19, 0:00", and the oldest data (purchase history on June 11, 0:00) to the newest data (purchase history on June 19, 0:00) are arranged in chronological order.

[0096] In the example shown in Figure 4, for such a dataset, the data from "June 11, 0:00" to "June 16, 17:32" is assigned to Trial A as training data for tuning. This example means that it has been decided that Trial A will be the process of tuning the best model using the data from "June 11, 0:00" to "June 16, 17:32" as training data.

[0097] Furthermore, in the example shown in Figure 4, data from "June 16, 17:32" to "June 17, 7:26" is assigned as evaluation data for Trial A. This example shows a case where it is decided to evaluate the best model after tuning by Trial A using data from "June 16, 17:32" to "June 17, 7:26".

[0098] Furthermore, in the example in Figure 4, data from "June 17, 7:26 AM" to "June 19, 12:00 AM" is assigned as test data for Trial A. This example shows that it was decided to evaluate the best model after tuning by Trial A by using the data from "June 17, 7:26 AM" to "June 19, 12:00 AM" as test data with unknown labels.

[0099] Furthermore, in the example in Figure 4, data from "June 11, 0:00" to "June 17, 7:26" is assigned to Trial B as training data for tuning. This example means that it has been decided that Trial B will be the process of tuning the best model using data from "June 11, 0:00" to "June 17, 7:26" as training data.

[0100] Furthermore, in the example shown in Figure 4, data from "June 17, 7:26 AM" to "June 17, 12:00 PM" is assigned as evaluation data for Trial B. This example shows a case where it is decided to evaluate the best model after tuning by Trial B using data from "June 17, 7:26 AM" to "June 17, 12:00 PM".

[0101] Furthermore, in the example in Figure 4, data from "June 17, 12:00" to "June 19, 0:00" is assigned as test data for Trial B. This example shows that it was decided to evaluate the best model after tuning by Trial B by using the data from "June 17, 12:00" to "June 19, 0:00" as test data with unknown labels.

[0102] Furthermore, in the example in Figure 4, data from "June 11th, 0:00" to "June 17th, 12:00" is assigned to Trial C as training data for tuning. This example means that it has been decided that Trial C will be the process of tuning the best model using data from "June 11th, 0:00" to "June 17th, 12:00" as training data.

[0103] Furthermore, in the example shown in Figure 4, data from "June 17, 12:00" to "June 19, 0:00" is assigned as evaluation data for Trial C. This example shows a case where it is decided to evaluate the best model after tuning by Trial C using data from "June 17, 12:00" to "June 19, 0:00".

[0104] Note that the allocation shown in Figure 4 is just one example, and depending on the tuning process, the data used for training, evaluation, and testing can be changed as needed to suit the needs of the model administrator.

[0105] Returning to Figure 3, the information processing device 100 uses the training data shown in Figure 4 to perform the tuning process by iterative learning described below on the best model, and repeatedly evaluates it using the evaluation data and test data shown in Figure 4. The information processing device 100 performs this series of processes for each trial. Furthermore, since the contents of this series of processes are the same regardless of the trial, an example of the contents of this series of processes will be explained below using Trial A as the subject.

[0106] For example, the information processing device 100 divides the training data into sets consisting of a predetermined number of data (step S21). The training data for each set is managed, for example, in a file corresponding to that set. For example, the information processing device 100 can divide the training data into several hundred sets (for example, 500 sets), but for the sake of simplicity, Figure 3 shows an example where the training data is divided into 10 sets. Specifically, in Figure 3, Files "1" to "10" are shown as an example of these 10 sets. A predetermined number of training data is stored in each file.

[0107] In this state, the information processing device 100 randomly selects one set from each set obtained by the division and adds it to the training data list (step S22). Then, each time a set is added, the information processing device 100 trains the best model on the features of the training data in the newly added set (step S23). For example, the information processing device 100 trains the training data in the newly added set for only one epoch. Then, the information processing device 100 evaluates the accuracy of the trained best model using evaluation data and test data (step S24), and this series of processes is repeated.

[0108] In this regard, the example in Figure 3 shows that in the first step S22, the information processing device 100 selects File "6" and adds the selected File "6" to the training data list. Furthermore, in the first step S23, the information processing device 100 trains the best model on the features of the training data contained in File "6", which was just added. Finally, in the first step S24, the information processing device 100 shows an example in which the best model, which has learned the features of the training data contained in File "6", is evaluated using evaluation data and test data.

[0109] Furthermore, in the example shown in Figure 3, the information processing device 100 selects File "9" in the second step S22 and adds the selected File "9" to the training data list. In the second step S23, the information processing device 100 trains the best model on the features of the training data contained in File "9", which was added this time. In the second step S24, the information processing device 100 shows an example in which the best model, which has so far learned the features of the training data contained in Files "6" and "9", is evaluated using evaluation data and test data.

[0110] Furthermore, in the example shown in Figure 3, the information processing device 100 selects File "3" in the third step S22 and adds the selected File "3" to the training data list. In the third step S23, the information processing device 100 trains the best model on the features of the training data contained in File "3", which was added this time. In the third step S24, the information processing device 100 shows an example in which the best model, which has so far trained on the features of the training data contained in Files "6", "9", and "3", is evaluated using evaluation data and test data.

[0111] Regarding the Loop from steps S22 to S24, in more detail, the information processing device 100 randomly selects one data file from the training data, adds the selected data file to the training data list of the Model Config, and then trains the best model with the training data contained in the added data file for one epoch.

[0112] Furthermore, based on the evaluation results to date, the information processing device 100 randomly selects one new data file for each of the top five Model Configs and adds the selected data file to the Model Config's training data list. The information processing device 100 then trains the best model for one epoch using the training data included in the training data list, which now has one additional data file.

[0113] Furthermore, the information processing device 100 continues the loop from steps S22 to S24 until it determines, based on the evaluation results, that the performance (accuracy) of the best model cannot be improved any further.

[0114] Furthermore, the information processing device 100 can process the best model with maximized performance as the serving target. For example, in response to access from a user, the information processing device 100 provides the best model with improved performance through fine-tuning according to the embodiment. With such an information processing device 100, users no longer need to spend time improving the model, and can instead focus on adjusting the data input to the model.

[0115] [6. Configuration of the Information Processing Device] Next, an information processing device 100 according to an embodiment will be described using Figure 5. Figure 5 is a diagram showing an example configuration of the information processing device 100 according to an embodiment. As shown in Figure 5, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0116] (Regarding Communications Unit 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information between it and, for example, the model generation server 2, the terminal device 3, the information provision device 10, and the execution control device 200.

[0117] (Regarding memory unit 120) The memory unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The memory unit 120 includes a learning data storage unit 121 and a model storage unit 122.

[0118] (Regarding the learning data storage unit 121) The learning data storage unit 121 stores various types of learning data. For example, the learning data storage unit 121 stores learning data that has been divided into learning data, evaluation data, and test data.

[0119] For example, the information processing device 100 divides all training data into training data, evaluation data, and test data, and registers these data obtained through the division into the training data storage unit 121. For example, the information processing device 100 can divide all training data using any method. For example, the information processing device 100 can divide all training data using the hold-out method, the cross-validation method, or the leave-one-out method.

[0120] Here, Figure 6 illustrates an example of data splitting for training. Figure 6 is a conceptual diagram illustrating the splitting of a dataset. As shown in Figure 6, the information processing device 100 uses the generate_data() function to generate training data consisting of N data sets and test data consisting of N data sets from the dataset (data).

[0121] Furthermore, in this state, the information processing device 100 uses the split_data() function to split the training data, which consists of N data sets, into training data and evaluation data. For example, the information processing device 100 splits the training data so that it obtains training data and evaluation data in a ratio of "N1:N2" (actually, 7:3, etc.). In addition, the information processing device 100 designates all of the test data, which consists of N data sets, as test data.

[0122] Furthermore, the information processing device 100 registers the learning data, evaluation data, and test data obtained in this manner into the learning data storage unit 121.

[0123] (Regarding the model memory unit 122) The model storage unit 122 stores information about the model. For example, the model storage unit 122 saves the model, which is updated with each epoch, in a checkpoint file format. For example, the information processing device 100 saves the parameters during the learning process at regular intervals in the model storage unit 122 and generates checkpoints.

[0124] (Regarding the control unit 130) The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), which executes various programs stored in the memory device inside the information processing device 100 using RAM as the working area. Alternatively, the control unit 130 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0125] As shown in Figure 3, the control unit 130 includes a generation unit 131, an acquisition unit 132, a first data control unit 133, a second data control unit 134, a first learning unit 135, a model selection unit 136, a second learning unit 137, a provision unit 138, and an attribute selection unit 139, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 5, and other configurations are also possible as long as they perform the information processing described later. Also, the connection relationships of the various processing units in the control unit 130 are not limited to the connection relationships shown in Figure 5, and other connection relationships are also possible.

[0126] (Regarding the generation unit 131) The generation unit 131 is a processing unit that performs the processing of steps S11 and S12 as described in Figure 3. Therefore, the generation unit 131 performs the processing of steps S11 and S12 using the first optimization algorithm.

[0127] Specifically, the generation unit 131 generates multiple models, each with different parameters. For example, the generation unit 131 generates multiple input values ​​(random number seeds) to be input to a predetermined first function that calculates random numbers based on input values, and for each of the generated input values, it generates multiple models having parameters (e.g., weights and biases) corresponding to the random numbers (pseudo-random numbers) output by the predetermined first function when that input value is input.

[0128] In this regard, the generation unit 131 generates multiple input values ​​to be input to a predetermined first function such that the random values ​​output by the predetermined first function satisfy predetermined conditions. For example, the generation unit 131 generates multiple input values ​​such that the random values ​​are within a predetermined range. Alternatively, for example, the generation unit 131 generates multiple input values ​​such that the distribution of random values ​​exhibits a predetermined probability distribution. Alternatively, for example, the generation unit 131 generates multiple input values ​​such that the average value of the random values ​​is a predetermined value. The input values ​​referred to here are parameters input to a random function (an example of a predetermined first function) and correspond to a random number seed.

[0129] For example, the generation unit 131 selects a function as a predetermined first function such that the distribution of random values ​​output when an input value is input exhibits a predetermined probability distribution (e.g., a uniform distribution), and generates multiple models having parameters corresponding to the random values ​​output by the selected function.

[0130] Furthermore, the generation unit 131 can register each generated model in the model storage unit 122.

[0131] (Regarding acquisition section 132) The acquisition unit 132 acquires various types of information and passes the acquired information to the optimal processing unit. For example, when optimization or learning is performed using training data, the acquisition unit 132 acquires training data from the training data storage unit 121. Then, the acquisition unit 132 outputs the acquired training data to the processing unit that performs optimization or learning.

[0132] (Regarding the first data control unit 133) The first data control unit 133 optimizes the data used for training using a second optimization algorithm when the process of step S13, as described in Figure 3, is performed.

[0133] Specifically, the first data control unit 133 divides predetermined training data (training data) used to train the model into multiple sets in chronological order. For example, the first data control unit 133 divides the training data into sets containing a predetermined number of data points.

[0134] Furthermore, the first data control unit 133 selects a set from among the sets obtained by dividing the training data into multiple sets in chronological order that will actually be used to train the model. For example, the first data control unit 133 selects a set from among the sets obtained by dividing the training data into multiple sets in chronological order that contains training data with a more recent time series.

[0135] The first data control unit 133 may randomly select a set to be used for training the model from among the sets obtained by dividing the training data into multiple sets in chronological order.

[0136] Furthermore, the first data control unit 133 may select a number of sets specified by the user from among the sets obtained by dividing the training data into multiple sets in chronological order. For example, the first data control unit 133 will continue to select sets in chronological order from among the sets obtained by dividing the training data into multiple sets in chronological order, with the sets containing the training data having more recent time series, until the number of selected sets reaches the number specified by the user.

[0137] Furthermore, the first data control unit 133 generates a single data set by concatenating the selected pairs. For example, the first data control unit 133 generates a single data set by concatenating them in the order they were selected. The first data control unit 133 can also pass the generated data set to, for example, the second data control unit 134 so that it can be used for training the model.

[0138] (Regarding the second data control unit 134) The second data control unit 134 optimizes the shuffle buffer size using a third optimization algorithm when the process in step S13, as described in Figure 3, is performed. For example, as part of the shuffle buffer size optimization, the second data control unit 134 generates training data of a size equal to the shuffle buffer size and stores this data in the shuffle buffer as the training data to be learned, which is the training data to be used in this iterative learning process.

[0139] For example, the second data control unit 134 divides the data set generated by the first data control unit 133 into multiple sets, each containing training data equal in size to the shuffle buffer.

[0140] For example, the second data control unit 134 divides the data set generated by the first data control unit 133 into multiple sets in chronological order. For example, the second data control unit 134 divides the data set generated by the first data control unit 133 into sets containing a number of training data specified by the user. Alternatively, for example, the second data control unit 134 may divide the training data set generated by the first data control unit 133 into multiple sets such that the number of training data included in each set falls within a range specified by the user.

[0141] Furthermore, the second data control unit 134 stores one set of the sets obtained by the splitting, corresponding to the time series of the training data it contains, in the shuffle buffer as the training data to be learned, which will be used in this iterative learning process. Specifically, the second data control unit 134 stores the set of the sets obtained by the splitting that contains the oldest time series of the training data it contains in the shuffle buffer as the training data to be learned.

[0142] (Regarding Section 135 of the First Learning Department) The first learning unit 135 causes each of the multiple models generated by the generation unit 131 to learn the features that a predetermined portion of the training data possesses.

[0143] For example, the first learning unit 135 causes each of the multiple models generated by the generation unit 131 to learn the features of the training data (training data to be learned) stored in the buffer (shuffle buffer) by the second data control unit 134. Thus, for example, the first learning unit 135 uses the sets selected by the first data control unit 133 in order from the oldest set of training data to learn the features of the training data contained in each set.

[0144] Furthermore, for example, the first learning unit 135 causes the model to learn the features of the training data (training data to be learned) included in each group divided by the second data control unit 134, in a predetermined order. For example, the first learning unit 135 causes the model to learn the features of the training data included in each group, starting with the groups corresponding to the time series, from among the groups divided by the second data control unit 134. As an example, the first learning unit 135 causes the model to learn the features of the training data included in each group, starting with the group whose training data has the oldest time series, from among the groups divided by the second data control unit 134.

[0145] Furthermore, the first learning unit 135 may, for each group divided by the second data control unit 134, train the model on the features of the training data included in that group in a random order.

[0146] Here, the first learning unit 135, when training each model with the features of the training data as described above, shuffles the learning order of each training data currently stored in the shuffle buffer. The first learning unit 135 then generates the final training data to be trained by associating the learning order obtained by shuffling with the training data. The first learning unit 135 then trains each training data to be trained in the order obtained by shuffling. The first learning unit 135 considers this series of processes related to shuffling as one epoch, and repeats this series of processes for a specified number of epochs, for example. The first learning unit 135 can generate the final training data to be trained each time the epoch is updated by shuffling the learning order.

[0147] For example, the first learning unit 135 uses a fourth optimization algorithm to optimize data shuffling, which shuffles the training data in the shuffle buffer.

[0148] For example, the first learning unit 135 uses the fourth optimization algorithm to generate a random number seed for each epoch for iterative learning, ensuring that there is no bias in the random order associated with each training data between epochs. The first learning unit 135 then inputs each generated random number seed into a random function to generate a random order. The first learning unit 135 also associates the generated random order with each training data to be learned, thereby generating the final training data for the target in a shuffle buffer.

[0149] Then, the first learning unit 135 sequentially trains each model on the features of the final training data in the randomly generated order. Specifically, once the first learning unit 135 has finished training the features of the training data in the randomly generated order (after one epoch is completed), it generates another random order and moves on to the next epoch, in which it trains each model on the features of the training data again in the generated random order.

[0150] Furthermore, during the actual training process in which each model learns the features of the training data within the shuffle buffer size, trials to search for hyperparameters are repeated. At this time, the first learning unit 135 performs a fifth optimization related to early stopping, which involves terminating trials that are not expected to yield good results early (pruning) without completing the trials, in order to achieve efficient searching.

[0151] According to the fifth optimization, the first learning unit 135 performs the following processing on each of the multiple models generated by the generation unit 131. For example, a trial is to search for the optimal combination of hyperparameters by repeatedly applying the hyperparameter combination to the model and learning for each combination of hyperparameters. In other words, a trial is an optimization of the set of hyperparameters.

[0152] Based on this, the first learning unit 135 selects several trials from among the trials (trials with different combinations of hyperparameters) in which the evaluation value for evaluating the accuracy of the model for the hyperparameter combination corresponding to that trial satisfies predetermined conditions. The first learning unit 135 then continues to train the model in the selected trials to learn the features of the training data being trained.

[0153] For example, the first learning unit 135 selects a number of trials such that the pattern based on the change in evaluation value satisfies a predetermined pattern. For example, the first learning unit 135 selects a number of trials such that the pattern based on the change in evaluation value during repeated learning of the features of the training data to be learned a predetermined number of times satisfies a predetermined pattern. For example, the first learning unit 135 selects a number of trials that satisfy a number of conditions specified by the user.

[0154] On the other hand, the first learning unit 135 stops (prunes) any trials (trials with different combinations of hyperparameters) where the evaluation value used to assess the accuracy of the model for the hyperparameter combination corresponding to that trial does not meet the predetermined conditions, and does not perform any further trials.

[0155] Furthermore, for example, the first learning unit 135 can select one of the models according to the accuracy of the model being trained, for each trial with a different combination of parameters and the training data to be trained.

[0156] (Regarding the model selection section 136) The model selection unit 136 selects the model with the highest accuracy (best model) from among the multiple models generated by the generation unit 131, based on the accuracy of each of the multiple models generated by the generation unit 131 and trained through a learning process to which optimization processing has been applied. For example, the model selection unit 136 calculates the accuracy of each model using evaluation data, and calculates a higher evaluation value the greater the variation in accuracy (amount of improvement in accuracy). Then, the model selection unit 136 selects the model with the highest evaluation value as the best model.

[0157] Furthermore, the model selection unit 136 may select one of the models according to the accuracy of the model trained by the first learning unit 135 for each combination of models with different parameters and training data. In addition, although the above example shows the first learning unit 135 selecting trials using the fifth optimization algorithm, the model selection unit 136 may also select trials using the fifth optimization algorithm.

[0158] (Regarding Section 2, Item 137) The second learning unit 137 performs tuning processing, for example, as described in steps S21 to S24 of Figure 3. Specifically, the second learning unit 137 trains the model (best model) selected by the model selection unit 136 using the training data used in the optimization process. Thus, the second learning unit 137 performs tuning processing to fine-tune the model for service purposes by modifying a part of the model (best model) selected by the model selection unit 136 using the training data used in the optimization process and retraining it.

[0159] (Regarding Section 138) The provisioning unit 138 processes the best model, whose performance has been maximized by the second learning unit 137, as the target for serving. Specifically, the provisioning unit 138 provides the best model, whose performance has been improved by the fine tuning according to the embodiment, in response to access from the user.

[0160] (Regarding attribute selection section 139) When using a pre-trained model to predict a certain target (for example, the click-through rate for advertising content), it may be possible to obtain more accurate results by excluding (i.e., masking) data with specific attributes (e.g., categories) from the input data for prediction, and inputting only the remaining data, compared to inputting all the data.

[0161] Therefore, it is thought that the accuracy of the model can be improved by optimizing the data to be input into the trained model by deciding which attributes of the input candidate data should not be input into the trained model. Accordingly, the attribute selection unit 139 selects target attributes from the input candidate data to be input into the model (e.g., the best model) trained by the learning unit (e.g., the first learning unit 135), which attributes of the data to be excluded from input. For example, the attribute selection unit 139 selects a combination of target attributes.

[0162] For example, the attribute selection unit 139 measures the accuracy of the model when training data containing attributes other than the target attribute in each candidate combination of target attributes is input to the model, and selects a combination of target attributes from among the candidates according to the measurement result.

[0163] The provisioning unit 138 may also provide the user with information indicating attributes other than the target attribute selected by the attribute selection unit 139. For example, as information indicating attributes other than the target attribute selected by the attribute selection unit 139, the provisioning unit 138 may provide information regarding the accuracy of the model when training data having attributes other than the target attribute selected by the attribute selection unit 139 is input to the model.

[0164] [7. An example of optimization processing according to the embodiment] From here, we will show examples of the first, second, third, fourth, and fifth optimization algorithms, which are optimization algorithms according to the embodiment.

[0165] In the example shown in Figure 3, the first to fifth optimization algorithms are executed sequentially within a series of learning processes. However, each of the first to fifth optimization algorithms may be executed independently or in any combination. For example, within the learning process shown in Figure 3, a configuration in which only the first optimization algorithm is executed may be adopted, or a configuration in which only the second and third algorithms are executed may be adopted.

[0166] [7-1-1. Regarding the first optimization algorithm] In deep learning, the optimal model parameters (e.g., weights and biases) are determined by repeatedly updating them. Therefore, initial values ​​for the model parameters are set in advance to facilitate updates. However, the learning results of the neural network change depending on these initial values. For this reason, it is considered necessary to optimize the settings to ensure appropriate initial values ​​are set.

[0167] For example, in deep learning, pseudorandom numbers are often used to initialize model parameters, but if the variation in the initial values ​​is too large or too small, the learning speed will be slow and the accuracy of the model may not improve. For this reason, it is important to set the initial values ​​of the model parameters more appropriately. The first optimization algorithm is an algorithm for optimizing the random number seed that is the source of the pseudorandom numbers so that more appropriate initial values ​​can be generated for the model parameters.

[0168] Therefore, the generation unit 131 uses a first optimization algorithm to optimize the random number seed used to generate the initial values ​​of the model parameters, so that there is no variation in the initial values ​​of each model parameter due to the initial values ​​of the model parameters being completely random. In other words, the generation unit 131 optimizes the random number seed so that the distribution of the generated model parameters falls within a predetermined distribution.

[0169] For example, the generation unit 131 generates multiple random number seeds such that the initial values ​​of the model parameters are within a predetermined range. Alternatively, the generation unit 131 generates multiple random number seeds such that the distribution of the initial values ​​of the model parameters follows a predetermined probability distribution (e.g., a uniform distribution or a normal distribution). Alternatively, the generation unit 131 generates multiple random number seeds such that the average value obtained by averaging the initial values ​​of each model parameter is a predetermined value.

[0170] Then, the generation unit 131 inputs each generated random number seed into a random function, thereby generating initial values ​​for the model parameters corresponding to each random number seed from the output random numbers.

[0171] For example, when the generation unit 131 generates model parameters that exhibit a uniform distribution in response to user instructions, it can select the initialization function "glorot_uniform" for initialization using Glorot's uniform distribution (also called Xavier's uniform distribution) as the random function (initialization function). Glorot's uniform distribution corresponds to a uniform distribution with range [limit, -limit] when limit is sqrt(6 / (fan_in + fan_out)).

[0172] Furthermore, for example, when the generation unit 131 generates model parameters that exhibit a uniform distribution in response to instructions from the user, it can also select the initialization function "he_uniform" as the random function (initialization function) for initialization using the uniform distribution of He. The uniform distribution of He corresponds to a uniform distribution with range [limit, -limit] when limit is sqrt(6 / fan_in).

[0173] The generation unit 131 then inputs the generated random number seed into the selected initialization function and generates initial values ​​for the model parameters from the output random numbers (pseudorandom numbers). The distribution of the random numbers and model parameters obtained here is uniform.

[0174] Furthermore, the generation unit 131 generates models that have initial values ​​for each model parameter. Specifically, the generation unit 131 generates a model for each initial value of the model parameter. For example, the generation unit 131 generates a model for each set of model parameters with different combinations from a group of initial values ​​for model parameters that fall within a predetermined distribution (e.g., uniform distribution, normal distribution, mean).

[0175] [7-1-2. Regarding the fourth optimization algorithm] For model training, it is considered important that the data is shuffled well within the shuffle buffer. However, simply shuffling the data is not enough; for example, biases in the training order or the data distribution per batch may occur, resulting in poor training. In such cases, it is not possible to improve the accuracy of the model.

[0176] Therefore, the first learning unit 135 uses the fourth optimization algorithm to optimize data shuffling, which shuffles the training data in the shuffle buffer.

[0177] Specifically, the first learning unit 135 optimizes the seed value used when generating random orders. For example, the first learning unit 135 uses the fourth optimization algorithm to generate a random seed for each epoch of iterative learning, ensuring that there is no bias in the random orders associated with each training data between epochs. The first learning unit 135 then inputs each generated random seed into a random function to generate a random order. The first learning unit 135 also associates the generated random orders with each training data to be learned, thereby generating the final training data for the target in a shuffle buffer.

[0178] In this regard, for example, the first learning unit 135 generates multiple random number seeds for each epoch for iterative learning, such that the random order follows a predetermined probability distribution (e.g., a uniform distribution or a normal distribution), so that there is no bias in the random order associated with each training data between epochs.

[0179] Furthermore, the first learning unit 135 can optimize data shuffling according to the current shuffle buffer size using an optimization function for data shuffling, such as dataset = dataset.shuffle(buffer_size, seed = seed, reshuffle_each_iteration = True).

[0180] [7-1-3. Examples of experimental results using the first and fourth optimization algorithms] Next, using Figures 7 to 9, we will explain an example of the effects when the first and fourth optimization algorithms are executed.

[0181] Figure 7 is (1) showing the change in model performance when the first and fourth optimization algorithms are executed. Specifically, Figure 7 shows a histogram comparing the accuracy distribution of the same model when the first and fourth optimization algorithms are executed and when they are not.

[0182] In the example shown in Figure 7, the training data used is the same whether the first and fourth optimization algorithms are executed or not, and the number of trials is also the same (e.g., 1000 times). The histogram shown in Figure 7 was obtained by plotting the recall rate on the horizontal axis and the number of trials on the vertical axis.

[0183] The histogram in Figure 7 shows that when the first and fourth optimization algorithms were not performed, the recall rate was "0.1793" even in the best trial, whereas when the first and fourth optimization algorithms were performed, the recall rate improved to "0.1840" in the best trial. Thus, the experimental results indicate that the accuracy of the model is improved by performing the first and fourth algorithms. In other words, the experimental results show that optimizing the computation graph and the random number seed for data shuffling improves the performance of the model.

[0184] Figure 8 is (2) showing the change in model performance when the first and fourth optimization algorithms are executed. Specifically, Figure 8 shows a graph comparing how the accuracy of the model changes when the first and fourth optimization algorithms are executed on the same model and when they are not. The graph shown in Figure 8 was obtained by plotting the number of epochs on the horizontal axis and the average loss on the vertical axis.

[0185] The graph in Figure 8 shows that when the first and fourth optimization algorithms were not executed, the average loss was reduced to "0.008213" through repeated learning, whereas when the first and fourth optimization algorithms were executed, the average loss was further reduced to "0.008208" through repeated learning. Thus, the experimental results show that the accuracy of the model is improved by executing the first and fourth algorithms. In other words, the experimental results show that the model's performance is improved by optimizing the computation graph and the random number seed for data shuffling.

[0186] Furthermore, it was investigated whether the model's performance changes when either the first optimization algorithm or the fourth optimization algorithm is executed alone, or when the first and fourth optimization algorithms are executed in combination. Figure 9 shows a comparative example comparing the model's performance according to the combination of the first and fourth optimization algorithms.

[0187] Figure 9 shows three graphs (Graphs G91, G92, and G93) obtained by plotting recall on the horizontal axis and the number of trials on the vertical axis. In Graphs G91, G92, and G93, the model, training data, and number of trials used in the experiment are all the same.

[0188] Furthermore, graph G91 is a histogram showing the accuracy distribution of the model when only the first optimization algorithm is executed. Graph G92 is a histogram showing the accuracy distribution of the model when only the fourth optimization algorithm is executed. Graph G93 is a histogram showing the accuracy distribution of the model when both the first and fourth optimization algorithms are executed.

[0189] Comparing graphs G91 to G93, we can see that they all have almost the same accuracy distribution. Therefore, the experimental results indicate that there is no significant difference in model performance when only the first optimization algorithm is executed, when only the fourth optimization algorithm is executed, or when both the first and fourth optimization algorithms are executed; the model's performance is maintained in all cases.

[0190] [7-2. Regarding the second optimization algorithm] In deep learning, the training dataset is divided into several subsets, and each subset is used for training as the epoch progresses. However, using all subsets to train the model does not necessarily result in the best-performing model. Furthermore, as the amount of training data increases, the time spent on training and the computer resources occupied become problematic, so it is necessary to narrow down the effective subsets to be used for training and make the training more efficient. The optimization process that was realized based on these premises is the second optimization algorithm. Below, a more detailed example of the second optimization algorithm described above is explained in Figure 10.

[0191] Figure 10 shows an example of the second optimization algorithm. The series of processes shown in Figure 10 corresponds to the process in step S13 shown in Figure 3.

[0192] First, the acquisition unit 132 acquires training data from the training data storage unit 121 and outputs the acquired training data to the first data control unit 133. When the first data control unit 133 receives the training data from the acquisition unit 132, it uses the second optimization algorithm to execute the following processes.

[0193] As explained in Figure 6, the training data has a time-series concept. More specifically, since the training data set consists of a predetermined number of training data, each training data is associated with, for example, time information as history.

[0194] Therefore, first, the first data control unit 133 sorts the included training data so that it is arranged in chronological order (S131). Next, the first data control unit 133 divides the sorted training data into a predetermined number of sets (step S132). For example, the first data control unit 133 can divide the training data into a predetermined number of sets such that each set contains an equal number of training data (for example, a number specified by the user). Alternatively, the first data control unit 133 may divide the training data into a predetermined number of sets such that each set contains a number of training data within a predetermined range.

[0195] Figure 10 shows an example in which the first data control unit 133 divides the training data set, resulting in data files corresponding to each set: "File #1", "File #2", "File #3", "File #4", "File #5", "File #6", "File #7", "File #8", "File #9", "File #10", and "File #11".

[0196] Furthermore, each of these data files contains training data arranged in chronological order. Therefore, as shown in the example in Figure 10, the larger the file number of a data file, the more recent the chronological order of the training data it contains. For example, when comparing "File #2" with another set, "File #3," "File #3" contains training data with a more recent chronological order.

[0197] Next, the first data control unit 133 selects a predetermined number of sets from all the sets obtained by the partitioning in step S132 to be used for training the model (step S133). For example, the first data control unit 133 randomly selects sets from all the sets obtained by the partitioning in step S132 to be used for training the model until the number of selected sets reaches a predetermined number. For example, the first data control unit 133 randomly selects sets from all the sets obtained by the partitioning in step S132 until a predetermined number is reached (for example, a number specified by the user). Alternatively, the first data control unit 133 randomly selects sets in order from the sets with the most recent time series of the training data included (file #11 in the example in Figure 10) until a predetermined number is reached (for example, a number specified by the user). Figure 10 shows an example in the first loop where the first data control unit 133 randomly selects four sets of training data, starting with the most recent set in terms of time series (selection order according to time series), such as "file #11", "file #9", "file #8", and "file #6".

[0198] As will be explained later, the process from step S133 is repeated until the specified number of loops is reached. Specifically, the process involves randomly selecting sets from the sets obtained by the splitting in step S132 that have not yet been selected, until a predetermined number is reached, or randomly selecting sets from the sets obtained by the splitting in step S132 that have not yet been selected, starting with the sets whose included training data have the most recent time series, until a predetermined number is reached. This process is repeated in each loop until the specified number of loops is reached. Therefore, for example, in the second loop, "file #10" may be randomly selected, as may "file #7", "file #5", and "file #4".

[0199] Next, the first data control unit 133 generates a single data set by concatenating the sets selected in step S133 (step S134). For example, the first data control unit 133 generates a single data set by concatenating the sets selected in step S133 in the order they were selected. The order of selection here corresponds to the order of selection in step S133, and specifically, it is the order in which sets of training data are selected to be used for training the model in order of the newest time series of the training data they contain.

[0200] Furthermore, the first data control unit 133 can pass the generated data set to the second data control unit 134 so that the training data included in the generated data set can be used for training. In the example in Figure 10, the first data control unit 133 passes a data file named "file #X" containing the generated data set to the second data control unit 134. As shown in Figure 10, in "file #X", the files "file #6", "file #8", "file #9", and "file #11" are arranged in the order they were selected. In other words, in "file #X", the training data is arranged in the order it was selected.

[0201] [7-3-1. Regarding the third optimization algorithm] In deep learning, when training a model, it is considered important that the dataset is properly batched and repeatedly trained to improve the model's accuracy. Furthermore, the order in which each subset of the training dataset is trained is also thought to contribute to the model's performance. Based on these premises, the optimization process that was realized is the third optimization algorithm. Below, a more detailed example of the third optimization algorithm described above is shown in Figure 11.

[0202] Figure 11 shows an example of the third optimization algorithm. Figure 11 also shows the fourth optimization algorithm. Furthermore, the series of processes shown in Figure 11 corresponds to the processes from steps S13 to S14 shown in Figure 3.

[0203] For example, the second data control unit 134 optimizes the shuffle buffer size using a third optimization algorithm. For example, as part of the shuffle buffer size optimization, the second data control unit 134 generates training data of a size equal to the shuffle buffer size and stores this data in the shuffle buffer as the training data to be learned, which is the training data to be used in this iterative learning. For example, as an example of such processing, the second data control unit 134 executes the following process following step S134 in Figure 10.

[0204] For example, the second data control unit 134 divides the training data set, which is bundled together as "file #X" (in this case, the training data is arranged in the order in which it was selected), into a predetermined number of sets (step S135). For example, the second data control unit 134 can divide the training data set into a predetermined number of sets such that each set equally contains a predetermined number of training data (for example, a number specified by the user). Alternatively, the second data control unit 134 may divide the training data set into a predetermined number of sets such that each set contains a number of training data within a predetermined range.

[0205] For example, the user can specify how to divide the training data set contained in "file #X" using various hyperparameters such as the upper limit (maxValue), lower limit (minValue), and minimumUnit. In other words, the user can specify the shuffle buffer size using the above hyperparameters. Therefore, the second data control unit 134 can optimize the shuffle buffer size based on the division content specified by the user. For example, the second data control unit 134 selects a shuffle buffer size according to the division content specified by the user and divides the training data set contained in "file #X" according to the selected shuffle buffer size.

[0206] For example, suppose the above hyperparameters are used to optimize a shuffle buffer size capable of storing "10,000" records to a shuffle buffer size corresponding to "2,500" records. In this case, the second data control unit 134 divides the 10,000 training data sets into two training data sets of 2,500 each.

[0207] Experiments have shown that the accuracy of the model changes depending on how many training data points are included in each set, i.e., how the shuffle buffer size is set. The results of this experiment are explained in Figure 12, and for example, these results may be reflected in the third optimization algorithm. Specifically, the second data control unit 134 may optimize the shuffle buffer size (the number of training data points to include in each set) using the third optimization algorithm that reflects the experimental results shown in Figure 12.

[0208] Furthermore, Figure 11 shows an example in which the second data control unit 134 divides the training data set contained in "file #X" to obtain four sets of training data sets: training data set #1 (Data #1), training data set #2 (Data #2), training data set #3 (Data #3), and training data set #4 (Data #4). In addition, according to the example in Figure 11, the second data control unit 134 stores training data set #1 in "file #X1", training data set #2 in "file #X2", training data set #3 in "file #X3", and training data set #4 in "file #X4".

[0209] Furthermore, as shown in the example in Figure 11, the training data sets are arranged from top to bottom in the order in which they were obtained by the splitting in step S135 (splitting order).

[0210] Next, the second data control unit 134 extracts one set from the sets obtained by the division in step S135 that are currently unprocessed and not being used for learning, according to the division order, and stores the extracted set in the shuffle buffer as the learning data to be learned, which is the learning data to be used in this iterative learning (step S136).

[0211] In the example shown in Figure 11, the second data control unit 134 extracts "file #X1", which is the first set obtained by the splitting. The second data control unit 134 then stores the training data contained in the extracted "file #X1" in the shuffle buffer as the training data to be trained.

[0212] Furthermore, in response to the fact that training data of a size (number) corresponding to the shuffle buffer size optimized by the third optimization algorithm has been stored in the shuffle buffer as in step S136, the first learning unit 135 continues the following process from step S136.

[0213] Specifically, the first learning unit 135 uses the fourth optimization algorithm to optimize data shuffling, which involves shuffling the training data to be learned stored in the shuffle buffer. Then, the first learning unit 135 trains each model with the final training data generated by the optimization.

[0214] For example, the first learning unit 135 generates the final training data to be learned by randomly determining the learning order using the fourth optimization algorithm (step S141). In other words, the first learning unit 135 generates the final training data to be learned by randomly determining the order using the fourth optimization algorithm.

[0215] Specifically, the first learning unit 135 uses the fourth optimization algorithm to generate a random seed (the seed that forms the basis of the random order) for each epoch of iterative learning, ensuring that there is no bias in the random order associated with each training data between epochs. The first learning unit 135 then inputs each generated random seed into a random function to generate a random order. Furthermore, the first learning unit 135 associates the generated random order with each training data to be learned, thereby generating the final training data for the target in a shuffle buffer.

[0216] Next, the first learning unit 135 sequentially trains each model on the features of the training data to be learned (the training data contained in "file #X1" stored in the shuffle buffer) in the learning order (random order) generated in step S141 (step S142).

[0217] Here, the first learning unit 135 performs repeated learning for a predetermined number of epochs, with steps S136 to S142 considered as one epoch, using the set obtained by the division in step S135. Specifically, the first learning unit 135 performs repeated learning for a number of epochs specified by the user, using the set obtained by the division in step S135, with steps S136 to S142 considered as one epoch.

[0218] Therefore, the first learning unit 135 first determines whether all of the sets obtained by the division in step S135 have been processed for one epoch (step S143). Specifically, the first learning unit 135 determines whether all of the sets obtained by the division in step S135 (in the example in Figure 11, "file #X1" to "file #X4") have been used for learning, with steps S136 to S142 forming one epoch.

[0219] If the first learning unit 135 determines that it has not been able to process all of the sets obtained by the partition in step S135 for one epoch (step S143; No), it will repeat the series of processes from step S136 to step S142.

[0220] Furthermore, if the first learning unit 135 determines that all the sets obtained by the division in step S135 have been processed for one epoch (step S143; Yes), it then determines whether the specified number of epochs has been reached for the sets obtained by the division in step S135 (step S144). Specifically, the first learning unit 135 determines whether repeated learning has been performed for the specified number of epochs (for example, user-specified) using the sets obtained by the division in step S135.

[0221] If the first learning unit 135 determines that the specified number of epochs has not been reached (step S144; No), it repeats the series of processes from step S136 to step S142.

[0222] On the other hand, if the model selection unit 136 determines that the specified number of epochs has been reached (step S144; Yes), it selects the best model at the present time based on the accuracy of each trained model at that time (step S145). For example, the model selection unit 136 calculates the accuracy of each model using evaluation data, and calculates a higher evaluation value the greater the variation in accuracy (amount of improvement in accuracy). The model selection unit 136 then selects the model with the highest evaluation value as the best model. Note that the method for selecting the best model is not limited to the method described above. Furthermore, the series of processes from step S133 onwards are repeated until the specified number of loops is reached in order to obtain a model with higher accuracy.

[0223] Therefore, the first learning unit 135 then determines whether or not the number of times the process from step S133 is repeated (looped) has been reached (step S146). The number of loops is a hyperparameter that can be specified by the user.

[0224] Therefore, as long as the first learning unit 135 determines that the specified number of loops has not been reached (step S146; No), it repeats the series of processes from step S136. This point will be explained in more detail using the example in Figure 10.

[0225] For example, if it is determined that the specified number of loops has not been reached, the first data control unit 133 performs the process of step S133, which involves randomly selecting, in order, the sets obtained by the division in step S132 that have not yet been selected until the specified number of loops has been reached. Here, for example, in the processes from step S133 onwards that are executed in the second and subsequent loops, the sets used by the best model for training are retained. Specifically, in the processes from step S133 onwards that are executed in the second and subsequent loops, new sets of data to be used for training are added to the sets used by the best model for training. For this reason, the first data control unit 133 selects sets of training data to be added to the sets used by the best model for training in the second and subsequent loops.

[0226] Furthermore, as shown in the example above, in the second loop, for example, "file #10" may be selected, as well as "file #7," "file #5," and "file #4" randomly.

[0227] Furthermore, based on previous examples, the model selection unit 136 can select the model with the highest accuracy at that point once it reaches the specified number of loops.

[0228] [7-3-2. An example of experimental results regarding the third optimization algorithm] Furthermore, when applying the third optimization algorithm, experiments were conducted to verify how to optimize the shuffle buffer size—that is, how many training data points should be included in each set—to effectively improve the model's accuracy. Figure 12 shows a comparison of the model's performance for each shuffle buffer size.

[0229] Figure 12 shows five graphs (Graphs G121, G122, G123, G124, and G125) obtained by plotting recall on the horizontal axis and the number of trials on the vertical axis. In graphs G121 to G125, the model, training data, and number of trials used in the experiment are all the same.

[0230] Furthermore, graph G121 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "1,000K" for a certain set of data including training data. Graph G122 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "2,000K" for a similar set of data. Graph G123 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "3,000K" for a similar set of data. Graph G124 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "4,000K" for a similar set of data. Graph G125 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "6,000K" for a similar set of data.

[0231] Comparing graphs G121 to G125, we can see that the model accuracy differs in each case. This suggests that optimizing the shuffle buffer size improves the model's performance. Therefore, it was found that optimizing the shuffle buffer size by executing the third optimization algorithm may improve the model's performance. Furthermore, the third optimization algorithm can be said to be an idea conceived from the experimental results obtained as shown in Figure 12.

[0232] Furthermore, the third optimization algorithm may reflect the experimental results shown in Figure 12. Specifically, the second data control unit 134 may optimize the shuffle buffer size (the number of training data points included in one set) using the third optimization algorithm that reflects the experimental results shown in Figure 12.

[0233] In this regard, in the example shown in Figure 12, since the number of data records is "5,518K", it was expected that the model would perform best with a shuffle buffer size of "6,000K", which is large enough to store all of this data. However, as shown in Figure 12, this experiment revealed that the model's performance may actually be best with a shuffle buffer size of "2,000K". Therefore, based on these experimental results, for example, the third optimization algorithm could be an algorithm that optimizes the shuffle buffer size to "2,000K". Alternatively, the third optimization algorithm could be an algorithm that optimizes the shuffle buffer size to be 1 / 3 of the total size (total number) of the training data.

[0234] Furthermore, using the example in Figure 11, users can appropriately consider how to divide the training data set contained in "File #X" based on such experimental results. For example, users can consider more optimal values ​​for various hyperparameters such as the upper limit (maxValue), lower limit (minValue), and minimumUnit.

[0235] [7-4-1. Regarding the fifth optimization algorithm] Furthermore, in deep learning, the optimal hyperparameters are searched for by repeatedly training the model to achieve the desired accuracy and generalization performance. However, depending on the algorithm used, the amount of data, and the computing environment, a single trial can take several hours. For example, in grid search, the optimal parameters are selected by searching for all possible hyperparameters within the range. In this case, as the number of types of hyperparameters increases, the number of combinations increases, leading to problems in terms of time and computer resource usage. Based on these premises, the optimization process that was realized is the fifth optimization algorithm. Below, a more detailed example of the fifth optimization algorithm described above is explained in Figure 13.

[0236] Figure 13 shows an example of condition information related to the fifth optimization algorithm. During the learning process, trials to search for hyperparameters are repeated, but the fifth optimization algorithm is executed as an optimization of trials by pruning in order to achieve efficient search. Specifically, the first learning unit 135 uses the fifth optimization algorithm to optimize trials by early stopping trials that are not expected to yield good results, without completing them.

[0237] Furthermore, the information processing device 100 can allow the user to set constraints that condition which trials will be targeted for early stopping (trials to be terminated early), from the perspective of evaluation values ​​that evaluate the accuracy of the model. For example, the information processing device 100 can allow the user to set multiple such constraints in combination. Figure 13 shows an example of constraints that can be set by the user. Note that the constraints shown in Figure 13 are just examples, and the user can set any number of arbitrary constraints in any combination in the information processing device 100. In addition, although not shown in Figure 5, the information processing device 100 may further have a reception unit that accepts the setting of constraints.

[0238] Furthermore, for each trial (trial with a different combination of hyperparameters), the first learning unit 135 determines whether the evaluation value (evaluation value for evaluating the accuracy of the model) for the hyperparameter combination corresponding to that trial satisfies the constraints. When it determines that the constraints are satisfied, it stops the trial that was the subject of the evaluation. The first learning unit 135 then continues only the remaining trials that were not stopped.

[0239] Next, we will explain the constraints shown in Figure 13. Figure 13 shows an example of a stopping condition (constraint) that conditions the trial to stop (prune) earlier than the learning process reaches all epochs. Specifically, Figure 13 shows five stopping conditions C1 to C5.

[0240] According to the stopping condition C1, the condition is set as follows: "function: stop_if_no_decrease_hook", "mtric_name: average_loss", "max_epochs_without_decrease: 3", and "min_epochs: 1". The example shows that stopping condition C1 "conditions the trial to stop if the average loss did not decrease (accuracy did not improve) for a maximum of 3 epochs".

[0241] Furthermore, according to the stopping condition C2, the condition is set as follows: "function: stop_if_no_decrease_hook", "mtric_name: auc", "max_epochs_without_increase: 3", and "min_epochs: 1". The example shows that the stopping condition C2 is "conditioned to stop trials where the auc did not increase (accuracy did not improve) for a maximum of 3 epochs".

[0242] Furthermore, according to the stopping condition C3, the condition is set as follows: "function: stop_if_lower_hook", "mtric_name: accuracy", "threshold: 0.8", and "min_epochs: 3". The example shows that stopping condition C3 is "conditioned to stop trials after 3 epochs in which the accuracy does not exceed the threshold of 0.8".

[0243] Furthermore, according to the stopping condition C4, the condition is set as follows: "function: stop_if_higher_hook", "mtric_name: loss", "threshold: 300", and "min_epochs: 5". The example shows that stopping condition C4 is "conditional to stop trials where the loss exceeds the threshold of 300 after 5 epochs".

[0244] Furthermore, according to the stopping condition C5, the condition is set as follows: "function: stop_if_not_in_top_k_hook", "mtric_name: auc", "top_k: 10", "epochs: 3". The example shows that stopping condition C5 is "conditioned to stop trials where the auc is not in the top 10 after 3 epochs".

[0245] [7-4-2. An example of experimental results when using the fifth optimization method] Next, using Figure 14, we will explain an example of a process in which the trial is stopped using the fifth optimization algorithm. Figure 14 is a diagram illustrating an example of the fifth optimization algorithm. In addition, the example in Figure 14 shows a scenario in which the fifth optimization algorithm is applied in combination with stopping conditions C6 and C7.

[0246] According to the stopping condition C6, the condition is set as follows: "function: stop_if_not_in_top_k_hook", "mtric_name: recall", "top_k: 8", "epochs: 3". The example shows that stopping condition C6 "conditions the trial to stop if recall is not among the top 8 after 3 epochs".

[0247] According to the stopping condition C7, the condition is set as follows: "function: stop_if_not_in_top_k_hook", "mtric_name: recall", "top_k: 4", "epochs: 6". The example shows that stopping condition C7 "conditions the program to stop any trial where recall is not among the top 4 at epoch 6".

[0248] Furthermore, Figure 14 shows an example in which, using a predetermined number of devices (e.g., 16), individual trials with different combinations of hyperparameters are processed in parallel, and the first learning unit 135 monitors the variation in the recall rate, which is an evaluation value (an evaluation value that evaluates the accuracy of the model) for each trial corresponding to the hyperparameter combination, and determines whether the pattern based on the variation in recall rate (in the example of Figure 14, the order of trials) satisfies the stopping conditions C6 and C7.

[0249] In this state, the first learning unit 135 stops a trial at the 3rd epoch if the recall is not among the top 8, based on the stopping condition C6. Furthermore, the first learning unit 135 stops a trial at the 6th epoch if the recall is not among the top 4.

[0250] Thus, experimental results showed that by using the fifth optimization algorithm to optimize trials by stopping trials early that are unlikely to improve model performance, processing time was improved by 45%. Specifically, the fifth optimization algorithm, which combines multiple stopping conditions to identify trials that are unlikely to improve model performance and stops those trials early, showed a 45% improvement in processing time. Furthermore, these results suggest that the fifth optimization algorithm can solve problems such as time and computer resource usage.

[0251] Furthermore, users may be required to set effective shutdown conditions to efficiently utilize computer resources. For this reason, the information processing device 100 may provide information to help users consider what kind of shutdown conditions they should set. For example, the information processing device 100 may provide a screen that displays the current optimization status for each trial so that the user can visually check the optimization status. For example, the information processing device 100 can deliver the screen displaying the current optimization status for each trial to the terminal device 3 in response to access from the terminal device 3 owned by the user.

[0252] With such an information processing device 100, it becomes possible to visually and easily recognize trials that are unlikely to improve the model's performance. Therefore, it becomes possible to consider effective stopping conditions to stop trials that are unlikely to improve the model's performance early on.

[0253] The screen displaying the optimization status may be provided, for example, by the providing unit 138, or by other processing units.

[0254] [7-5-1. Optimization of Mask Targets] So far, we have shown the first to fifth optimization algorithms as algorithms for optimizing the learning method. In addition to these optimizations, the information processing device 100 may also optimize the data to be masked, determining which data from the candidate input data to be input to the trained model will not be input to the model. Specifically, the information processing device 100 uses an algorithm to optimize the data to be masked to select the data to be not input to the model from the candidate input data to be input to the trained model.

[0255] For example, when using a pre-trained model to predict an object, it may be possible to obtain more accurate results by excluding (i.e., masking) data with certain attributes (e.g., categories) from the input data and inputting only the remaining data, compared to inputting all the data. In other words, the accuracy of a pre-trained model can sometimes be improved by excluding (i.e., masking) data with certain attributes (e.g., categories) and inputting only the remaining data, rather than inputting all the data.

[0256] According to this, it is necessary to optimize the data that should be input to the trained model by determining which attributes of the candidate input data should not be input to the trained model. Based on this premise, the optimization process that was realized is the masked optimization algorithm.

[0257] For example, the attribute selection unit 139 uses an optimization algorithm for masked data to select target attributes from the input candidate data to be input to the trained model, which data having certain attributes will not be input to the model. For example, for each candidate combination of target attributes, the attribute selection unit 139 measures the accuracy of the model when training data having attributes other than the target attribute in that candidate is input to the model, and selects a combination of target attributes from the candidates according to the measurement result.

[0258] Here, for example, when predicting a certain target (e.g., the click-through rate of an advertisement) using the best model selected by the model selection unit 136, it was hypothesized that better prediction results could be obtained by not inputting data with specific attributes from the test data used for prediction, and inputting only the remaining test data excluding the non-input data into the best model, compared to inputting all of the test data.

[0259] Figure 15 illustrates an example of optimization of a masked object, using experimental results that verify the effectiveness of the optimization algorithm for the masked object based on the aforementioned hypothesis. Figure 15 is a diagram showing an example of an optimization algorithm for optimizing a masked object.

[0260] Here, the learning data (which may also be evaluation data) used in the optimization process so far has multiple attributes. For example, the learning data is classified into various categories such as learning data related to "business", learning data related to "economy", learning data related to "gender", and learning data related to "users' interests". Therefore, the learning data has attributes such as these categories, for example.

[0261] Therefore, for example, the attribute selection unit 139 measures the accuracy (recall rate) of the model when the learning data included in the other categories excluding the categories in the combination is input to the best model for each combination of categories that holds for the categories of the learning data. Then, according to the measurement results, for example, when the highest accuracy is obtained, based on which combination in the combination of categories was excluded, among the test data (from FIG. 6) paired with this learning data, the target category (target attribute) that is the category of the data not to be input to the best model is selected.

[0262] Also, from such a situation, the attribute selection unit 139 automatically searches for combinations of categories (attributes) whose masking improves the performance of the model. For example, the attribute selection unit 139 can search for combinations of categories (attributes) whose masking improves the performance of the model using a genetic algorithm.

[0263] In FIG. 15, the recall rate in each trial by the attribute selection unit 139 is plotted. Also, FIG. 15 shows an example of a combination of attributes when the highest accuracy is obtained. For convenience of explanation, such a combination of categories is defined as "combination CB".

[0264] Then, when the highest accuracy is obtained, since the combination CB was excluded among the category combinations, the attribute selection unit 139 determines that the data included in the category in the combination CB is non-input target data that is not input to the best model. That is, the attribute selection unit 139 determines that when inputting the test data to the best model by selecting the combination CB as the target attribute, the data included in the category in the combination CB is to be masked.

[0265] In addition, the providing unit 138 can provide information indicating categories other than the categories selected by the attribute selection unit 139 and the best model. The information indicating categories other than the categories selected by the attribute selection unit 139 is, for example, information regarding the accuracy of the best model when the learning data included in the categories other than the categories selected by the attribute selection unit 139 is input to the best model, and may be, for example, the recall rate shown in FIG. 15.

[0266] Also, according to the fact that such information provision is performed in accordance with the optimization of the masking target, for example, when the user wants to predict the target using the best model, instead of inputting all the data of the test data prepared by himself / herself, the user can know that for the data having a specific attribute, it should be masked and only the remaining data should be input. As a result, the user can obtain a more legitimate prediction result than when using all of the test data. Also, from this, the information processing apparatus 100 having an optimization function for optimizing the masking target can assist the user in obtaining a more legitimate result using the learned model.

[0267] 〔7-5-2. An Example of Experimental Results When Using Optimization of Masking Target〕 As explained above, when masked optimization is performed, some of the test data is not input, so the actual number of test data points input is less than when masked optimization is not performed. Therefore, an experiment was conducted to verify whether the reduction in the number of test data points input due to masked optimization would affect the accuracy of the model. Figure 16 shows a comparison of the model accuracy with and without masked optimization.

[0268] Figure 16 compares the evaluation results (recall) when the model was evaluated using the evaluation data used during training with the evaluation results (recall) when the model was evaluated using the remaining data after optimizing the evaluation data to exclude data with selected attributes. As shown in the comparative example in Figure 16, the experiment revealed that the generality of the model is maintained even after performing optimization on the masked data.

[0269] In the above example, the information processing device 100 determines which attributes of the input candidate data to be input to the trained model will not be input to the trained model. By doing so, data with the determined attribute is masked, and only data with attributes other than the determined attribute is used. However, instead of controlling the information processing device 100 to mask some of the input candidate data to be input to the trained model in this way, it may also control the system so that, for example, learning using the optimization of the masked data is performed while learning is being carried out using the fifth optimization algorithm described above.

[0270] Specifically, the information processing device 100 further includes a determination unit that determines multiple new combinations of target attributes based on combinations of target attributes in multiple models whose accuracy satisfies predetermined conditions, and determines whether the accuracy of each model satisfies these predetermined conditions when training data having attributes other than the target attributes in the determined combinations is input to the multiple models. The first learning unit 135 then trains the models that the determination unit has determined to satisfy the predetermined conditions with the training data. The first learning unit 135 may also perform the processing of the determination unit.

[0271] [8. Configuration of the execution control unit] Up to this point, we have focused on explaining the first to fifth optimization algorithms, as well as the information processing device 100, which has an optimizer OP function that performs the optimization algorithm for the masked target. From here, we will explain the execution control device 200. First, the background to the realization of the execution control device 200 is as follows.

[0272] For example, if we consider a case where a pre-trained model is used to predict an object, the computer uses the pre-trained model to predict whether an arbitrary image data is the same as the ground truth image data. This prediction process includes multiple steps, such as extracting features from an image, i.e., a two-dimensional array of pixels, and detecting matching parts from another image.

[0273] Each process included in the prediction process is executed by the computer's processor, but the total processing time spent on the prediction process varies depending on which device in the processor performs which process.

[0274] Therefore, in order to further reduce the overall processing time spent on prediction, it is important to optimize the execution entity of each process so that the most suitable device (processing unit) is assigned to perform that process. However, it is impossible for a computer to dynamically determine the optimal execution entity.

[0275] Based on these premises, the execution control unit 200 performs a process to optimize the execution entity that will execute a process using the model (for example, a process that predicts a specific target). Specifically, the execution control unit 200 optimizes the execution entity by determining which execution entity will execute the process using the model (for example, a process that predicts a specific target) based on the characteristics of the trained model. Therefore, the execution control unit 200 has an execution entity optimization algorithm.

[0276] First, the execution control device 200 according to the embodiment will be described using Figure 17. Figure 17 is a diagram showing an example of the configuration of the execution control device 200 according to the embodiment. As shown in Figure 17, the execution control device 200 has a communication unit 210, a storage unit 220, and a control unit 230.

[0277] (Regarding memory unit 220) The storage unit 220 is implemented by, for example, semiconductor memory elements such as RAM and flash memory, or storage devices such as hard disks and optical discs. The storage unit 120 has a model architecture storage unit 221.

[0278] (Regarding the model architecture memory unit 221) The model architecture memory unit 221 stores the architecture of the neural network. Here, Figure 18 shows an example of the model architecture memory unit 221 according to the embodiment. In the example in Figure 18, the model architecture memory unit 221 has items such as "Model ID" and "Architecture Information".

[0279] The "Model ID" indicates identification information that identifies the model. The "Architecture Information" is information that describes the characteristics of the model identified by the "Model ID". Specifically, the "Architecture Information" is information that describes the overall structure, including the learning mechanism of the model identified by the "Model ID".

[0280] In the example of FIG. 18, an example is shown in which the model ID "MD#1" and the architecture information "architecture#1" are associated. Such an example shows an example in which the architecture of the model identified by the model ID "MD#1" is "architecture#1". Note that in FIG. 18, the architecture of the neural network is conceptually shown as "architecture#1", but in reality, legitimate information indicating the architecture of the neural network is registered.

[0281] (Regarding control unit 230) The control unit 230 is realized by various programs stored in the storage device inside the execution control device 200 being executed with the RAM as a work area by a CPU, MPU, etc. Further, the control unit 130 is realized by an integrated circuit such as an ASIC or an FPGA, for example.

[0282] As shown in FIG. 17, the control unit 230 has a specifying unit 231, a determining unit 232, and an execution control unit 233, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 230 is not limited to the configuration shown in FIG. 17, and other configurations may be used as long as they perform the information processing described later. Also, the connection relationship of each processing unit included in the control unit 230 is not limited to the connection relationship shown in FIG. 17, and other connection relationships may be used.

[0283] (Regarding specifying unit 231) The specifying unit 231 specifies the characteristics of a model (a learned model) used when a plurality of arithmetic units with different architectures execute a predetermined process (for example, a process such as estimation using a model). For example, the specifying unit 231 specifies the characteristics of a plurality of processes executed as a model as the characteristics of the model.

[0284] (Regarding determining unit 232) The determination unit 232 determines which of the multiple computing units will execute the processing using the model, based on the characteristics of the model identified by the identification unit 231. For example, based on the characteristics of the multiple processes identified by the identification unit 231, the determination unit 232 determines which of the multiple computing units will execute each process.

[0285] For example, the determination unit 232 determines the arithmetic unit to be executed from among a plurality of arithmetic units, specifically a first arithmetic unit that is guaranteed to output the same value when the same processing is performed using the same data, and a second arithmetic unit that is not guaranteed to output the same value when the same processing is performed using the same data.

[0286] Furthermore, for example, the determination unit 232 determines which arithmetic unit to execute from among a plurality of arithmetic units, namely a first arithmetic unit that performs scalar operations and a second arithmetic unit that performs vector operations.

[0287] Furthermore, for example, the determination unit 232 determines the arithmetic unit to be executed from among a plurality of arithmetic units, namely a first arithmetic unit employing an out-of-order method and a second arithmetic unit that does not employ an out-of-order method.

[0288] In other words, the decision unit 232 determines which computing device to execute from either a central computing unit (CPU) with branch prediction functionality as the first computing device, or an image processing unit (GPU) without branch prediction functionality as the second computing device. For example, if the model is a multi-class classification model, the decision unit 232 determines an image processing unit as the computing device to execute. On the other hand, if the model is a two-class classification model, the decision unit 232 determines a central computing unit as the computing device to execute.

[0289] (Regarding the execution control unit 233) The execution control unit 233 causes the arithmetic unit determined by the decision unit 232 to execute processing using the model.

[0290] [9-1. An example of the operation of the execution control unit] From here, we will describe an example of the processing performed by the execution control device 200 using the execution entity's optimization algorithm.

[0291] For example, a user might want to operate a model whose performance has been enhanced through the fine-tuning performed by the information processing device 100, as described above, in a production environment (e.g., a server or edge device). Specifically, the user might want to operate a model whose performance has been enhanced through the fine-tuning performed by the information processing device 100 on a server that supports a specific service.

[0292] In the following, we will explain the cases separately for the following: when the model in question (for example, the best model) is a multi-class classification model, model MD1 (a model identified by model ID "MD#1") (pattern PT1), and when it is a two-class classification model, model MD2 (a model identified by model ID "MD#2") (pattern PT2).

[0293] Both the processing using Model MD1 and the processing using Model MD2 are assumed to be predictive processes that predict a predetermined target. Furthermore, according to the above example, the predictive processing using Model MD1 and the predictive processing using Model MD2 are performed by a server corresponding to the user's production environment (for example, an API server).

[0294] (Regarding pattern PT1) The identification unit 231 uses the model ID "MD#1" to refer to the model architecture storage unit 221 and identify the architecture of the neural network corresponding to model MD1. The architecture specifies the computing device on which to execute each of the multiple processes executed as a model (for example, a process to extract features from an image and a process to detect matching parts of features from another image). For example, the architecture specifies whether only a GPU or a CPU is the computing device on which to execute each of the multiple processes executed as a model. Thus, the identification unit 231 identifies the architecture of each process included in the prediction process, among the neural network architectures corresponding to model MD1.

[0295] Furthermore, the determination unit 232 determines which computing device, a GPU or a CPU, will execute the process based on the architecture for each process identified by the identification unit 231. For example, if the architecture corresponding to a certain process A1 identified by the identification unit 231 specifies that process A1 should be executed by the GPU, the determination unit 232 will determine that the GPU will be the computing device to execute process A1. Similarly, if the architecture corresponding to another process A2 identified by the identification unit 231 specifies that process A2 should be executed by the CPU, the determination unit 232 will determine that the CPU will be the computing device to execute process A2.

[0296] In this state, for example, the execution control unit 233 controls the user's API server to execute process A1 on the GPU and process A2 on the CPU.

[0297] (Regarding pattern PT2) The identification unit 231 uses the model ID "MD#2" to refer to the model architecture storage unit 221 and identifies the architecture of the neural network corresponding to model MD2. Similarly, for this architecture, the computing device on which the process is executed is defined for each of the multiple processes executed as a model (for example, a process to extract features from an image and a process to detect matching parts of features from another image). In other words, for each of the multiple processes executed as a model in this architecture, it is defined whether only a GPU or a CPU is to be used as the computing device on which the process is executed. Thus, the identification unit 231 identifies the architectures representing each of the processes included in the prediction process, for example, among the neural network architectures corresponding to model MD2.

[0298] Furthermore, the determination unit 232 determines which computing device, a GPU or a CPU, will execute the process, based on the architecture for each process identified by the identification unit 231. For example, if the architecture corresponding to a certain process B1 identified by the identification unit 231 specifies that the process should be executed by the CPU, the determination unit 232 will determine the CPU to be the computing device to execute process B1. Similarly, if the architecture corresponding to another process B2 identified by the identification unit 231 specifies that process B2 should be executed by the GPU, the determination unit 232 will determine the GPU to be the computing device to execute process B2.

[0299] Furthermore, the processing of the decision unit 232 will be explained in more detail using Figure 19. Figure 19 is a diagram showing an example of a model architecture to which information indicating the target computing device for execution is associated. Figure 19 shows the architecture of the neural network corresponding to model MD1 that corresponds to process A1. As shown in Figure 19, the architecture of the neural network corresponding to model MD1 that corresponds to process A1 has information indicating the target computing device for executing process A1 pre-built into it. Specifically, in the example in Figure 19, the architecture corresponding to process A1 has a description pre-associated with it that specifies that process A1 should be executed on the GPU. Therefore, based on this description, the decision unit 232 can determine that the GPU is the target computing device for executing process A1.

[0300] Furthermore, for the execution control unit 200 to operate as described above using the execution entity's optimization algorithm, it is necessary that information indicating the target computing device for executing each process is pre-built into the neural network architecture corresponding to the trained model, for each architecture associated with the process using that model. In other words, for each process, the target computing device for executing that process must be given in a rule-based manner.

[0301] Therefore, in order to implement such a rule-based approach, experiments were conducted to verify the difference in processing time when a multi-class classification model was executed on both a GPU and a CPU. Furthermore, experiments were conducted to verify the difference in processing time when a two-class classification model was executed on both a GPU and a CPU.

[0302] [9-2. An example of experimental results regarding the optimization algorithm of the execution entity] From here, we will use Figures 20 to 24 to explain an example of the effects when processing using the model is executed on both the GPU and the CPU.

[0303] (A model for multi-class classification) First, using Figures 20 and 21, we will explain an example of the effect when processing using a multi-class classification model is executed on both the GPU and the CPU. Here, for each multi-class classification model for a given service, the processing that was initially performed on the CPU was arbitrarily combined, and the extent to which performance (processing time) was improved was investigated by having the processing for that combination performed on the GPU. Figure 20 shows the experimental results.

[0304] Figure 20 shows the improvement in performance achieved through experiments targeting a multi-class classification model. For example, Figure 20 shows the elements that resulted in the best performance in the above experiment.

[0305] In the example shown in Figure 20, for the model corresponding to service SV1 (model "1"), an experiment was conducted to see how much performance (processing time) could be improved by arbitrarily combining processes that were initially performed on the CPU and having those combinations executed on the GPU. As shown in Figure 20, it was found that by having some of the processes that were initially performed on the CPU executed on the GPU, performance improved by up to "30.8%" (processing speed was reduced by "30.8%") compared to before optimization. It is also shown that the GPU utilization changed from "28%" to "38%" compared to before optimization.

[0306] Furthermore, in the example shown in Figure 20, for the model corresponding to service SV2 (model "2"), an experiment was conducted to see how much performance (processing time) could be improved by arbitrarily combining processes that were initially performed on the CPU side and having those combinations executed on the GPU side. As shown in Figure 20, it was found that by having some of the processes that were initially performed on the CPU side executed on the GPU side, performance improved by up to "44.2%" (processing speed was reduced by "44.2%") compared to before optimization. It is also shown that the GPU utilization changed from "15%" to "42%" compared to before optimization.

[0307] Furthermore, in the example shown in Figure 20, for the model corresponding to service SV3 (model "3"), an experiment was conducted to see how much performance (processing time) could be improved by arbitrarily combining processes that were initially performed on the CPU and having those combinations executed on the GPU. As shown in Figure 20, it was found that by having some of the processes that were initially performed on the CPU executed on the GPU, performance improved by up to "12.3%" (processing speed was reduced by "12.3%") compared to before optimization. It is also shown that the GPU utilization changed from "15%" to "18%" compared to before optimization.

[0308] Furthermore, in the example shown in Figure 20, for the model corresponding to service SV4 (model "4"), an experiment was conducted to see how much performance (processing time) could be improved by arbitrarily combining processes that were initially performed on the CPU side and having those combinations executed on the GPU side. As shown in Figure 20, it was found that by having some of the processes that were initially performed on the CPU side executed on the GPU side, performance improved by up to "65.1%" (processing speed was reduced by "65.1%") compared to before optimization. It is also shown that the GPU utilization changed from "54%" to "56%" compared to before optimization.

[0309] Furthermore, as shown in Figure 20, for the model corresponding to service SV5 (model "5"), an experiment was conducted to see how much performance (processing time) could be improved by arbitrarily combining processes that were initially performed on the CPU side and having those combinations executed on the GPU side. As shown in Figure 20, it was found that by having some of the processes that were initially performed on the CPU side executed on the GPU side, performance improved by up to "39.1%" (processing speed was reduced by "39.1%") compared to before optimization. It is also shown that the GPU utilization changed from "39%" to "45%" compared to before optimization.

[0310] Furthermore, the experimental results described above show that even with different models used for different services, for multi-class classification models, performance can always be improved by offloading some of the processing that was initially done on the CPU to the GPU, resulting in an average performance improvement of 38.8%.

[0311] Furthermore, according to the experimental results shown in Figure 20, it is believed that the best optimization can be achieved by incorporating information indicating the computing device "GPU" into the architecture of the neural network corresponding to the multi-class classification model that was associated with the processing that was executed on the GPU when the maximum performance was obtained, thereby creating a rule-based architecture.

[0312] Next, we will focus on the experiment conducted for the model corresponding to service SV1 (model "1"), among the experiments conducted for each model corresponding to each service shown in Figure 20, and show an example of the experiment's content. Figure 21 is a diagram showing an example of the experiment's content conducted for the model corresponding to service SV1. Figure 21 shows the experiment's content when performance was improved by a maximum of "30.8%".

[0313] As shown in the example in Figure 21, an experiment was conducted in which processes A11, A12, and A13, which were initially performed on the CPU, were forcibly moved to the GPU so that they would be performed on the GPU.

[0314] Thus, in the model corresponding to service SV1, which is a multi-class classification model, if information indicating the computing device "GPU" is incorporated into the architecture associated with processes A11, A12, and A13, the execution control unit 200 can have a more powerful optimization algorithm. Consequently, as a result, it becomes possible to effectively improve the performance of user-side computers (e.g., servers and edge devices) used to operate the model corresponding to service SV1 in a production environment.

[0315] (Model for two-class classification) Next, using Figures 22 and 23, we will explain an example of the effect when processing using a two-class classification model is executed on both the CPU and the GPU. Here, for each two-class classification model for a given service, we experimented to see how much performance (processing time) could be improved by having specific processing that was initially performed on the GPU run on the CPU. Figure 22 shows the experimental results.

[0316] Figure 22 shows the improvement in performance achieved through experiments with a two-class classification model. For example, Figure 22 shows the elements that resulted in the best performance in the above experiment.

[0317] In the example shown in Figure 22, an experiment was conducted to see how much performance (processing time) could be improved by moving a specific process, originally performed on the GPU, to the CPU for the model corresponding to service SV6 (model "6"). As shown in Figure 22, it was found that moving the specific process, originally performed on the GPU, to the CPU resulted in a performance improvement of up to 50.3% (a reduction in processing speed of 50.3%) compared to before optimization.

[0318] Furthermore, in the example shown in Figure 22, an experiment was conducted to see how much performance (processing time) could be improved by having a specific process, originally performed on the GPU, run on the CPU for the model corresponding to service SV7 (model "7"). As shown in Figure 22, it was found that by having a specific process, originally performed on the GPU, run on the CPU, performance improved by up to 30.2% (processing speed was reduced by 30.2%) compared to before optimization.

[0319] Furthermore, the experimental results described above showed that, even with different models used for different services, performance for two-class classification models could always be improved by shifting specific processing tasks, initially performed on the GPU, to the CPU. It was also found that parallel computing on the CPU is effective for many processes using two-class classification models.

[0320] Furthermore, according to the experimental results shown in Figure 22, it is believed that the best optimization can be achieved by making the neural network architecture corresponding to the model for two-class classification rule-based by incorporating information indicating the processing unit "CPU" into the architecture associated with the processing that was executed by the CPU when the best performance was obtained.

[0321] Next, we will focus on the experiment conducted for the model corresponding to service SV6 (model "6"), among the experiments conducted for each model corresponding to each service shown in Figure 22, and show an example of the experiment's contents. Figure 23 is a diagram showing an example of the experiment's contents for the model corresponding to service SV6. Figure 23 shows the experiment's contents when performance was improved by a maximum of "50.3%".

[0322] As shown in the example in Figure 23, an experiment was conducted in which a process requiring MATMUL calculations, which was initially performed on the GPU, was moved to the CPU.

[0323] Thus, in the model corresponding to Service SV6, which is a two-class classification model, if information indicating the processing unit "CPU" is incorporated for the architecture associated with the processing that requires MATMUL calculations, the execution control unit 200 can have a more powerful optimization algorithm. Consequently, as a result, it becomes possible to effectively improve the performance of user-side computers (e.g., servers and edge devices) used to operate the model corresponding to Service SV6 in a production environment.

[0324] Furthermore, regardless of the model being compatible with Service SV6, if the architecture of the two-class classification model is rule-based by incorporating information indicating the processing unit "CPU" for the architecture associated with the processing that requires MATMUL operations, it can be said that the performance of the user's computer (e.g., server or edge device) can be effectively improved.

[0325] [10. Processing Flow of Information Processing Devices] We have so far described the algorithms for the optimization processes performed by the information processing device 100 and the execution control device 200, respectively. Next, we will describe the procedure for the processes performed by the information processing device 100. Specifically, we will describe the procedure for the information processing device 100 to perform a series of tuning processes (fine-tuning according to the embodiment) including the first to fifth optimization processes.

[0326] Figure 24 is a flowchart showing an example of the fine-tuning process according to the embodiment. In Figure 24, the part of the fine-tuning according to the embodiment that is executed by the optimizer function (optimizer OP) of the information processing device 100 is shown.

[0327] First, the generation unit 131 performs steps S2401 and S2402 using an algorithm (first optimization algorithm) that optimizes the random number seed used to generate the model (computation graph).

[0328] Specifically, the generation unit 131 generates multiple random number seeds for the computation graph (step S2401). For example, the generation unit 131 generates multiple random number seeds optimized so that the initial values ​​of the weights exhibit a uniform distribution. The generation unit 131 also generates initial values ​​for the weights corresponding to each of the generated random number seeds (step S2402). For example, the generation unit 131 inputs the generated random number seeds into a random function, and from the multiple pseudorandom numbers obtained as output, which are uniformly distributed, it generates weights corresponding to each of those pseudorandom numbers. Furthermore, the initial values ​​of the weights obtained in this way also exhibit a uniform distribution.

[0329] Then, the generation unit 131 generates multiple models corresponding to each of the initial values ​​generated in step S2402 (step S2403). In the example in Figure 24, weights are shown as an example of model parameters, but the model parameters may be, for example, weights or biases. In such cases, the generation unit 131 may generate a model having each set of different combinations of model parameters (for example, a set of weights and biases) from the group of initial values ​​of model parameters generated in step S2402.

[0330] Next, the first data control unit 133 performs the following steps S2404 to S2406 using an algorithm (second optimization algorithm) that optimizes the training data used for training the model.

[0331] Specifically, the first data control unit 133 divides the training data set, which has been sorted so that the included training data is in chronological order, into a predetermined number of sets (step S2404). Then, the first data control unit 133 selects from the sets obtained by the division in step S2404 to be used for training each model generated in step S2403 (step S2405). For example, the first data control unit 133 randomly selects sets to be used for training the models from all the sets obtained by the division in step S2404 until the number of selected sets reaches a predetermined number. For example, the first data control unit 133 randomly selects sets from the sets obtained by the division in step S2404 that have not yet been selected until a specified number of loops have been reached. Furthermore, the first data control unit 133 may randomly select sets from the sets obtained by the division in step S2404 that have not yet been selected until the specified number of Loops is reached, starting with sets containing the most recent time series of training data, until a predetermined number (for example, a number specified by the user) is reached.

[0332] Then, the first data control unit 133 generates a single training data set by concatenating the sets of training data selected in step S2405 (step S2406). For example, the first data control unit 133 generates a single training data set by concatenating the sets selected in step S2405 in the order they were selected at that time.

[0333] Next, the second data control unit 134 performs the following steps S2407 and S2408 using an algorithm (third optimization algorithm) to optimize the shuffle buffer size.

[0334] Specifically, in step S2406, the second data control unit 134 divides the training data set generated by the first data control unit 133 (step S2407). For example, the second data control unit 134 divides the training data set generated by the first data control unit 133 as a process to generate training data of a size equal to the size of the shuffle buffer. For example, the second data control unit 134 can divide the training data set generated by the first data control unit 133 into a predetermined number of sets such that each set after division contains a predetermined number of training data (for example, a number specified by the user).

[0335] Then, the second data control unit 134 extracts one set from the sets obtained by the division in step S2407 according to the order in which the divisions were obtained (division order), and stores the training data included in the extracted set in the shuffle buffer as the training data to be learned (step S2408). For example, the second data control unit 134 extracts one set from the sets obtained by the division in step S2407 that are currently unprocessed and not being used for learning, according to the division order. Then, the second data control unit 134 stores the extracted set in the shuffle buffer as the training data to be learned, which is the training data to be used in this iterative learning.

[0336] Next, the first learning unit 135 performs the following steps S2409 to S2411 using an algorithm (fourth optimization algorithm) that optimizes the random number seed (random number seed for data shuffling) used to determine the learning order when sequentially training the training data in the shuffle buffer.

[0337] Specifically, the first learning unit 135 generates a random seed for the random order of the training data in the shuffle buffer (step S2409). For example, for each epoch for iterative learning, the first learning unit 135 generates a random seed for the current learning (the seed that forms the basis of the random order) so that there is no bias in the random order associated with each training data between epochs.

[0338] Furthermore, the first learning unit 135 generates a random order corresponding to each of the random number seeds generated in step S2409 (step S2410). For example, the first learning unit 135 generates a random order by inputting each random number seed into a random function. Then, the first learning unit 135 generates the final training data to be trained in the shuffle buffer by associating this generated random order with the training data in the shuffle buffer (step S2411).

[0339] Furthermore, the first learning unit 135 trains each model on the features of the final training data in the learning order indicated by the random order determined in step S2410 (step S2412). During this learning process, trials to search for hyperparameters are repeated, but in order to achieve efficient search, the first learning unit 135 performs a fifth optimization as an optimization of trials by pruning, and terminates trials that are not expected to yield good results early without continuing to the end.

[0340] Furthermore, the first learning unit 135 repeatedly performs learning for a specified number of epochs, using the sets obtained by the division in step S2407, with steps S2408 to S2412 considered as one epoch. Specifically, the first learning unit 135 repeatedly performs learning for a user-specified number of epochs, using the sets obtained by the division in step S2407, with steps S2408 to S2412 considered as one epoch.

[0341] Therefore, the first learning unit 135 then determines whether all of the sets obtained by the third optimization (specifically, the sets obtained by the partition in step S2407) have been processed for one epoch (step S2413). Specifically, the first learning unit 135 determines whether all of the sets obtained by the partition in step S2407 have been used in the learning process from step S2408 to S2412, which constitutes one epoch. As long as the first learning unit 135 determines that not all of the sets obtained by the partition in step S2407 have been processed for one epoch (step S2413; No), it repeats the series of processes from step S2408 to step S2412 until it determines that all of the sets have been processed for one epoch.

[0342] On the other hand, if the first learning unit 135 determines that all of the sets obtained by the partition in step S2407 have been processed for one epoch (step S2413; Yes), it then determines whether the specified number of epochs has been reached for the sets obtained by the partition in step S2407 (step S2414). Specifically, the first learning unit 135 determines whether repeated learning has been performed for the specified number of epochs using the sets obtained by the partition in step S2407.

[0343] The first learning unit 135, while it determines that the specified number of epochs has not been reached (step S2414; No), repeats the series of processes from step S2408 onward until it determines that the specified number of epochs has been reached.

[0344] On the other hand, if the model selection unit 136 determines that the specified number of epochs has been reached (step S2414; Yes), it selects the best model at the moment based on the accuracy of each model that has been trained at that time (step S2415). Here, as explained in Figure 11, the series of processes from step S2408 onwards are repeated until the specified number of loops is reached in order to obtain a model with higher accuracy.

[0345] Therefore, the first learning unit 135 then determines whether the specified number of times to repeat (loop) the series of processes from step S2408 onwards has been reached (step S2416). As long as the first learning unit 135 determines that the specified number of loops has not been reached (step S2416; No), it repeats the series of processes from step S2408 onwards. On the other hand, if the first learning unit 135 determines that the specified number of loops has been reached (step S2416; Yes), it terminates the process at this point.

[0346] Furthermore, at this point when processing is complete, the best model selected by the model selection unit 136 is likely to be the most accurate among the models selected for each loop.

[0347] Furthermore, the second learning unit 137 corresponds to the selector function (selector SE) of the information processing device 100 among the fine tuning according to the embodiment, and although not shown in Figure 24, it subsequently performs the tuning process described in steps S21 to S24 of Figure 3, for example. Specifically, the second learning unit 137 performs the tuning process on the best model selected by the model selection unit 136.

[0348] [11. An example of experimental results regarding fine tuning] Next, using Figures 25A to 25C, we will explain an example of the effects when fine tuning according to the embodiment is performed.

[0349] Figure 25A shows Comparative Example (1), which compares the accuracy of the model when fine tuning according to the embodiment is performed and when fine tuning according to the embodiment is not performed. Specifically, Figure 25A shows a comparative example comparing the evaluation results corresponding to Trial A when fine tuning is performed and the evaluation results corresponding to Trial A when fine tuning is not performed.

[0350] Corresponding with the example in Figure 4, in the example in Figure 25A, the data from "June 16, 17:32" to "June 17, 7:26" from the dataset was used as evaluation data, and the accuracy of the best model was evaluated. Also in the example in Figure 25A, the data from "June 17, 7:26" to "June 19, 0:00" from the dataset was used as test data with unknown labels, and the accuracy of the best model was evaluated. According to the example in Figure 25A, the evaluation results showed that the accuracy of the best model improved by "4.5%" by performing fine tuning according to the embodiment.

[0351] Figure 25B is a comparative example (2) showing a comparison of the model's accuracy when fine tuning according to the embodiment is performed and when fine tuning according to the embodiment is not performed. Specifically, Figure 25B shows a comparative example comparing the evaluation results corresponding to Trial B when fine tuning is performed with the evaluation results corresponding to Trial B when fine tuning is not performed.

[0352] Corresponding with the example in Figure 4, in the example in Figure 25B, the data from "June 17, 7:26 AM" to "June 17, 12:00 PM" in the dataset was used as evaluation data, and the accuracy of the best model was evaluated. Also in the example in Figure 25B, the data from "June 17, 12:00 PM" to "June 19, 12:00 AM" in the dataset was used as test data with unknown labels, and the accuracy of the best model was evaluated. According to the example in Figure 25B, the evaluation results showed that by performing fine tuning according to the embodiment, the accuracy of the best model improved by "9.0%".

[0353] Figure 25C is a comparative example (3) showing a comparison of the model's accuracy when fine tuning according to the embodiment is performed and when fine tuning according to the embodiment is not performed. Specifically, Figure 25C shows a comparative example comparing the evaluation results corresponding to Trial C when fine tuning is performed with the evaluation results corresponding to Trial C when fine tuning is not performed.

[0354] Corresponding to the example in Figure 4, in the example in Figure 25C, data from "June 17, 12:00" to "June 19, 0:00" was used as evaluation data, and the accuracy of the best model was evaluated. According to the example in Figure 25C, the evaluation results showed that by performing fine tuning according to the embodiment, the accuracy of the best model improved by "10.2%".

[0355] Furthermore, as shown in the examples in Figures 25A to 25C, the effects of fine-tuning were verified from multiple perspectives by appropriately changing the time ranges used to define the training data, the evaluation data, and the evaluation data with unknown labels within a time-series dataset.

[0356] Furthermore, the evaluation results shown in Figures 25A to 25B demonstrate that, regardless of how the dataset is used according to its application, performing the fine-tuning according to the embodiment maintains improved performance compared to not performing the fine-tuning according to the embodiment. This also demonstrates that the information processing device 100 according to the embodiment can improve the accuracy of the model.

[0357] [12. Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0358] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0359] Furthermore, the embodiments described above can be combined as appropriate, provided that the processing content is not contradictory.

[0360] [13. Program] Furthermore, the information processing device 100 and execution control device 200 according to the above embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 26. Figure 26 is a hardware configuration diagram showing an example of the computer 1000. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0361] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0362] The HDD 1400 stores programs executed by the CPU 1100, as well as data used by such programs. The communication interface 1500 receives data from other devices via the communication network 50 and sends it to the CPU 1100, and transmits the data generated by the CPU 1100 to other devices via the communication network 50.

[0363] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0364] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0365] For example, when computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto RAM 1200. Furthermore, data from the storage unit 120 is stored in HDD 1400.

[0366] Furthermore, for example, if computer 1000 functions as the execution control device 200 according to the embodiment, the CPU 1100 of computer 1000 realizes the functions of the control unit 230 by executing a program loaded on RAM 1200. Also, data in the storage unit 220 is stored in HDD 1400.

[0367] The CPU 1100 of computer 1000 reads and executes these programs from the recording medium 1800, but as an alternative example, these programs may be obtained from other devices via the communication network 50.

[0368] [14. Effects] (Effects of one embodiment of the information processing device 100 according to the embodiment (1)) As described above, the information processing device 100 according to the embodiment (an example of a learning device) includes a generation unit 131, a first learning unit 135, a model selection unit 136, and a second learning unit 137. The generation unit 131 generates a plurality of models, each with different parameters. The first learning unit 135 causes each of the plurality of models generated by the generation unit 131 to learn the features that a predetermined portion of the learning data possesses. The model selection unit 136 selects one of the models according to the accuracy of the model learned by the first learning unit 135. The second learning unit 137 causes the model selected by the model selection unit 136 to learn the features that a predetermined portion of the learning data possesses.

[0369] Such an information processing device 100 can provide users with models that have improved accuracy and better performance, thereby effectively supporting users in putting the models into practical use for specific services.

[0370] Furthermore, the generation unit 131 generates multiple input values ​​to be input to a predetermined first function that calculates random values ​​based on the input values, and for each of the generated input values, it generates multiple models having parameters corresponding to the random values ​​output by the predetermined first function when that input value is input.

[0371] Such an information processing device 100 can improve the accuracy of the model.

[0372] Furthermore, the generation unit 131 generates multiple input values ​​to be input to a predetermined first function such that the random value output by the predetermined first function satisfies predetermined conditions.

[0373] This information processing device 100 allows for control over the variation in the initial values ​​of the model parameters, thereby improving the accuracy of the model.

[0374] Furthermore, the generation unit 131 generates multiple input values ​​such that the random values ​​fall within a predetermined range.

[0375] With such an information processing device 100, it is possible to control the variation in the initial values ​​of the model parameters so that they exhibit a uniform distribution, thereby improving the accuracy of the model.

[0376] Furthermore, the generation unit 131 generates multiple input values ​​such that the distribution of random values ​​exhibits a predetermined probability distribution.

[0377] With such an information processing device 100, it is possible to control the variation in the initial values ​​of the model parameters so that they exhibit a uniform distribution, thereby improving the accuracy of the model.

[0378] Furthermore, the generation unit 131 generates multiple input values ​​such that the average value of the randomized values ​​is a predetermined value.

[0379] With such an information processing device 100, it is possible to control the variation in the initial values ​​of the model parameters so that they exhibit a uniform distribution, thereby improving the accuracy of the model.

[0380] Furthermore, the generation unit 131 selects a function as a predetermined first function such that the distribution of random values ​​output when an input value is input exhibits a predetermined probability distribution, and generates multiple models having parameters corresponding to the random values ​​output by the selected function.

[0381] With such an information processing device 100, it is possible to control the variation in the initial values ​​of the model parameters so that they exhibit a uniform distribution, thereby improving the accuracy of the model.

[0382] Furthermore, the first learning unit 135 (an example of a selection unit) selects multiple models from among the trained models whose evaluation values ​​for evaluating accuracy satisfy predetermined conditions, and trains the selected multiple models to learn the features that a portion of the predetermined training data possesses.

[0383] With this information processing device 100, among the trials for exploring hyperparameters, trials that satisfy a termination condition defined using the evaluation value of the model can be terminated early, while trials that do not satisfy this termination condition (multiple models whose evaluation values ​​for evaluating accuracy satisfy a predetermined condition) can be continued. This solves problems related to time and computer resource usage, and also improves the accuracy of the model by pruning trials that are not expected to yield good results early.

[0384] Furthermore, the first learning unit 135 selects multiple models whose characteristics, based on the change in evaluation values ​​during repeated learning of a predetermined number of times a portion of predetermined learning data, satisfy a predetermined characteristic.

[0385] With this information processing device 100, as the model is repeatedly trained by applying each trial with different hyperparameter combinations, trials that satisfy the termination condition are terminated early, while trials that do not satisfy the termination condition (multiple models whose evaluation values ​​for evaluating accuracy satisfy predetermined conditions) are allowed to continue. This solves problems related to time and computer resource usage, and also improves the accuracy of the model by pruning trials that are not expected to produce good results early.

[0386] Furthermore, the first learning unit 135 selects a model that satisfies multiple conditions specified by the user, as a predetermined condition.

[0387] According to this information processing device 100, by combining multiple stopping conditions that condition the early termination of trials where no improvement in model performance can be expected, and which are defined using the evaluation value of the model, it is possible to improve the accuracy of the model compared to when a general early stopping algorithm is used.

[0388] Furthermore, the first learning unit 135 may generate multiple input values ​​to be input to a predetermined second function that calculates random values ​​based on input values, and for each of the generated input values, generate a portion of predetermined training data based on the random values ​​output by the predetermined second function when that input value is input. For this reason, the first learning unit 135 may also be an example of a training data generation unit.

[0389] Furthermore, this information processing device 100 can solve the problem of the model not learning properly due to bias in the order in which training data is distributed to the model, thereby improving the accuracy of the model.

[0390] Furthermore, the first learning unit 135 generates multiple input values ​​to be input to a predetermined second function for each repeated learning session, thereby generating learning data to be learned in that session, and uses this learning data generated for each repeated learning session to train the model.

[0391] With such an information processing device 100, for each epoch of repeated learning, it is possible to determine the learning order for the current epoch so that there is no bias in the learning order associated with each learning data between epochs.

[0392] Furthermore, the first learning unit 135 generates learning data in which random values ​​are associated with a learning order, as part of predetermined learning data.

[0393] With such an information processing device 100, for example, an optimized learning order can be associated with each piece of training data in the shuffle buffer, thus solving the problem of the model not learning properly due to a bias in the learning order in which the training data is used to train the model.

[0394] Furthermore, the model selection unit 136 selects one of the models according to the accuracy of the model trained by the first learning unit 135 for each combination of models with different parameters and predetermined training data.

[0395] With such an information processing device 100, it becomes possible to select the model with the best performance from among models with different parameters and provide it to the user.

[0396] (Effects of one embodiment of the information processing device 100 according to the embodiment (2)) As described above, the information processing device 100 (an example of a learning device) according to the embodiment has a second data control unit 134. The second data control unit 134 divides predetermined learning data for which a model learns features into a plurality of sets in chronological order, and controls the first learning unit 135 to learn the features of the learning data included in each divided set in a predetermined order. Thus, the second data control unit 134 is a processing unit corresponding to an example of a division unit and a learning unit.

[0397] Furthermore, with such an information processing device 100, the accuracy of the model can be improved because the shuffle buffer size can be optimized based on the fact that the accuracy of the model changes depending on the shuffle buffer size, and the training data can be divided to match the optimized shuffle buffer size.

[0398] Furthermore, the second data control unit 134 controls the model so that the features of the training data included in each divided set are learned in a random order.

[0399] Such an information processing device 100 can improve the accuracy of the model.

[0400] Furthermore, the second data control unit 134 controls the model so that the features of the training data included in each of the divided sets are learned sequentially, starting with the sets corresponding to the time series.

[0401] With this information processing device 100, learning is performed sequentially from older time-series training data to newer time-series training data, which allows for the calculation of the trends in the features of the training data with high accuracy, thereby improving the accuracy of the model.

[0402] Furthermore, the second data control unit 134 divides the predetermined training data into sets containing a number of training data specified by the user.

[0403] With such an information processing device 100, users who have verified how the accuracy of the model changes depending on the shuffle buffer size can then divide the training data based on the results obtained from this verification, thereby improving the usability of the shuffle buffer size optimization.

[0404] Furthermore, the second data control unit 134 divides the predetermined learning data into multiple sets such that the number of learning data points included in each set into which the predetermined learning data is divided falls within the range specified by the user.

[0405] With such an information processing device 100, users can, for example, specify a range they have estimated if it is difficult to specify an appropriate number, thereby improving usability in optimizing the shuffle buffer size.

[0406] (Effects of one embodiment of the information processing device 100 according to the embodiment (3)) As described above, the information processing device 100 (an example of a learning device) according to the embodiment has a first data control unit 133. The first data control unit 133 divides predetermined learning data for learning features in a model into a plurality of sets in chronological order, and selects a set from the divided sets to be used for learning the model. The first data control unit 133 also uses the selected sets in order from the oldest set of learning data to the first set, and controls the first learning unit 135 to learn the features of the learning data contained in each set. Thus, the first data control unit 133 is a processing unit corresponding to an example of a division unit, a selection unit, and a learning unit.

[0407] Furthermore, with such an information processing device 100, it is possible to optimize the training data used for actual training from the dataset, thereby improving the accuracy of the model.

[0408] Furthermore, the first data control unit 133 divides a predetermined training data into sets having a predetermined number of training data.

[0409] With this information processing device 100, it is possible to divide the dataset so that each set obtained by the division contains a predetermined number of training data, thereby optimizing each set that contains the training data actually used for training.

[0410] Furthermore, the first data control unit 133 randomly selects from the divided sets which sets will be used for training the model.

[0411] With such an information processing device 100, it is possible to fairly select which of the sets obtained by partitioning will contain the training data to be used for actual learning.

[0412] Furthermore, the first data control unit 133 selects from the divided sets the set in which the time series of the training data it contains is more recent.

[0413] With such an information processing device 100, it is possible to control the learning process so that features from more recent training data are learned, thereby improving the accuracy of the model.

[0414] Furthermore, the first data control unit 133 selects a number of sets specified by the user from among the divided sets.

[0415] Such an information processing device 100 can improve usability when dividing a dataset.

[0416] Furthermore, the first data control unit 133 selects sets of the divided sets in chronological order, prioritizing those with more recent time series of the included training data, until the number of selected sets reaches the number specified by the user.

[0417] With such an information processing device 100, the features of the training data specified by the user can be learned in order to improve the accuracy of the model to the greatest extent possible.

[0418] (Effects of one embodiment of the information processing device 100 according to the embodiment (4)) As described above, the information processing device 100 according to the embodiment (an example of a classification device) includes a first learning unit 135 (which may be a second learning unit 137), an attribute selection unit 139, and a provision unit 138. The first learning unit 135 trains a model on the characteristics of learning data having multiple attributes. The attribute selection unit 139 selects target attributes from among the input candidate data to be input to the model learned by the first learning unit 135, which data have attributes that will not be input to the model. The provision unit 138 provides the model with information indicating attributes other than the target attributes selected by the attribute selection unit 139.

[0419] With this information processing device 100, users can learn that when they want to use a pre-trained model, they don't need to input all of the test data they have prepared. Instead, they only need to mask data with specific attributes and input the remaining data. As a result, users can obtain more accurate output results than when using all of the test data. Furthermore, the information processing device 100 can help users obtain more accurate results when using a pre-trained model.

[0420] Furthermore, the attribute selection unit 139 selects a combination of target attributes.

[0421] With this information processing device 100, it becomes possible to measure the accuracy of the model for all possible combinations of target attributes and compare the accuracy of the model across different combinations. Therefore, it becomes possible to determine with high accuracy which combinations of training data should not be input into the model in order to obtain the highest possible accuracy.

[0422] Furthermore, the attribute selection unit 139 measures the accuracy of the model when training data containing attributes other than the target attribute in each candidate combination of target attributes is input to the model, and selects a combination of target attributes from among the candidates according to the measurement result.

[0423] With such an information processing device 100, it becomes possible to compare the accuracy of the model among possible combinations of target attributes, making it possible to determine with high accuracy which combinations of training data should not be input into the model in order to obtain the highest accuracy.

[0424] Furthermore, the first learning unit 135 determines multiple new combinations of target attributes based on the combinations of target attributes in multiple models whose accuracy satisfies predetermined conditions, and determines whether the accuracy of each model satisfies predetermined conditions when training data having attributes other than the target attributes in the determined combinations is input to the multiple models. Then, the first learning unit 135 trains the models that it has determined to satisfy the predetermined conditions with the training data.

[0425] According to this information processing device 100, multiple models whose evaluation values ​​for evaluating accuracy satisfy predetermined conditions can be selected, and when training the selected multiple models to learn features from a portion of the training data, it is possible to control the training data so that training data that may degrade the performance of the models is not learned, thereby improving the accuracy of the models.

[0426] Furthermore, the provisioning unit 138 provides information indicating attributes other than the target attribute selected by the attribute selection unit 139, specifically information regarding the accuracy of the model when training data having attributes other than the target attribute selected by the attribute selection unit 139 is input to the model.

[0427] Such an information processing device 100 can help users obtain more accurate results using a pre-trained model.

[0428] (Effects of one embodiment of the information processing device 100 according to the embodiment (5)) As described above, the execution control device 200 according to the embodiment includes a specification unit 231, a decision unit 232, and an execution control unit 233. The specification unit 231 identifies the characteristics of a model used when multiple arithmetic units, each with a different architecture, execute a predetermined process. Based on the characteristics of the model identified by the specification unit 231, the decision unit 232 determines which of the multiple arithmetic units will execute the process using the model. The execution control unit 233 causes the arithmetic unit determined by the decision unit 232 to execute the process using the model.

[0429] With such an information processing device 100, the computing device to be executed can be optimized based on the characteristics of the model so that each process using the model is executed by an appropriate computing device. Furthermore, with such an information processing device 100, the processing time spent on processing using the model can be further reduced. In addition, with such an information processing device 100, the accuracy of the model can be indirectly improved from the perspective of the computer on which the user intends to perform the processing using the model.

[0430] Furthermore, the identification unit 231 identifies the characteristics of multiple processes to be executed as a model, and the determination unit 232, based on the characteristics of the multiple processes identified by the identification unit 231, determines which of the multiple processing units will execute each process.

[0431] With such an information processing device 100, each of the multiple processes executed as a model can be assigned to a computing unit that is better suited to that particular process, thereby reducing the processing time spent on processing using the model.

[0432] Furthermore, the determination unit 232 determines the arithmetic unit to be executed from among a plurality of arithmetic units, specifically a first arithmetic unit that is guaranteed to output the same value when the same processing is performed using the same data, and a second arithmetic unit that is not guaranteed to output the same value when the same processing is performed using the same data.

[0433] Such an information processing device 100 can improve the accuracy of the model.

[0434] Furthermore, the determination unit 232 determines which arithmetic unit to execute from among a plurality of arithmetic units, namely a first arithmetic unit that performs scalar operations and a second arithmetic unit that performs vector operations.

[0435] With this type of information processing device 100, among the multiple processes executed as a model, the processes requiring scalar operations can be executed by the first arithmetic unit, and the processes requiring vector operations can be executed by the second arithmetic unit, thereby further reducing the processing time spent on processing using the model.

[0436] Furthermore, the determination unit 232 determines which arithmetic unit to execute from among a plurality of arithmetic units, namely a first arithmetic unit employing an out-of-order method and a second arithmetic unit that does not employ an out-of-order method.

[0437] Such an information processing device 100 can improve the accuracy of the model.

[0438] The determination unit 232 determines the computing device to be executed from either a central computing device having a branch prediction function as the first computing device, or an image computing device without a branch prediction function as the second computing device.

[0439] With such an information processing device 100, among the multiple processes executed as a model, the CPU can be assigned to processes that the CPU excels at, and the GPU can be assigned to processes that the GPU excels at, thereby further reducing the processing time spent on processing using the model.

[0440] Furthermore, if the model is a multi-class classification model, the decision unit 232 determines the image processing unit to be used as the processing unit.

[0441] Such an information processing device 100 can further reduce the processing time spent on model-based processing.

[0442] Furthermore, if the model is a two-class classification model, the decision unit 232 determines the central processing unit to be the processing unit to be executed.

[0443] Such an information processing device 100 can further reduce the processing time spent on model-based processing.

[0444] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0445] Furthermore, the terms "section, module, unit" used above can be replaced with "means" or "circuit," etc. For example, a generation unit can be replaced with a generation means or a generation circuit. [Explanation of symbols]

[0446] 1. Information Provision System 2 Model generation servers 3 Terminal devices 10 Information provision device Sy Information Processing System 100 Information Processing Devices 120 Storage section 121 Learning Data Storage Unit 122 Model Memory Unit 130 Control Unit 131 Generation part 132 Acquisition Department 133 First Data Control Unit 134 Second Data Control Unit 135 First Learning Department 136 Model Selection Section 137 Second Learning Department 138 Provision Department 139 Attribute Selection Section 200 Execution Control Unit 220 Storage section 221 Model Architecture Memory Unit 230 Control Unit 231 Specific section 232 Decision Section 233 Execution Control Unit

Claims

1. A splitting unit divides predetermined training data, which is used to train the model on features, into multiple sets in chronological order. A learning unit randomly selects multiple sets from the sets obtained by the division by the division unit, and inputs the learning data contained in the selected sets into the model one by one in a predetermined order, thereby training the model with the features of the learning data in the predetermined order. A learning device characterized by having the following features.

2. The learning unit randomly selects a number of sets specified by the user from the sets obtained by the division by the division unit, and trains the model on the features of the training data included in the selected sets in the time-series order. The learning device according to feature 1.

3. The learning unit divides the training data included in the selected set into multiple sets in chronological order, selects each of the divided sets in chronological order, and trains the model on the features of the training data included in the selected set in a random order. The learning device according to feature 2.

4. The division unit divides the predetermined training data into sets having a number of training data specified by the user. A learning device according to any one of claims 1 to 3.

5. The division unit divides the predetermined learning data into multiple sets such that the number of learning data points included in each set into which the predetermined learning data is divided falls within a range specified by the user. A learning device according to any one of features 1 to 4.

6. A learning method performed by a learning device, A splitting process in which predetermined training data, used to train the model on features, is divided into multiple sets in chronological order, A learning process in which multiple sets are randomly selected from the sets obtained by the division process, and the learning data contained in the selected sets are input into the model one by one in a predetermined order, thereby allowing the model to learn the features of the learning data in the predetermined order. A learning method characterized by including the following.

7. A splitting procedure for dividing predetermined training data, which is used to train a model on its features, into multiple sets in chronological order, A learning procedure in which multiple sets are randomly selected from the sets obtained by the division procedure described above, and the learning data contained in the selected sets are input into the model one by one in a predetermined order, thereby training the model to learn the features of the learning data in the predetermined order, and A learning program to get a computer to execute something.

Images