Execution control device, execution control method, and execution control program
The execution control device optimizes model performance by dynamically selecting appropriate arithmetic units and adjusting model parameters, addressing the limitations of conventional methods in improving model accuracy through hyperparameter adjustments.
Patent Information
- Application Number
- JP2021147733
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-09-11
- Filing Date
- 2021-09-10
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2041-09-10
AI Technical Summary
Conventional techniques for improving model accuracy by dynamically changing learning data based on hyperparameter values are inadequate, often leading to suboptimal performance due to inappropriate hyperparameter settings.
An execution control device and method that includes a specifying unit to identify model characteristics, a determining unit to select appropriate arithmetic units, and an execution control unit to execute processing, optimizing the model's architecture and hyperparameters based on the characteristics of the learning data.
Enhances model accuracy by dynamically selecting the most suitable arithmetic units and optimizing model parameters, leading to improved performance and accuracy.
Smart Images

Figure 0007587485000001 
Figure 0007587485000002 
Figure 0007587485000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an execution control device, an execution control method, and an execution control program. [Background technology]
[0002] In recent years, techniques have been proposed for making various models, such as SVM (Support Vector Machine) and DNN (Deep Neural Network), perform various predictions and classifications by learning the features of training data. As an example of such a learning method, a technique has been proposed that dynamically changes the learning mode of training data according to the values of hyperparameters, etc. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-164793 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, the above-mentioned conventional techniques do not always improve the accuracy of the model.
[0005] For example, in the above-mentioned conventional technology, the learning data to be used for learning features is merely dynamically changed according to the values of hyperparameters, etc. Therefore, if the values of the hyperparameters are not appropriate, it may not be possible to improve the accuracy of the model.
[0006] The present application has been made in consideration of the above, and has an object to provide an execution control device, an execution control method, and an execution control program that can improve the accuracy of a model. [Means for solving the problem]
[0007] The execution control device of the present application is characterized by having an identification unit that identifies characteristics of a model used by multiple arithmetic devices, each with a different architecture, when executing a specified process, a determination unit that determines which of the multiple arithmetic devices is to execute the process using the model based on the characteristics of the model identified by the identification unit, and an execution control unit that causes the arithmetic device determined by the determination unit to execute the process using the model. Effect of the Invention
[0008] According to one aspect of the embodiment, it is possible to improve the accuracy of the model. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a process executed by an information providing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of an information processing system according to the embodiment. [Diagram 3] FIG. 3 is a diagram showing an overall view of the processing executed by the information processing device according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of division into trials when a data set is divided according to the purpose. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 6] FIG. 6 is an explanatory diagram conceptually explaining division of a data set. [Figure 7] FIG. 7 is a graph (1) showing the change in model performance when the first and fourth optimization algorithms are executed. [Figure 8] FIG. 8 is a graph (2) showing the change in model performance when the first and fourth optimization algorithms are executed. [Figure 9] FIG. 9 is a diagram showing a comparative example comparing the performance of models according to combinations of the first and fourth optimization algorithms. [Figure 10]FIG. 10 is a diagram illustrating an example of the second optimization algorithm. [Figure 11] FIG. 11 is a diagram illustrating an example of the third optimization algorithm. [Figure 12] FIG. 12 is a diagram showing a comparative example in which the performance of the models is compared for each shuffle buffer size. [Figure 13] FIG. 13 is a diagram showing an example of condition information related to the fifth optimization algorithm. [Figure 14] FIG. 14 is a diagram illustrating an example of the fifth optimization algorithm. [Figure 15] FIG. 15 is a diagram illustrating an example of an optimization algorithm for optimizing the mask target. [Figure 16] FIG. 16 is a diagram showing a comparative example in which the accuracy of a model is compared between a case in which optimization of a mask target is performed and a case in which optimization of a mask target is not performed. [Figure 17] FIG. 17 is a diagram illustrating an example of the configuration of an execution control device according to the embodiment. [Figure 18] FIG. 18 illustrates an example of a model architecture storage unit according to the embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of a model architecture associated with information indicating a target arithmetic unit for execution. [Figure 20] FIG. 20 is a diagram showing the performance improvement status in an experiment targeting a model for multi-class classification. [Figure 21] FIG. 21 is a diagram showing an example of the experiment content of an experiment conducted on a model corresponding to a service SV1. [Figure 22] FIG. 22 is a diagram showing the performance improvement status in an experiment targeting a two-class classification model. [Figure 23] FIG. 23 is a diagram showing an example of the experiment content of an experiment conducted on a model corresponding to service SV6. [Figure 24] FIG. 24 is a flowchart showing an example of the flow of fine tuning according to the embodiment. [Figure 25A]FIG. 25A is a diagram showing a comparative example (1) in which the accuracy of a model is compared between a case in which fine tuning according to the embodiment is performed and a case in which fine tuning according to the embodiment is not performed. [Figure 25B] FIG. 25B is a diagram showing a comparative example (2) in which the accuracy of the model is compared between a case where the fine tuning according to the embodiment is performed and a case where the fine tuning according to the embodiment is not performed. [Figure 25C] FIG. 25C is a diagram showing a comparative example (3) in which the accuracy of the model is compared between a case where the fine tuning according to the embodiment is performed and a case where the fine tuning according to the embodiment is not performed. [Figure 26] FIG. 26 is a hardware configuration diagram illustrating an example of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Below, a detailed description will be given of modes (hereinafter referred to as "embodiments") for implementing the device, method, and program according to the present application (specifically, a learning device, a learning method, a learning program / classification device, a classification method, a classification program / execution control device, an execution control method, and an execution control program) with reference to the drawings. Note that the learning device, learning method, and learning program according to the present application are not limited to these embodiments. Furthermore, the embodiments can be appropriately combined to the extent that no contradiction occurs in the processing contents. Furthermore, the same parts in the following embodiments are given the same symbols, and duplicated explanations will be omitted.
[0011] 1. About the embodiment In the following embodiment, the description will mainly focus on the information processing executed by the information processing device 100, which is an example of a learning device and a classification device, and the information processing executed by the execution control device 200. Meanwhile, the processing executed by the information providing device 10 included in a system having the information processing device 100 and the execution control device 200 will first be described as a premise for the information processing according to the embodiment.
[0012] 2. Configuration of the information provision system Fig. 1 is a diagram showing an example of a process executed by an information providing device 10 according to an embodiment. In the example of Fig. 1, the information processing device 100 and the execution control device 200 are not shown, but an information providing system 1 is shown as an example of a system having these.
[0013] 1, the information providing system 1 includes an information providing device 10, a model generation server 2, and a terminal device 3. The information providing system 1 may include a plurality of model generation servers 2 and a plurality of terminal devices 3. The information providing device 10 and the model generation server 2 may be realized by the same server device, a cloud system, or the like. Here, the information providing device 10, the model generation server 2, and the terminal device 3 are connected to each other via a network N so as to be able to communicate with each other via wired or wireless communication.
[0014] The information providing device 10 is an information processing device that executes an index generation process that generates generation indicators, which are indicators for generating a model (i.e., a recipe for the model), and a model generation process that generates a model according to the generation indicators, and provides the generated generation indicators and models, and is realized, for example, by a server device, a cloud system, etc.
[0015] The model generation server 2 is a generation device that generates a model that has been trained on the characteristics of the training data, and is realized by, for example, a server device or a cloud system. For example, when the model generation server 2 receives a configuration file that indicates the type and behavior of the model to be generated and how to train the characteristics of the training data as a generation index of the model, the model generation server 2 automatically generates a model according to the received configuration file. Note that the model generation server 2 may train the model using any model training method. Also, for example, the model generation server 2 may be any of various existing services such as AutoML.
[0016] The terminal device 3 is a terminal device used by a user U, and is realized by, for example, a PC (Personal Computer) or a server device. For example, the terminal device 3 generates a generation index of a model through an exchange with the information providing device 10, and acquires a model generated by the model generation server 2 according to the generated generation index.
[0017] 3. Overview of Processing Executed by Information Providing Device Next, an overview of the process executed by the information providing device 10 will be described. First, the information providing device 10 receives, from the terminal device 3, an indication of learning data for causing the model to learn features (step S1). For example, the information providing device 10 stores various learning data used for learning in a predetermined storage device, and receives an indication of learning data designated by the user U as the learning data. Note that the information providing device 10 may obtain the learning data used for learning, for example, from the terminal device 3 or various external servers.
[0018] Here, any data can be used as the learning data. For example, the information providing device 10 may use various information related to the users as the learning data, such as the location history of each user, the history of web content viewed by each user, the purchase history of each user, and the search query history of each user. The information providing device 10 may also use demographic attributes, psychographic attributes, and the like of the users as the learning data. The information providing device 10 may also use metadata such as the type, content, and creator of various web contents to be distributed as the learning data.
[0019] In such a case, the information providing device 10 generates candidates for generation indicators based on statistical information of the learning data used for learning (step S2). For example, the information providing device 10 generates candidates for generation indicators indicating what model should be used for learning and what learning method should be used for learning, based on the characteristics of the values included in the learning data. In other words, the information providing device 10 generates, as generation indicators, models that can accurately learn the characteristics of the learning data and learning methods for making the models accurately learn the characteristics. That is, the information providing device 10 optimizes the learning method. Note that the content of the generation indicators to be generated when what type of learning data is selected will be described later.
[0020] Next, the information providing device 10 provides the candidates for the generation index to the terminal device 3 (step S3). In such a case, the user U modifies the candidates for the generation index according to preferences, rules of thumb, etc. (step S4). Then, the information providing device 10 provides the candidates for the generation index and the learning data to the model generation server 2 (step S5).
[0021] Meanwhile, the model generation server 2 generates a model for each generation index (step S6). For example, the model generation server 2 trains the model having the structure indicated by the generation index with the features of the training data by the learning method indicated by the generation index. Then, the model generation server 2 provides the generated model to the information providing device 10 (step S7).
[0022] Here, it is considered that the accuracy of each model generated by the model generation server 2 differs due to the difference in the generation index. Therefore, the information providing device 10 generates a new generation index by a genetic algorithm based on the accuracy of each model (step S8), and repeatedly generates a model using the newly generated generation index (step S9).
[0023] For example, the information providing device 10 divides the learning data into evaluation data and learning data, and acquires a plurality of models that are models that have been trained based on features of the learning data and that are generated according to different generation indices. For example, the information providing device 10 generates 10 generation indices, and generates 10 models using the generated 10 generation indices and the learning data. In such a case, the information providing device 10 measures the accuracy of each of the 10 models using the evaluation data.
[0024] Next, the information providing device 10 selects a predetermined number of models (for example, five) from the ten models in order of increasing accuracy. Then, the information providing device 10 generates new generation indices from the generation indices adopted when generating the five selected models. For example, the information providing device 10 regards each generation index as an individual of a genetic algorithm, and regards the type of model, the structure of the model, and various learning methods indicated by each generation index (i.e., various indices indicated by the generation index) as genes in the genetic algorithm. Then, the information providing device 10 generates ten new generation indices for the next generation by selecting individuals to perform gene crossover and performing gene crossover. Note that the information providing device 10 may take mutation into consideration when performing gene crossover. Also, the information providing device 10 may perform two-point crossover, multi-point crossover, uniform crossover, or random selection of genes to be crossovered. Furthermore, the information providing device 10 may adjust a crossover rate when performing crossover so that, for example, the genes of an individual with a higher model accuracy are passed on to individuals of the next generation.
[0025] Furthermore, the information providing device 10 again generates new 10 models using the generation index of the next generation. Then, based on the accuracy of the new 10 models, the information providing device 10 generates new generation indexes by the above-mentioned genetic algorithm. By repeatedly executing such processing, the information providing device 10 can bring the generation indexes closer to generation indexes according to the characteristics of the learning data, that is, optimized generation indexes.
[0026] Furthermore, when a predetermined condition is met, such as when a new generation index has been generated a predetermined number of times or when the maximum, average, or minimum value of the model accuracy exceeds a predetermined threshold, the information providing device 10 selects the model with the highest accuracy as the model to be provided. Then, the information providing device 10 provides the selected model and the corresponding generation index to the terminal device 3 (step S10). As a result of such processing, the information providing device 10 can generate a generation index for an appropriate model and provide a model that conforms to the generated generation index, simply by selecting learning data from the user.
[0027] In the above example, the information providing device 10 realizes stepwise optimization of the generation index by using a genetic algorithm, but the embodiment is not limited to this. As will be clear from the following description, the accuracy of a model varies greatly depending on the index when generating a model (i.e., when learning the characteristics of the learning data), such as not only the characteristics of the model itself, such as the type and structure of the model, but also what learning data is input to the model and how, and what hyperparameters are used to learn the model.
[0028] Therefore, the information providing device 10 may not perform optimization using a genetic algorithm as long as it generates a generation index estimated to be optimal according to the learning data. For example, the information providing device 10 may present to the user a generation index generated according to whether or not the learning data satisfies various conditions generated according to empirical rules, and generate a model according to the presented generation index. Furthermore, when the information providing device 10 receives a correction to the presented generation index, it may generate a model according to the received corrected generation index, present the accuracy of the generated model to the user, and receive a correction to the generation index again. That is, the information providing device 10 may cause the user U to find an optimal generation index by trial and error.
[0029] [4. Generation of generated indicators] An example of how to generate a generation index for a certain type of learning data will be described below. Note that the following example is merely an example, and any process can be used as long as the generation index is generated according to the characteristics of the learning data.
[0030] [4-1. About generation indicators] First, an example of information indicated by the generation index will be described. For example, when a model is made to learn features of training data, it is considered that the manner in which the training data is input to the model, the manner of the model, and the learning manner of the model (i.e., the features indicated by the hyperparameters) contribute to the accuracy of the model finally obtained. Therefore, the information providing device 10 improves the accuracy of the model by generating a generation index in which each aspect is optimized according to the features of the training data.
[0031] For example, it is considered that the learning data includes data to which various labels are attached, that is, data showing various characteristics. However, if data showing characteristics that are not useful when classifying data is used as the learning data, the accuracy of the model finally obtained may deteriorate. Therefore, the information providing device 10 determines the characteristics of the learning data to be input as a mode when inputting the learning data to the model. For example, the information providing device 10 determines which label-attached data (i.e., data showing which characteristics) is to be input from among the learning data. In other words, the information providing device 10 optimizes the combination of the characteristics to be input.
[0032] In addition, the learning data is considered to include columns of various formats, such as data containing only numerical values and data containing character strings. When such learning data is input to a model, the accuracy of the model is considered to change depending on whether the data is input as is or converted into another format. For example, when multiple types of learning data (learning data showing different characteristics) are input, and learning data of character strings and learning data of numerical values are input, the accuracy of the model is considered to change depending on whether the character strings and numerical values are input as is, whether the character strings are converted into numerical values and only numerical values are input, or whether the numerical values are considered as character strings and input. Therefore, the information providing device 10 determines the format of the learning data to be input to the model. For example, the information providing device 10 determines whether the learning data to be input to the model is to be a numerical value or a character string. In other words, the information providing device 10 optimizes the column type of the input feature.
[0033] In addition, when there are learning data showing different features, it is considered that the accuracy of the model changes depending on which combination of features is input simultaneously. That is, when there are learning data showing different features, it is considered that the accuracy of the model changes depending on which combination of features (i.e., the relationship of a combination of multiple features) is learned. For example, when there is learning data showing a first feature (e.g., gender), learning data showing a second feature (e.g., address), and learning data showing a third feature (e.g., purchase history), it is considered that the accuracy of the model changes when the learning data showing the first feature and the learning data showing the second feature are input simultaneously and when the learning data showing the first feature and the learning data showing the third feature are input simultaneously. Therefore, the information providing device 10 optimizes the combination of features (cross features) that allows the model to learn the relationship.
[0034] Here, various models project input data into a space of a predetermined dimension divided by a predetermined hyperplane, and classify the input data according to which of the divided spaces the projected position belongs to. Therefore, if the number of dimensions of the space to which the input data is projected is lower than the optimal number of dimensions, the classification ability of the input data deteriorates, and the accuracy of the model deteriorates. In addition, if the number of dimensions of the space to which the input data is projected is higher than the optimal number of dimensions, the inner product value with the hyperplane changes, and as a result, it may become impossible to properly classify data different from the data used during learning. Therefore, the information providing device 10 optimizes the number of dimensions of the input data input to the model. For example, the information providing device 10 optimizes the number of dimensions of the input data by controlling the number of nodes in the input layer of the model. In other words, the information providing device 10 optimizes the number of dimensions of the space in which the input data is embedded.
[0035] In addition to SVM, there are also models such as neural networks with multiple intermediate layers (hidden layers). Various types of neural networks are known, such as feedforward DNNs in which information is transmitted in one direction from the input layer to the output layer, convolutional neural networks (CNNs) that convolve information in intermediate layers, recurrent neural networks (RNNs) with directed loops, and Boltzmann machines. These various neural networks include long short-term memory (LSTM) and various other neural networks.
[0036] In this way, when the type of model that learns various features of the learning data is different, the accuracy of the model is considered to change. Therefore, the information providing device 10 selects a type of model that is estimated to learn the features of the learning data with high accuracy. For example, the information providing device 10 selects a type of model depending on what label is attached as the label of the learning data. To give a more specific example, when data with a term related to "history" attached as a label exists, the information providing device 10 selects an RNN that is considered to be able to learn the features of history better, and when data with a term related to "image" attached as a label exists, the information providing device 10 selects a CNN that is considered to be able to learn the features of images better. In addition to these, the information providing device 10 may determine whether the label is a term specified in advance or a term similar to a term, and may select a model of a type that is previously associated with a term determined to be the same or similar.
[0037] In addition, when the number of intermediate layers of the model or the number of nodes included in one intermediate layer changes, the learning accuracy of the model is considered to change. For example, when the number of intermediate layers of the model is large (when the model is deep), it is considered that classification according to more abstract features can be realized, but as a result, local errors in backpropagation are less likely to propagate to the input layer, and learning may not be performed appropriately. In addition, when the number of nodes included in the intermediate layers is small, more advanced abstraction can be performed, but when the number of nodes is too small, there is a high possibility that information required for classification will be lost. Therefore, the information providing device 10 optimizes the number of intermediate layers and the number of nodes included in the intermediate layers. That is, the information providing device 10 optimizes the architecture of the model.
[0038] In addition, it is considered that the accuracy of the nodes changes depending on whether attention is present or not, whether the nodes included in the model have autoregression or not, and which nodes are connected. Therefore, the information providing device 10 optimizes the network by determining whether or not the network has autoregression and which nodes are connected.
[0039] Furthermore, when learning a model, the optimization method for the model (algorithm used during learning), the dropout rate, the activation function of the node, the number of units, etc. are set as hyperparameters. It is considered that the accuracy of the model changes when such hyperparameters change. Therefore, the information providing device 10 optimizes the learning mode when learning the model, that is, the hyperparameters.
[0040] Furthermore, when the size of the model (the number of input layers, intermediate layers, and output layers, and the number of nodes) changes, the accuracy of the model also changes. Therefore, the information providing device 10 also optimizes the size of the model.
[0041] In this way, the information providing device 10 optimizes the indices used when generating the various models described above. For example, the information providing device 10 holds in advance conditions corresponding to each index. Note that such conditions are set, for example, based on empirical rules such as the accuracy of various models generated from past learning models. Then, the information providing device 10 determines whether the learning data satisfies each condition, and adopts an index that is previously associated with a condition that the learning data satisfies or does not satisfy as a generated index (or a candidate thereof). As a result, the information providing device 10 can generate a generated index that can accurately learn the features of the learning data.
[0042] As described above, when the generation index is automatically generated from the training data and the process of creating a model according to the generation index is automatically performed, the user does not need to refer to the inside of the training data and determine what kind of distribution data is present. As a result, the information providing device 10 can reduce the effort required for a data scientist or the like to recognize the training data when creating a model, and can prevent the loss of privacy that accompanies the recognition of the training data.
[0043] [4-2. Generated indicators according to data type] An example of a condition for generating a generation index will be described below. First, an example of a condition according to what kind of data is adopted as the learning data will be described.
[0044] For example, the learning data used for learning includes integers, floating points, character strings, etc. As a result, it is estimated that if a model appropriate for the format of input data is selected, the learning accuracy of the model will be higher. Therefore, the information providing device 10 generates a generation index based on whether the learning data is an integer, a floating point, or a character string.
[0045] For example, when the training data is an integer, the information providing device 10 generates a generation index based on the continuity of the training data. For example, when the density of the training data exceeds a predetermined first threshold, the information providing device 10 regards the training data as data having continuity, and generates a generation index based on whether the maximum value of the training data exceeds a predetermined second threshold. When the density of the training data is below the predetermined first threshold, the information providing device 10 regards the training data as sparse training data, and generates a generation index based on whether the number of unique values included in the training data exceeds a predetermined third threshold.
[0046] A more specific example will be described. In the following example, an example of a process for selecting a feature function as a generation index from a configuration file to be transmitted to a model generation server 2 that automatically generates a model by AutoML will be described. For example, when the learning data is an integer, the information providing device 10 determines whether or not the density exceeds a predetermined first threshold. For example, the information providing device 10 calculates the density by dividing the number of unique values included in the learning data by the maximum value of the learning data plus 1.
[0047] Next, when the density exceeds a predetermined first threshold, the information providing device 10 determines that the learning data is learning data having continuity, and determines whether the value obtained by adding 1 to the maximum value of the learning data exceeds a second threshold. Then, when the value obtained by adding 1 to the maximum value of the learning data exceeds the second threshold, the information providing device 10 selects "Categorical_colum_with_identity & embedding_column" as the feature function. On the other hand, when the value obtained by adding 1 to the maximum value of the learning data is below the second threshold, the information providing device 10 selects "Categorical_column_with_identity" as the feature function.
[0048] On the other hand, when the density is below a predetermined first threshold, the information providing device 10 determines that the training data is sparse, and determines whether the number of unique values included in the training data exceeds a predetermined third threshold. When the number of unique values included in the training data exceeds the predetermined third threshold, the information providing device 10 selects "Categorical_column_with_hash_bucket & embedding_column" as the feature function, and when the number of unique values included in the training data is below the predetermined third threshold, the information providing device 10 selects "Categorical_column_with_hash_bucket" as the feature function.
[0049] Furthermore, when the learning data is a character string, the information providing device 10 generates a generation index based on the number of types of character strings included in the learning data. For example, the information providing device 10 counts the number of unique character strings (the number of unique data) included in the learning data, and when the counted number is below a predetermined fourth threshold, selects "categorical_column_with_vocabulary_list" and / or "categorical_column_with_vocabulary_file" as the feature function. When the counted number is below a fifth threshold that is greater than the predetermined fourth threshold, the information providing device 10 selects "categorical_column_with_vocabulary_file & embedding_column" as the feature function. When the counted number is greater than a fifth threshold that is greater than the predetermined fourth threshold, the information providing device 10 selects "categorical_column_with_hash_bucket & embedding_column" as the feature function.
[0050] Furthermore, when the learning data is a floating point, the information providing device 10 generates a conversion index for inputting the learning data into the model as a generation index for the model. For example, the information providing device 10 selects "bucketized_column" or "numeric_column" as a feature function. That is, the information providing device 10 bucketizes (groups) the learning data and selects whether to input the bucket number or to input the numerical value as it is. Note that the information providing device 10 may bucketize the learning data so that the range of numerical values associated with each bucket is approximately the same, and may associate a numerical range with each bucket so that the number of learning data classified into each bucket is approximately the same. Furthermore, the information providing device 10 may select the number of buckets or the range of numerical values associated with the bucket as a generation index.
[0051] Furthermore, the information providing device 10 acquires learning data indicating multiple features, and generates a generation index indicating a feature to be learned by the model among the features possessed by the learning data, as a generation index for the model. For example, the information providing device 10 determines which label of the learning data is to be input to the model, and generates a generation index indicating the determined label. Furthermore, the information providing device 10 generates a generation index indicating multiple types of the learning data types, the correlation of which is to be learned by the model, as a generation index for the model. For example, the information providing device 10 determines a combination of labels to be simultaneously input to the model, and generates a generation index indicating the determined combination.
[0052] Furthermore, the information providing device 10 generates a generation index indicating the number of dimensions of the learning data input to the model as a generation index of the model. For example, the information providing device 10 may determine the number of nodes in the input layer of the model according to the number of unique data included in the learning data, the number of labels input to the model, a combination of the numbers of labels input to the model, the number of buckets, etc.
[0053] Furthermore, the information providing device 10 generates a generation index indicating the type of model that learns the features of the learning data as a generation index of the model. For example, the information providing device 10 determines the type of model to be generated according to the density and sparseness of the learning data previously used as the learning target, the label contents, the number of labels, the number of label combinations, and the like, and generates a generation index indicating the determined type. For example, the information providing device 10 generates a generation index indicating "BaselineClassifier", "LinearClassifier", "DNNClassifier", "DNNLinearCombinedClassifier", "BoostedTreesClassifier", "AdaNetClassifier", "RNNClassifier", "DNNResNetClassifier", "AutoIntClassifier", and the like as the class of the model in AutoML.
[0054] The information providing device 10 may generate generation indices indicating various independent variables of the models of each class. For example, the information providing device 10 may generate, as the generation indices of the model, a generation indices indicating the number of intermediate layers the model has or the number of nodes included in each layer. The information providing device 10 may also generate, as the generation indices of the model, a generation indices indicating the connection state between the nodes in the model or a generation indices indicating the size of the model. These independent variables are appropriately selected depending on whether various statistical features of the learning data satisfy predetermined conditions.
[0055] Furthermore, the information providing device 10 may generate, as a generation index of the model, a generation index indicating a learning mode when the model learns the features of the learning data, i.e., a hyperparameter. For example, the information providing device 10 may generate a generation index indicating "stop_if_no_decrease_hook", "stop_if_no_increase_hook", "stop_if_higher_hook", or "stop_if_lower_hook" in the setting of the learning mode in AutoML.
[0056] That is, the information providing device 10 generates a generation index indicating the characteristics of the learning data to be learned by the model, the mode of the model to be generated, and the learning mode when the model is trained on the characteristics of the learning data, based on the labels of the learning data used for learning and the characteristics of the data itself. More specifically, the information providing device 10 generates a configuration file for controlling the generation of a model in AutoML.
[0057] [4-3. Order of determining generation indicators] Here, the information providing device 10 may optimize the various indices described above simultaneously or in an appropriate order. The information providing device 10 may also be able to change the order in which the indices are optimized. That is, the information providing device 10 may receive from the user a designation of the order in which the features of the learning data to be learned by the model, the mode of the model to be generated, and the learning mode when the features of the learning data are learned by the model are determined, and may determine the indices in the order received.
[0058] For example, when the information providing device 10 starts generating a generation index, it optimizes input features such as the features of the input learning data and the manner in which the learning data is input, and then optimizes input cross features to determine which combination of features is to be learned. Next, the information providing device 10 selects a model and optimizes the model structure. After that, the information providing device 10 optimizes hyperparameters and ends the generation of the generation index.
[0059] Here, in the input feature optimization, the information providing device 10 may iteratively optimize the input features by selecting or modifying various input features such as the characteristics and input mode of the learning data to be input, and selecting new input features using a genetic algorithm. Similarly, in the input cross feature optimization, the information providing device 10 may iteratively optimize the input cross features, and may iteratively perform model selection and model structure optimization. Furthermore, the information providing device 10 may iteratively optimize hyperparameters. Furthermore, the information providing device 10 may iteratively perform a series of processes, including input feature optimization, input cross feature optimization, model selection, model structure optimization, and hyperparameter optimization, to optimize each index.
[0060] Furthermore, the information providing device 10 may, for example, perform hyperparameter optimization before performing model selection or model structure optimization, or may perform input feature optimization or input cross feature optimization after model selection or model structure optimization. Furthermore, the information providing device 10 may, for example, repeatedly perform input feature optimization and then repeatedly perform input cross feature optimization. Thereafter, the information providing device 10 may repeatedly perform input feature optimization and input cross feature optimization. In this way, any setting can be adopted for which indicators are optimized in what order, and which optimization processes are repeatedly executed in the optimization.
[0061] 5. Information Processing According to the Embodiment So far, various types of processing executed by the information providing device 10 have been explained using Fig. 1. From here on, information processing executed by the information processing device 100 and information processing executed by the execution control device 200 will be explained.
[0062] [5-1, Configuration of information processing system] First, prior to describing information processing according to the embodiment, an information processing system Sy, which is a part of a system included in the information providing system 1, will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the information processing system Sy according to the embodiment. The information processing system Sy corresponds to a partial system of the information providing system 1 that includes only the information processing device 100 and the execution control device 200.
[0063] 2, the information processing system Sy includes an information processing device 100 and an execution control device 200. In this embodiment, the information processing device 100 is described as a server device, but may be realized by a cloud system or the like. In this embodiment, the execution control device 200 is described as a server device, but may be realized by a cloud system or the like.
[0064] As described with reference to FIG. 1, the information providing device 10 optimizes the architecture of a model in accordance with the characteristics of data and automatically generates a model in order to facilitate the creation of the model.
[0065] In contrast, the information processing device 100 performs main information processing of optimizing a learning / generation method, such as how to learn or generate a model. The information processing device 100 can also operate as the information providing device 10 by having some or all of the functions of the information providing device 10. The information processing device 100 can also have some or all of the functions of the model generation server 2. The information processing device 100 executes various processes shown in the following embodiments in addition to the processes described in FIG. 1 as being executed by the information providing device 10.
[0066] Furthermore, the execution control device 200 performs information processing mainly for optimizing an execution subject that executes processing using a model (for example, processing for predicting a specific target).
[0067] The optimization process executed by the information processing device 100 is roughly divided into an optimization process for optimizing a learning method for learning or generating a model, and an optimization process for optimizing data to be input to a learned model in a situation where the learned model is actually used. Therefore, in the following embodiment, first, the optimization process for optimizing a learning method and the optimization process for optimizing data to be input to a learned model are described in order for the information processing device 100, and then the optimization process executed by the execution control device 200 is described.
[0068] Furthermore, the optimization process for optimizing the learning technique can be further classified into five optimization processes, such as the first optimization to the fifth optimization, which will be described later. Therefore, regarding the optimization process for optimizing the learning technique, first, an overview of each of the first optimization to the fifth optimization and an example of the execution order in which the first optimization to the fifth optimization are executed will be described using FIG. 3 below. After that, a detailed example of each of the first optimization to the fifth optimization will be described based on the functional configuration diagram shown in FIG. 5.
[0069] 5-2. Example of processing executed by information processing device From here, an example of the process executed by the information processing device 100 will be described with reference to FIG. 3. FIG. 3 is a diagram showing an overall image of the process executed by the information processing device 100 according to the embodiment. For example, when a model is put into practical use, there is a motivation to make the model size as small as possible, reduce unnecessary calculations, and increase the inference speed. Therefore, FIG. 3 shows a scene in which a computation graph is optimized to improve the size of the model and performance in a serving environment when providing (serving) inference by the model as an API. A computation graph is a representation of computation processing by a directed graph, and the vertices (nodes) of the graph represent the computation content to be executed, and the edges represent the input and output of each node. For this reason, a model is defined as, for example, a graph of tensor computation.
[0070] Furthermore, according to the above, the information processing device 100 optimizes the learning method to tune the model so as to serve a higher performance model. For this reason, Fig. 3 explains a series of tuning algorithms (fine tuning according to the embodiment) including various optimizations according to the embodiment.
[0071] Furthermore, fine tuning according to the embodiment is divided into an optimization process for optimizing a learning method, and a tuning process for fine-tuning a trained model obtained by the optimization process by partially modifying and re-training the model, as shown in Fig. 3. The optimization process is executed by, for example, an optimization function (referred to as "optimizer OP") of the information processing device 100. Furthermore, the tuning process is executed by a data selection function (referred to as "selector SE") of the information processing device 100.
[0072] First, the information processing device 100 generates multiple initial values of model parameters (e.g., weights and biases) based on random numbers (pseudo-random numbers) (step S11). At this time, the information processing device 100 performs a first optimization for optimizing a seed for obtaining random numbers (i.e., a random number seed), thereby controlling the model parameters to be initialized more appropriately. For this reason, the first optimization is an optimization of a random number seed in a computation graph.
[0073] In deep learning, the initial values of the model parameters are determined based on pseudorandom numbers, and the model is made to learn the characteristics of the training data. As a result of this processing, the values of the model parameters gradually change (converge) to values corresponding to the characteristics of the training data. Therefore, if the initial values of the model parameters are significantly different from the values corresponding to the characteristics of the training data, it takes a long time to learn, and the learning speed becomes slow. From this perspective, a processing method can be considered in which multiple models each having different initial values are generated, and the model with the highest improvement in accuracy among the generated models is adopted as the learning result.
[0074] On the other hand, considering the relationship between the model parameters and the accuracy achieved by the set of the model parameters, it is estimated that, due to the structure of the model, there is a substantially continuous relationship in which the accuracy is higher as the model parameters are closer to the optimum value, rather than a relationship in which the accuracy changes intermittently for each model parameter. Also, if the initial value of the model parameter is not an optimal value according to the learning data but is closer to a minimum value, the model parameter may remain at the minimum value, and the accuracy may not be improved. For this reason, when generating multiple models with different initial values, it is considered desirable to generate a group of initial values of the model parameters having a certain degree of spread (i.e., distribution).
[0075] Therefore, the information processing device 100 executes the first optimization so that a plurality of models can be generated in which the set of model parameters has a predetermined distribution. For example, when generating model parameters for each model, the information processing device 100 generates the model parameters using a predetermined random function from a predetermined initial value. Such a random function allows various settings such as what distribution the random numbers to be generated, such as random numbers with uniform distribution or random numbers with normal distribution, what average value the random numbers to be generated from an input seed value, and what range of random numbers to be generated. Therefore, the information processing device 100 optimizes the random number seed value, such as the seed value to be input to the random function and various settings.
[0076] More specifically, the information processing device 100 sets a plurality of random number seeds that satisfy a predetermined distribution through the first optimization. Then, the information processing device 100 inputs each of the set random number seeds into a random function to generate random numbers corresponding to the random number seeds for each random number seed. The random numbers thus generated exhibit a predetermined distribution. Therefore, by using such random numbers, the information processing device 100 can generate a group of initial values of model parameters having a predetermined distribution in step S11.
[0077] Next, the information processing device 100 generates a model for each initial value of the model parameters generated in step S11 (step S12). Specifically, the information processing device 100 generates a model having a set of model parameters for each set of different combinations of model parameters among the initial value group of the model parameters that fall within a predetermined distribution.
[0078] Next, the information processing device 100 randomly extracts data for this repeated learning (i.e., learning data to be learned) from the learning data, and stores the extracted data in a buffer. Then, when the information providing device 10 has completed learning of the features of the data stored in the buffer, it extracts new data and stores it in the buffer, and executes learning of the data stored in the buffer, thereby controlling so that repeated learning is performed according to the shuffle (step S13).
[0079] Here, when the learning data set is divided into several subsets, the model with the best performance is not necessarily learned when all the subsets are used for learning the model. On the other hand, when the model is learned by the above-mentioned repeated learning, it is considered that the accuracy of the model can be improved by optimizing the combination of data included in one subset. Therefore, when performing step S13, the information processing device 100 performs a second optimization for optimizing the learning data to determine which learning data from the data set is actually used for learning, and a third optimization for optimizing the size of the buffer in which shuffling is performed. In this way, the second optimization is to optimize the data used for learning. Also, the third optimization is to optimize the shuffle buffer size.
[0080] For example, in step S13, the information processing device 100 performs second and third optimizations to generate learning data to be used in the current iterative learning (learning data according to the optimized buffer size) and stores the learning data in the buffer.
[0081] Furthermore, the information processing device 100 causes each model generated in step S12 to learn the features of the learning data stored in the buffer in step S13 (step S14).
[0082] For example, the information processing device 100 learns the features of the learning data stored in the buffer one by one in order, and at this time, shuffles the learning order (order of the learning data) in the buffer. Specifically, the information processing device 100 shuffles the learning order into a random order every epoch.
[0083] Here, it is considered important to shuffle data well in learning a model, but if data is simply shuffled, there is a risk that learning will not be performed well, for example, because bias will occur in the learning order or the data distribution for each batch. For example, when learning a model, the features of the learning data are sequentially learned, such as learning a model using certain learning data (modifying model parameters) and then learning a model using different learning data. For this reason, when the learning data has a time series, it is considered better to disperse the time series of the learning data to a certain extent in order to widely and generally learn the features of the learning data. On the other hand, when there is a large gap in the time series of the learning data inputted into the model in succession, the correction range of the model parameters will become large, and there is a risk that appropriate learning will not be performed. In other words, when learning the features of the learning data having a time series into a model, it is necessary to use the learning data in order so that the time series varies to a certain extent in order to learn features that are not bound by the time series, but if the variation in the time series is too large, there is a risk that appropriate model learning will not be performed. In such a case, the accuracy of the model cannot be improved.
[0084] For this reason, when performing step S14, the information processing device 100 optimizes the seed value used in generating the random order so that the random order is not biased between epochs (so that the distribution is uniform). Specifically, the information processing device 100 performs a fourth optimization that optimizes the seed (i.e., the random number seed) for random order generation, thereby generating an optimal random order so that specific learning data is not learned in the same order every time. For this reason, the fourth optimization is optimization of the random number seed in data shuffling.
[0085] For example, as the fourth optimization, the information processing device 100 generates random number seeds for the current learning so that the random order associated with each learning data between epochs is not biased. Then, the information processing device 100 generates a random order by inputting each generated random number seed into a random function. Also, the information processing device 100 generates the final learning data of the learning target in the buffer by associating the generated random order with the learning data of each learning target. As a result, in the actual learning, learning is performed for each pair of the model and the learning data obtained by multiplying the model having each model parameter generated to show a predetermined distribution by the first optimization with the learning data whose order is randomly determined by the fourth optimization.
[0086] Then, the information processing device 100 sequentially trains each model on the features of the learning data of the final learning target in the generated random order. Specifically, when the information processing device 100 finishes learning the features of the learning data of the learning target in the generated random order (when one epoch is completed), it moves to the next epoch in which it generates a random order again and trains each model on the features of the learning data in the generated random order again. In this way, the information processing device 100 repeats a loop in which learning is repeated a specified number of epochs.
[0087] When the loop of repeatedly learning the specified number of epochs ends, the buffer becomes empty. Therefore, the information processing device 100 stores the unprocessed learning data among the learning data of the learning target obtained in step S13 in the now empty buffer, and further repeats step S14 on the stored learning data of the learning target, thereby learning all of the learning data of the learning target obtained in step S13.
[0088] A detailed example of the second to fourth optimizations and a detailed example of the iterative learning in steps S13 and S14 will be described later.
[0089] In addition, in the actual learning in step S14, trials to search for hyperparameters are repeated, and in order to realize an efficient search, the information processing device 100 executes a fifth optimization as an optimization of trials by pruning. For this reason, the fifth optimization is an optimization related to early stopping, which terminates trials that are not expected to produce good results early without continuing them to the end.
[0090] For example, the information processing device 100 allows the user to specify constraint conditions that condition trials to be subject to early stopping from the viewpoint of an evaluation value that evaluates the accuracy of the model. The information processing device 100 then monitors whether the constraint conditions are satisfied for each trial, and when it is determined that the constraint conditions are satisfied, it terminates the trial and continues only the remaining trials. In other words, the information processing device 100 selects only trials whose evaluation values for evaluating the accuracy of the model satisfy a predetermined condition (for example, the inverse of the constraint conditions) (trials not selected are subject to pruning), and continues learning for the selected trials. A detailed example of the fifth optimization will be described later.
[0091] Furthermore, the information processing device 100 selects a best model from among the generated models based on the accuracy of each model trained by the learning process to which the optimization process has been applied (step S15). For example, the information processing device 100 calculates the accuracy of each model using the evaluation data, and calculates a higher evaluation value as the accuracy variation (amount of improvement in accuracy) increases. Then, the information processing device 100 selects the model with the highest calculated evaluation value as the best model.
[0092] So far, we have explained the learning method in which optimization processing is applied by the optimizer OP, but from here on, we will explain the tuning processing by the selector SE.
[0093] For example, the information processing device 100 executes a selector SE to perform a tuning process in which the best model is fine-tuned by modifying a part of the best model and re-learning it. The information processing device 100 can also use the learning data used in the learning process to which the optimization process is applied as a set of data in the tuning process.
[0094] Here, in the above datasets, each tuning process using learning data with different ranges (time ranges according to the time series) is treated as one trial, and in order to effectively evaluate each tuning result (the accuracy of the best model), the above datasets were divided according to their intended use as shown in Figure 4. Figure 4 is a diagram showing an example of division for each trial when the dataset is divided according to its intended use.
[0095] The data included in the dataset corresponds to a purchase history of products purchased using a specific service (e.g., a specific shopping service), and has a concept of a time series. Therefore, the data included in the dataset is arranged in chronological order. In the example of FIG. 4, the dataset has a time range from "June 11th, 0:00" to "June 19th, 0:00", and the data are arranged in chronological order from the oldest data (purchase history at June 11th, 0:00) to the newest data (purchase history at June 19th, 0:00).
[0096] 4, for such a data set, data from "June 11th, 0:00" to "June 16th, 17:32" is assigned as learning data for tuning to trial A. This example means that it has been decided that trial A will be a process of tuning the best model using data from "June 11th, 0:00" to "June 16th, 17:32" as learning data.
[0097] 4, data from "June 16th, 17:32" to "June 17th, 7:26" is assigned as evaluation data to trial A. This example shows an example in which it has been decided to evaluate the best model after tuning by trial A using data from "June 16th, 17:32" to "June 17th, 7:26".
[0098] 4, data from "June 17th, 7:26" to "June 19th, 0:00" is assigned as test data for trial A. This example shows an example in which it has been decided to evaluate the best model after tuning by trial A by using data from "June 17th, 7:26" to "June 19th, 0:00" as test data with unknown labels.
[0099] 4, data from "June 11th, 0:00" to "June 17th, 7:26" is assigned as learning data for tuning to trial B. This example means that it has been decided that trial B will be a process of tuning the best model using data from "June 11th, 0:00" to "June 17th, 7:26" as learning data.
[0100] 4, data from "June 17th, 7:26" to "June 17th, 12:00" is assigned as evaluation data for trial B. This example shows an example in which it has been decided to evaluate the best model after tuning in trial B using data from "June 17th, 7:26" to "June 17th, 12:00".
[0101] 4, data from "June 17th, 12:00" to "June 19th, 0:00" is assigned as test data for trial B. This example shows an example in which it has been decided to evaluate the best model after tuning in trial B by using data from "June 17th, 12:00" to "June 19th, 0:00" as test data with unknown labels.
[0102] 4, data from "June 11th, 0:00" to "June 17th, 12:00" is assigned as learning data for tuning to trial C. This example means that it has been decided that trial C will be a process of tuning the best model using data from "June 11th, 0:00" to "June 17th, 12:00" as learning data.
[0103] 4, data from "June 17th, 12:00" to "June 19th, 0:00" is assigned as evaluation data to trial C. This example shows an example in which it has been decided to evaluate the best model after tuning in trial C using data from "June 17th, 12:00" to "June 19th, 0:00".
[0104] Note that the allocation shown in FIG. 4 is just an example, and for example, what data in the dataset is defined as learning data, what data is defined as evaluation data, and what data is defined as test data depending on the tuning process may be changed as appropriate to suit the convenience of an administrator who manages the model.
[0105] Returning to Fig. 3, the information processing device 100 performs the tuning process by iterative learning described below on the best model using the learning data shown in Fig. 4, and repeats evaluation using the evaluation data and test data shown in Fig. 4. The information processing device 100 also performs such a series of processes for each trial. Since the contents of the series of processes are the same regardless of the trial, an example of the series of processes will be described below for trial A.
[0106] For example, the information processing device 100 divides the learning data into sets each consisting of a predetermined number of data (step S21). The learning data for each set is managed, for example, in a file corresponding to the set. For example, the information processing device 100 can divide the learning data into several hundred sets (for example, 500 sets), but for ease of explanation, FIG. 3 shows an example in which the learning data is divided into 10 sets. Specifically, in FIG. 3, File "1" to File "10" are shown as an example of the 10 sets. Furthermore, a predetermined number of learning data are stored in each file.
[0107] In this state, the information processing device 100 randomly selects one set from each set obtained by division and adds it to the learning data list (step S22). Then, each time a new set is added, the information processing device 100 trains the best model with the features of the learning data in the currently added set (step S23). For example, the information processing device 100 trains the learning data in the currently added set for only one epoch. Then, the information processing device 100 repeats a series of processes in which the accuracy of the trained best model is evaluated using the evaluation data and the test data (step S24).
[0108] In this regard, the example of Fig. 3 shows an example in which the information processing device 100 selects File "6" in step S22 for the first time and adds the selected File "6" to the learning data list. Also, an example is shown in which the information processing device 100 causes the best model to learn the features of the learning data included in File "6", which is the currently added set, in step S23 for the first time. Also, an example is shown in which the information processing device 100 evaluates the best model that has learned the features of the learning data included in File "6" using evaluation data and test data in step S24 for the first time.
[0109] 3 shows an example in which the information processing device 100 further selects File "9" in step S22 for the second time and adds the selected File "9" to the learning data list. Also, an example in which the information processing device 100 causes the best model to learn the features of the learning data contained in File "9", which is the pair added this time, in step S23 for the second time. Also, an example in which the information processing device 100 evaluates the best model that has learned the features of the learning data contained in Files "6" and "9" so far, using the evaluation data and the test data in step S24 for the second time.
[0110] 3 shows an example in which the information processing device 100 further selects File "3" in step S22 for the third time and adds the selected File "3" to the learning data list. Also, an example is shown in which the information processing device 100 causes the best model to learn the features of the learning data contained in File "3," which is the currently added set, in step S23 for the third time. Also, an example is shown in which the information processing device 100 evaluates the best model, which has learned the features of the learning data contained in Files "6," "9," and "3," so far, in step S24 for the third time, using the evaluation data and the test data.
[0111] In more detail, regarding the loop from steps S22 to S24, the information processing device 100 randomly selects one data file from the training data, adds the selected data file to the training data list of Model Config, and trains the best model on the training data included in the added data file for one epoch.
[0112] Furthermore, the information processing device 100 randomly selects one new data file for each Model Config that has been determined to be within the top five based on the evaluation results so far, and adds the selected data file to the training data list of the Model Config. Then, the information processing device 100 trains the best model for one epoch using the training data included in the training data list with the one additional data file.
[0113] Furthermore, the information processing device 100 continues the loop from steps S22 to S24 until it is determined that the performance (accuracy) of the best model will not be improved any further, based on the evaluation result.
[0114] Furthermore, the information processing device 100 can process the best model with the performance improved to the maximum as a serving target. For example, the information processing device 100 provides the best model with the performance improved by the fine tuning according to the embodiment in response to access from a user. According to such an information processing device 100, the user does not need to take the time to improve the model, and can focus on adjusting the data to be input to the model.
[0115] 6. Configuration of Information Processing Device Next, the information processing device 100 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 5, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.
[0116] (Regarding communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from, for example, the model generation server 2, the terminal device 3, the information providing device 10, and the execution control device 200.
[0117] (Regarding the storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk, an optical disk, etc. The storage unit 120 has a training data storage unit 121 and a model storage unit 122.
[0118] (Regarding the learning data storage unit 121) The learning data storage unit 121 stores various types of data related to learning. For example, the learning data storage unit 121 stores learning data that is divided into learning data, evaluation data, and test data.
[0119] For example, the information processing device 100 divides all the learning data into learning data, evaluation data, and test data, and registers these data obtained by the division in the learning data storage unit 121. For example, the information processing device 100 can divide all the learning data using any method. For example, the information processing device 100 can divide all the learning data using a hold-out method, a cross validation method, a leave one out method, or the like.
[0120] Here, an example of division of learning data will be shown with reference to Fig. 6. Fig. 6 is an explanatory diagram conceptually explaining division of a data set. As shown in Fig. 6, the information processing device 100 uses the generate_data() function to generate learning data composed of N data groups and test data composed of N data groups from the data set (data).
[0121] In addition, in this state, the information processing device 100 divides the learning data consisting of N data groups into learning data and evaluation data using the split_data() function. For example, the information processing device 100 divides the learning data so that the learning data and evaluation data are obtained in a ratio of "N1:N2" (actually, 7:3, etc.). Furthermore, the information processing device 100 determines all of the test data consisting of N data groups as test data.
[0122] Furthermore, the information processing device 100 registers the thus obtained learning data, evaluation data, and test data in the learning data storage unit 121.
[0123] (Regarding the model storage unit 122) The model storage unit 122 stores information about the model. For example, the model storage unit 122 saves a model updated for each epoch in a checkpoint file format. For example, the information processing device 100 saves parameters during learning at regular intervals in the model storage unit 122 and generates checkpoints.
[0124] (Regarding the control unit 130) The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
[0125] As shown in Fig. 3, the control unit 130 has a generation unit 131, an acquisition unit 132, a first data control unit 133, a second data control unit 134, a first learning unit 135, a model selection unit 136, a second learning unit 137, a provision unit 138, and an attribute selection unit 139, and realizes or executes functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 5, and may be other configurations as long as they perform information processing described below. Also, the connection relationship of each processing unit included in the control unit 130 is not limited to the connection relationship shown in Fig. 5, and may be other connection relationships.
[0126] (Regarding the generation unit 131) The generating unit 131 is a processing unit that performs the processes of steps S11 and S12 described with reference to Fig. 3. For this reason, the generating unit 131 performs the processes of steps S11 and S12 using a first optimization algorithm.
[0127] Specifically, the generating unit 131 generates a plurality of models each having different parameters. For example, the generating unit 131 generates a plurality of input values (random number seeds) to be input to a predetermined first function that calculates a random number value based on an input value, and generates a plurality of models having parameters (for example, weights and biases) for each generated input value according to the random number value (pseudo-random number) output by the predetermined first function when the input value is input.
[0128] In this regard, the generating unit 131 generates a plurality of input values as input values to the predetermined first function such that the random number value output by the predetermined first function satisfies a predetermined condition. For example, the generating unit 131 generates a plurality of input values such that the random number value falls within a predetermined range. Also, for example, the generating unit 131 generates a plurality of input values such that the distribution of the random number value shows a predetermined probability distribution. Also, for example, the generating unit 131 generates a plurality of input values such that the average value of the random number value becomes a predetermined value. Note that the input value here is a parameter input to the random function (one example of the predetermined first function) and corresponds to a random number seed.
[0129] For example, the generation unit 131 selects a function as a predetermined first function such that the distribution of random number values output when an input value is input exhibits a predetermined probability distribution (e.g., uniform distribution), and generates a plurality of models having parameters according to the random number values output by the selected function.
[0130] Furthermore, the generation unit 131 can register each generated model in the model storage unit 122 .
[0131] (Regarding the acquisition unit 132) The acquisition unit 132 acquires various information and passes the acquired information to an appropriate processing unit. For example, when optimization or learning is performed using the learning data, the acquisition unit 132 acquires the learning data from the learning data storage unit 121. Then, the acquisition unit 132 outputs the acquired learning data to the processing unit that performs the optimization or learning.
[0132] (Regarding the first data control unit 133) When the process of step S13 described with reference to FIG. 3 is performed, the first data control unit 133 optimizes the data used for learning by using the second optimization algorithm.
[0133] Specifically, the first data control unit 133 divides predetermined learning data (learning data) for causing a model to learn features into a plurality of sets in chronological order. For example, the first data control unit 133 divides the learning data into sets each having a predetermined number of data.
[0134] Furthermore, the first data control unit 133 selects a set to be actually used for model training from among the sets obtained by dividing the learning data into a plurality of sets in chronological order. For example, the first data control unit 133 selects a set that includes newer chronological learning data from among the sets obtained by dividing the learning data into a plurality of sets in chronological order.
[0135] The first data control unit 133 may randomly select a set to be used for model training from among the sets obtained by dividing the training data into a plurality of sets in chronological order.
[0136] Furthermore, the first data control unit 133 may select a number of sets designated by a user from among the sets obtained by dividing the learning data into a plurality of sets in chronological order. For example, the first data control unit 133 selects, in chronological order, sets that include newer chronological learning data from among the sets obtained by dividing the learning data into a plurality of sets in chronological order, until the number of selected sets reaches the number designated by the user.
[0137] Furthermore, the first data control unit 133 generates one data group by connecting the selected sets. For example, the first data control unit 133 generates one data group by connecting them in the selected order. Furthermore, the first data control unit 133 can pass the generated data group to, for example, the second data control unit 134 so that the generated data group can be used for model training.
[0138] (Regarding the second data control unit 134) The second data control unit 134 optimizes the shuffle buffer size using a third optimization algorithm when performing the process of step S13 described in Fig. 3. For example, in order to optimize the shuffle buffer size, the second data control unit 134 generates learning data of a size equal to the size of the shuffle buffer and stores this data in the shuffle buffer as learning data for the learning target, which is the learning data used in the current iterative learning.
[0139] For example, the second data control unit 134 divides the data group generated by the first data control unit 133 into a plurality of sets, each of which includes learning data of a size equal to the size of the shuffle buffer.
[0140] For example, the second data control unit 134 divides the data group generated by the first data control unit 133 into multiple sets in chronological order. For example, the second data control unit 134 divides the data group generated by the first data control unit 133 into sets having a number of pieces of learning data designated by a user. Also, for example, the second data control unit 134 may divide the learning data group generated by the first data control unit 133 into multiple sets so that the number of pieces of learning data included falls within a range designated by the user.
[0141] Furthermore, the second data control unit 134 stores, among the sets obtained by division, one set according to the time series of the included learning data in the shuffle buffer as the learning data to be learned, which is the learning data to be used in the current iterative learning. Specifically, the second data control unit 134 stores, among the sets obtained by division, the set including the oldest time series of included learning data in the shuffle buffer as the learning data to be learned.
[0142] (Regarding the first learning unit 135) The first learning unit 135 causes each of the multiple models generated by the generation unit 131 to learn features that are included in a portion of predetermined training data.
[0143] For example, the first learning unit 135 causes each of the multiple models generated by the generation unit 131 to learn features of the learning data (learning data to be learned) stored in a buffer (shuffle buffer) by the second data control unit 134. In this manner, for example, the first learning unit 135 uses, from among the groups selected by the first data control unit 133, groups containing learning data in ascending order of time series, and causes the model to learn features of the learning data contained in each group.
[0144] Also, for example, the first learning unit 135 causes the model to learn the features of the learning data (learning data to be learned) included in each group divided by the second data control unit 134 in a predetermined order. For example, the first learning unit 135 causes the model to learn the features of the learning data included in each group, in order from the group according to the time series, among the groups divided by the second data control unit 134. As an example, the first learning unit 135 causes the model to learn the features of the learning data included in each group, in order from the group with the oldest time series of the learning data included, among the groups divided by the second data control unit 134.
[0145] Furthermore, the first learning unit 135 may cause a model to learn the features of the learning data included in each group divided by the second data control unit 134 in a random order.
[0146] Here, when the first learning unit 135 makes each model learn the characteristics of the learning data as described above, it shuffles the learning order of each of the learning data currently stored in the shuffle buffer. Then, the first learning unit 135 generates the learning data of the final learning target by associating the learning order obtained by shuffling with the learning data. Then, the first learning unit 135 makes each of the learning data of the learning target learn in the learning order obtained by shuffling. Moreover, the first learning unit 135 repeats this series of processes related to shuffling as one epoch, for example, a specified number of epochs. Note that the first learning unit 135 can generate the learning data of the final learning target each time by shuffling the learning order every time an epoch is updated.
[0147] For example, the first learning unit 135 uses the fourth optimization algorithm to optimize data shuffling for shuffling the learning data in the shuffle buffer.
[0148] For example, the first learning unit 135 uses the fourth optimization algorithm to generate random number seeds for each epoch for repeated learning so that the random order associated with each learning data between epochs is not biased. The first learning unit 135 then generates a random order by inputting each generated random number seed into a random function. The first learning unit 135 also generates the final learning data of the learning target in the shuffle buffer by associating the generated random order with the learning data of each learning target.
[0149] Then, the first learning unit 135 sequentially trains each model on the features of the learning data of the final learning target in the generated random order. Specifically, when the first learning unit 135 finishes training the features of the learning data of the learning target in the generated random order (when one epoch is completed), it moves to the next epoch in which it generates a random order again and trains each model on the features of the learning data again in the generated random order.
[0150] In addition, in the actual learning process in which each model learns the characteristics of the learning data within the shuffle buffer size, trials are repeated to search for hyperparameters. At this time, the first learning unit 135 performs a fifth optimization related to early stopping, in which trials that are not expected to produce good results are terminated early (pruned) without continuing the trials to the end, in order to realize efficient search.
[0151] According to the fifth optimization, the first learning unit 135 performs the following process on each of the multiple models generated by the generation unit 131. For example, a trial is a process of searching for an optimal combination from among combinations of hyperparameters by applying each combination of hyperparameters to a model and repeating learning for each combination of hyperparameters. In other words, a trial is a process of optimizing a set of hyperparameters.
[0152] For this reason, the first learning unit 135 selects multiple trials (trials with different combinations of hyperparameters) for which the evaluation value for evaluating the accuracy of the model in the combination of hyperparameters corresponding to the trial satisfies a predetermined condition. Then, the first learning unit 135 continues to make the model in the selected trial learn the features of the learning data to be learned.
[0153] For example, the first learning unit 135 selects multiple trials whose aspects based on changes in evaluation values satisfy a predetermined aspect. For example, the first learning unit 135 selects multiple trials whose aspects based on changes in evaluation values while repeatedly learning the features of the learning data of the learning target a predetermined number of times satisfy a predetermined aspect. For example, the first learning unit 135 selects trials whose aspects satisfy a plurality of conditions specified by the user.
[0154] On the other hand, the first learning unit 135 stops (prunes) any trial (trial with a different combination of hyperparameters) whose evaluation value for evaluating the accuracy of the model for the combination of hyperparameters corresponding to that trial does not satisfy a predetermined condition, and does not perform any further trials.
[0155] Furthermore, for example, the first learning unit 135 can select one of the models for each combination of trials, each of which has a different combination of parameters, and the learning data to be learned, depending on the accuracy of the model that was learned.
[0156] (About the model selection unit 136) The model selection unit 136 selects a model (best model) that is evaluated to have the highest accuracy from among the multiple models based on the accuracy of each of the multiple models generated by the generation unit 131. For example, the model selection unit 136 selects a best model from among the multiple models generated by the generation unit 131 based on the accuracy of each model that has been trained by a learning process to which an optimization process has been applied. For example, the model selection unit 136 calculates the accuracy of each model using evaluation data, and calculates a higher evaluation value as the variation in accuracy (amount of improvement in accuracy) increases. Then, the model selection unit 136 selects the model with the highest evaluation value calculated as the best model.
[0157] Note that the model selection unit 136 may select one of the models for each combination of models having different parameters and learning data, depending on the accuracy of the model learned by the first learning unit 135. In addition, in the above example, the first learning unit 135 selects trials using the fifth optimization algorithm, but the model selection unit 136 may select trials using the fifth optimization algorithm.
[0158] (Regarding the second learning unit 137) The second learning unit 137 performs the tuning process described in steps S21 to S24 in Fig. 3, for example. Specifically, the second learning unit 137 causes the model (best model) selected by the model selection unit 136 to learn the learning data used in the optimization process. For this reason, the second learning unit 137 performs a tuning process to fine-tune the model selected by the model selection unit 136 (best model) to be more suitable for the service by modifying a part of the model (best model) selected by the model selection unit 136 and re-learning it using the learning data used in the optimization process.
[0159] (About the provider 138) The providing unit 138 processes, as a serving target, the best model whose performance has been improved to the maximum by the second learning unit 137. Specifically, the providing unit 138 provides, in response to access from a user, the best model whose performance has been improved by fine tuning according to the embodiment.
[0160] (Regarding attribute selection unit 139) When using a trained model to predict a certain target (e.g., click-through rate for advertising content), it may be possible to obtain a more accurate result by not inputting (i.e., masking) data with certain attributes (e.g., category) from the data to be input for prediction, and inputting only the remaining data, compared to inputting all the data.
[0161] For this reason, it is considered that the accuracy of the model can be improved by optimizing the data to be input to the trained model by determining which attribute of the input candidate data should not be input to the trained model. Therefore, the attribute selection unit 139 selects a target attribute, which is a target attribute of data that is not to be input, among the input candidate data to be input to a model (e.g., a best model) trained by a training unit (e.g., the first training unit 135), by determining which attribute of the data should not be input to the model. For example, the attribute selection unit 139 selects a combination of target attributes.
[0162] For example, for each candidate combination of target attributes, the attribute selection unit 139 measures the accuracy of the model when learning data having attributes excluding the target attribute for that candidate is input to the model, and selects a combination of target attributes from among the candidates based on the measurement results.
[0163] The providing unit 138 may also provide the user with information indicating attributes other than the target attribute selected by the attribute selecting unit 139. For example, the providing unit 138 provides, as information indicating attributes other than the target attribute selected by the attribute selecting unit 139, information regarding the accuracy of the model when learning data having attributes excluding the target attribute selected by the attribute selecting unit 139 is input to the model.
[0164] 7. Example of Optimization Process According to the Embodiment From here on, examples of the first optimization algorithm, the second optimization algorithm, the third optimization algorithm, the fourth optimization algorithm, and the fifth optimization algorithm, which are the optimization algorithms according to the embodiment, will be shown.
[0165] In the example of Fig. 3, the first to fifth optimization algorithms are executed consecutively in a series of learning processes, but the first to fifth optimization algorithms may be executed independently or in any combination. For example, in the learning process shown in Fig. 3, a configuration may be adopted in which only the first optimization algorithm is executed, or a configuration may be adopted in which only the second and third algorithms are executed.
[0166] [7-1-1. First optimization algorithm] In deep learning, optimal model parameters are found by repeatedly updating model parameters (e.g., weights and biases). Therefore, initial values of the model parameters are set in advance so that the model parameters can be updated, but the learning results of the neural network change depending on the initial values of the model parameters that are set. For this reason, it is considered necessary to perform optimization so that appropriate initial values are set.
[0167] For example, in deep learning, pseudorandom numbers are often used to initialize model parameters, but if the variation in the initial values is too large or too small, the learning speed will slow down and the accuracy of the model may not improve. For this reason, it is important to set the initial values of the model parameters more appropriately. The first optimization algorithm is an algorithm for optimizing the random number seed itself, which is the source of the pseudorandom numbers, so that more appropriate initial values can be generated as the initial values of the model parameters.
[0168] For this reason, the generation unit 131 uses the first optimization algorithm to optimize the random number seed that is the source of generating the initial values of the model parameters so that the initial values of the model parameters are completely random and do not vary from one another. In other words, the generation unit 131 optimizes the random number seed so that the distribution of the generated model parameters falls within a predetermined distribution.
[0169] For example, the generation unit 131 generates a plurality of random number seeds such that the initial values of the model parameters are within a predetermined range. Also, for example, the generation unit 131 generates a plurality of random number seeds such that the distribution of the initial values of the model parameters exhibits a predetermined probability distribution (for example, uniform distribution or normal distribution). Also, for example, the generation unit 131 generates a plurality of random number seeds such that the average value obtained by averaging the initial values of the model parameters is a predetermined value.
[0170] Then, for each generated random number seed, the generating unit 131 inputs the random number seed to a random function, thereby generating initial values of model parameters according to each random number seed from the output random numbers.
[0171] For example, when generating model parameters that show a uniform distribution in response to an instruction from a user, the generating unit 131 can select the initialization function "glorot_uniform" for initialization by Glorot's uniform distribution (also called Xavier's uniform distribution) as a random function (initialization function). Glorot's uniform distribution corresponds to a uniform distribution in the range [limit, -limit] when limit is sqrt(6 / (fan_in + fan_out)).
[0172] Furthermore, for example, when generating model parameters that exhibit a uniform distribution in response to an instruction from a user, the generating unit 131 can select the initialization function "he_uniform" as a random function (initialization function) for initialization with He's uniform distribution. He's uniform distribution corresponds to a uniform distribution in the range [limit, -limit] when limit is sqrt(6 / fan_in).
[0173] Then, the generating unit 131 generates initial values of the model parameters from the random numbers (pseudo-random numbers) output by inputting the generated random number seed to the selected initialization function. The distribution of the random numbers and model parameters obtained here indicates a uniform distribution.
[0174] Furthermore, the generating unit 131 generates a model having each initial value of the model parameters. Specifically, the generating unit 131 generates a model for each initial value of the model parameters. For example, the generating unit 131 generates a model having a set of model parameters with different combinations among a group of initial values of the model parameters that fall within a predetermined distribution (for example, uniform distribution, normal distribution, average value).
[0175] [7-1-2. About the fourth optimization algorithm] In order to train a model, it is important that the data is well shuffled in the shuffle buffer, but simply shuffling the data may result in bias in the training order or the data distribution for each batch, which may result in poor training. In such cases, the accuracy of the model cannot be improved.
[0176] For this reason, the first learning unit 135 uses the fourth optimization algorithm to optimize data shuffling for shuffling the learning data in the shuffle buffer.
[0177] Specifically, the first learning unit 135 optimizes a seed value used when generating a random order. For example, the first learning unit 135 uses a fourth optimization algorithm to generate a random number seed for the current learning for each epoch for repeated learning so that a bias does not occur in the random order associated with each learning data between epochs. Then, the first learning unit 135 generates a random order by inputting each generated random number seed into a random function. In addition, the first learning unit 135 generates the learning data of the final learning target in the shuffle buffer by associating the generated random order with the learning data of each learning target.
[0178] In this regard, for example, the first learning unit 135 generates multiple random number seeds for each epoch for repeated learning, such that the random order exhibits a predetermined probability distribution (e.g., uniform distribution or normal distribution) so that no bias occurs in the random order associated with each learning data between epochs.
[0179] In addition, the first learning unit 135 can optimize the data shuffle according to the current shuffle buffer size by using an optimization function for data shuffle, such as, for example, dataset = dataset.shuffle(buffer_size, seed = seed, reshuffle_each_iteration = True).
[0180] [7-1-3. An example of experimental results using the first and fourth optimization algorithms] Next, an example of the effect obtained when the first and fourth optimization algorithms are executed will be described with reference to FIGS.
[0181] Figure 7 is a diagram (1) showing the change in model performance when the first and fourth optimization algorithms are executed. Specifically, in Figure 7, a histogram is shown comparing the accuracy distribution of the model when the first and fourth optimization algorithms are executed and when they are not executed for the same model.
[0182] In the example of Fig. 7, the learning data used when the first and fourth optimization algorithms are executed and when they are not executed are the same, and the number of trials is also the same (e.g., 1000 times). The histogram shown in Fig. 7 was obtained by plotting the recall rate on the horizontal axis and the number of trials on the vertical axis.
[0183] The histogram shown in Figure 7 shows that when the first and fourth optimization algorithms were not executed, the recall rate was "0.1793" even in the best trial, whereas when the first and fourth optimization algorithms were executed, the recall rate improved to "0.1840" in the best trial. Thus, the experimental results show that the accuracy of the model is improved by executing the first and fourth algorithms. In other words, the experimental results show that the performance of the model can be improved by optimizing the computation graph and the random number seed for data shuffling.
[0184] FIG. 8 is a diagram (2) showing the change in model performance when the first and fourth optimization algorithms are executed. Specifically, FIG. 8 shows a graph comparing how the accuracy of the model changes when the first and fourth optimization algorithms are executed and when they are not executed for the same model. The graph shown in FIG. 8 was obtained by plotting the number of epochs on the horizontal axis and the average loss on the vertical axis.
[0185] The graph in Figure 8 shows that when the first and fourth optimization algorithms were not executed, the average loss was reduced to "0.008213" by repeated learning, whereas when the first and fourth optimization algorithms were executed, the average loss was further reduced to "0.008208" by repeated learning. Thus, the experimental results show that the accuracy of the model is improved by executing the first and fourth algorithms. In other words, the experimental results show that the performance of the model can be improved by optimizing the computation graph and the random number seed for data shuffling.
[0186] In addition, it was verified whether the performance of the model changes when only one of the first optimization algorithm and the fourth optimization algorithm is executed, or when the first and fourth optimization algorithms are executed in combination. Fig. 9 is a diagram showing a comparative example comparing the performance of the model according to the combination of the first and fourth optimization algorithms.
[0187] 9 shows three graphs (graph G91, graph G92, graph G93) obtained by plotting the recall rate on the horizontal axis and the number of trials on the vertical axis. In graphs G91, G92, and G93, the model, learning data, and number of trials used in the experiments are all consistent.
[0188] Graph G91 is a histogram showing the accuracy distribution of the model when only the first optimization algorithm is executed. Graph G92 is a histogram showing the accuracy distribution of the model when only the fourth optimization algorithm is executed. Graph G93 is a histogram showing the accuracy distribution of the model when the first and fourth optimization algorithms are executed.
[0189] Comparing the graphs G91 to G93, it can be seen that the accuracy distributions are almost the same in all cases. From this, according to the experimental results, it was found that there is no significant difference in the model performance when only the first optimization algorithm is executed, when only the fourth optimization algorithm is executed, and when both the first and fourth optimization algorithms are executed, and the model performance is maintained in all cases.
[0190] [7-2. About the second optimization algorithm] In deep learning, a training data set is divided into several subsets, and each subset is used for training as the epoch progresses. However, when all subsets are used for model training, the best-performing model is not necessarily trained. In addition, the more training data there is, the more problems there are in terms of time spent on training and computer resource occupation, so it is necessary to narrow down the effective subsets to be used for training and make the training more efficient. The optimization process that has been realized based on such a premise is the second optimization algorithm. Below, a more detailed example of the second optimization algorithm described above is explained with reference to FIG. 10.
[0191] Fig. 10 is a diagram showing an example of the second optimization algorithm. Note that the series of processes shown in Fig. 10 corresponds to the process in step S13 shown in Fig. 3.
[0192] First, the acquiring unit 132 acquires learning data from the learning data storage unit 121, and outputs the acquired learning data to the first data control unit 133. Upon receiving the learning data from the acquiring unit 132, the first data control unit 133 executes the following process by using a second optimization algorithm.
[0193] Here, as described in Fig. 6, the learning data has a concept of a time series. More specifically, since the learning data group is composed of a predetermined number of learning data, each learning data is associated with time information as a history, for example.
[0194] Therefore, first, the first data control unit 133 sorts the included learning data so that it is arranged in chronological order (S131). Next, the first data control unit 133 divides the learning data group in a state in which the included learning data has been sorted into a predetermined number of sets (step S132). For example, the first data control unit 133 can divide the learning data group into the predetermined number of sets so that each set equally contains a predetermined number of learning data (e.g., a number specified by a user). The first data control unit 133 may also divide the learning data group into the predetermined number of sets so that each set contains a number of learning data within a predetermined range.
[0195] Figure 10 shows an example in which the first data control unit 133 divides the learning data group to obtain “File #1”, “File #2”, “File #3”, “File #4”, “File #5”, “File #6”, “File #7”, “File #8”, “File #9”, “File #10”, and “File #11” as data files corresponding to each group.
[0196] Moreover, each of these data files contains learning data arranged in chronological order. For this reason, in the example of FIG. 10, the larger the file number of a data file, the newer the chronological sequence of the learning data contained therein. For example, when comparing one set, "File #2," with another set, "File #3," it can be said that "File #3" contains a newer chronological sequence of the learning data.
[0197] Next, the first data control unit 133 selects a predetermined number of sets to be used for model learning from among all sets obtained by the division in step S132 (step S133). For example, the first data control unit 133 randomly selects sets to be used for model learning from among all sets obtained by the division in step S132 until the number of selected sets reaches a predetermined number. In one example, the first data control unit 133 randomly selects sets from among all sets obtained by the division in step S132 until the number reaches a predetermined number (for example, a number specified by a user). Alternatively, the first data control unit 133 randomly selects sets in order from the set with the newest time series of included learning data (file #11 in the example of FIG. 10) until the number reaches a predetermined number (for example, a number specified by a user). Figure 10 shows an example in which, in the first loop, the first data control unit 133 randomly selects four pairs, namely "File #11," "File #9," "File #8," and "File #6," in a selection order (selection order according to the chronological order) in which the pairs containing the learning data are selected in order from the most recent.
[0198] Also, as described later, the process from step S133 is repeated until the specified number of loops is reached. Specifically, from among the groups obtained by the division in step S132 and not yet selected at present, a group is randomly selected until a specified number is reached, or from among the groups obtained by the division in step S132 and not yet selected at present, a group is randomly selected in order from the group containing the newest time series of learning data until a specified number is reached. This operation is repeated for each loop until the specified number of loops is reached. Therefore, for example, in the second loop, there is a possibility that "file #7", "file #5", and "file #4" are randomly selected starting with "file #10".
[0199] Next, the first data control unit 133 generates one data group by connecting the pairs selected in step S133 (step S134). For example, the first data control unit 133 generates one data group by connecting the pairs selected in step S133 in the selection order. The selection order here corresponds to the selection order in step S133, and specifically, is a selection order in which pairs to be used for model training are selected in the order of the most recent time series of included training data.
[0200] In addition, the first data control unit 133 can pass the generated data group to the second data control unit 134 so that the learning data included in the data group can be used for learning. In the example of FIG. 10, the first data control unit 133 passes "File #X", which is a data file in which the generated data group is stored, to the second data control unit 134. As shown in FIG. 10, in "File #X", the files "File #6", "File #8", "File #9", and "File #11" are arranged in the order in which they were selected. That is, in "File #X", the learning data are arranged in the order in which they were selected.
[0201] [7-3-1. About the third optimization algorithm] In deep learning, when learning a model, it is considered important to improve the accuracy of the model that the data set is appropriately batched and repeatedly learned. In addition, the order in which each subset obtained by batching the learning data set is learned is also considered to contribute to the performance of the model. The optimization process that has been realized based on such a premise is the third optimization algorithm. Below, a more detailed example of the third optimization algorithm explained so far is explained with reference to FIG. 11.
[0202] Fig. 11 is a diagram showing an example of the third optimization algorithm. Note that Fig. 11 also shows the fourth optimization algorithm. The series of processes shown in Fig. 11 corresponds to the processes from step S13 to S14 shown in Fig. 3.
[0203] For example, the second data control unit 134 optimizes the shuffle buffer size using a third optimization algorithm. For example, the second data control unit 134 generates learning data of a size equal to the size of the shuffle buffer as the optimization of the shuffle buffer size, and stores this data in the shuffle buffer as learning data for the learning target, which is the learning data used in the current iterative learning. For example, as an example of such processing, the second data control unit 134 executes the following processing following step S134 in FIG. 10.
[0204] For example, the second data control unit 134 divides a group of learning data (here, each learning data is arranged in the order of selection) grouped together as "file #X" into a predetermined number of sets (step S135). For example, the second data control unit 134 can divide the group of learning data into a predetermined number of sets so that each set contains a predetermined number of learning data (e.g., a number designated by a user). The second data control unit 134 may also divide the group of learning data into a predetermined number of sets so that each set contains a number of learning data within a predetermined range.
[0205] For example, the user can specify how to divide the learning data group included in "file #X" using various hyperparameters such as an upper limit (maxValue), a lower limit (minValue), and a minimumUnit. In other words, the user can specify the shuffle buffer size using the above-mentioned hyperparameters, etc. Therefore, the second data control unit 134 can optimize the shuffle buffer size based on the division content specified by the user. For example, the second data control unit 134 selects a shuffle buffer size according to the division content specified by the user, and divides the learning data group included in "file #X" according to the selected shuffle buffer size.
[0206] For example, it is assumed that the shuffle buffer size capable of storing "10,000" records is optimized to a shuffle buffer size corresponding to "2,500" records using the above hyperparameters, etc. In this case, the second data control unit 134 divides the 10,000 learning data groups into learning data groups of 2,500 each.
[0207] Here, an experiment has revealed that the accuracy of the model changes depending on how many pieces of learning data are divided to be included in one set, i.e., how the shuffle buffer size is set. The results of this experiment are described in FIG. 12, and for example, the third optimization algorithm may reflect these experimental results. Specifically, the second data control unit 134 may optimize the shuffle buffer size (the number of pieces of learning data to be included in one set) using the third optimization algorithm in which the experimental results shown in FIG. 12 are reflected.
[0208] Also, Fig. 11 shows an example in which the second data control unit 134 divides the learning data group contained in "file #X" to obtain four learning data groups, namely, learning data group #1 (Data #1), learning data group #2 (Data #2), learning data group #3 (Data #3), and learning data group #4 (Data #4). Also, according to the example of Fig. 11, the second data control unit 134 stores learning data group #1 in "file #X1", learning data group #2 in "file #X2", learning data group #3 in "file #X3", and learning data group #4 in "file #X4".
[0209] Also, according to the example of FIG. 11, the learning data groups are arranged from the top in the order in which the learning data groups were obtained by the division in step S135 (division order).
[0210] Next, the second data control unit 134 extracts one pair according to the division order from among the unprocessed pairs obtained by the division in step S135 and not currently used for learning, and stores the extracted pair in the shuffle buffer as learning data to be learned, which is the learning data to be used in the current iterative learning (step S136).
[0211] 11, the second data control unit 134 extracts "file #X1", which is the first set obtained by division. Then, the second data control unit 134 stores the learning data included in the extracted "file #X1" in the shuffle buffer as the learning data to be learned.
[0212] Furthermore, in response to the fact that learning data of a size (number) according to the shuffle buffer size optimized by the third optimization algorithm has been stored in the shuffle buffer as in step S136, the first learning unit 135 executes the following processing subsequent to step S136.
[0213] Specifically, the first learning unit 135 optimizes data shuffling to shuffle the learning data of the learning target stored in the shuffle buffer using the fourth optimization algorithm. Then, the first learning unit 135 trains each model on the final learning data of the learning target generated by the optimization.
[0214] For example, the first learning unit 135 generates the learning data of the final learning target by randomly determining the learning order using the fourth optimization algorithm (step S141). That is, the first learning unit 135 generates the learning data of the final learning target by determining the random order using the fourth optimization algorithm.
[0215] Specifically, the first learning unit 135 uses the fourth optimization algorithm to generate a random number seed (a seed that is the basis of the random order) for the current learning so that the random order associated with each learning data between epochs is not biased for each epoch for repeated learning. The first learning unit 135 then generates a random order by inputting each generated random number seed into a random function. The first learning unit 135 also generates the final learning data of the learning target in the shuffle buffer by associating the generated random order with the learning data of each learning target.
[0216] Next, the first learning unit 135 causes each model to learn the features of the learning data to be learned (the learning data contained in “File #X1” stored in the shuffle buffer) in the learning order (random order) generated in step S141 (step S142).
[0217] Here, the first learning unit 135 performs repeated learning for a predetermined number of epochs on the set obtained by the division in step S135, with steps S136 to S142 being one epoch. Specifically, the first learning unit 135 performs repeated learning for the number of epochs specified by the user, with steps S136 to S142 being one epoch, using the set obtained by the division in step S135.
[0218] For this reason, the first learning unit 135 first determines whether or not all of the groups obtained by the division in step S135 have been processed for one epoch (step S143). Specifically, the first learning unit 135 determines whether or not all of the groups obtained by the division in step S135 ("file #X1" to "file #X4" in the example of FIG. 11) have been used in learning with steps S136 to S142 as one epoch.
[0219] While the first learning unit 135 determines that all of the groups obtained by the division in step S135 have not been processed for one epoch (step S143; No), it repeats a series of processes from step S136 to step S142.
[0220] Furthermore, when the first learning unit 135 determines that all of the sets obtained by the division in step S135 have been processed for one epoch (step S143; Yes), it then determines whether or not the specified number of epochs has been reached for the sets obtained by the division in step S135 (step S144). Specifically, the first learning unit 135 determines whether or not repeated learning has been performed for the specified number of epochs (for example, specified by the user) using the sets obtained by the division in step S135.
[0221] While it is determined that the designated number of epochs has not been reached (step S144; No), the first learning unit 135 repeats a series of processes from step S136 to step S142.
[0222] On the other hand, when it is determined that the designated number of epochs has been reached (step S144; Yes), the model selection unit 136 selects the best model at the current time based on the accuracy of each model that has been trained at the current time (step S145). For example, the model selection unit 136 uses the evaluation data to calculate the accuracy of each model, and calculates a higher evaluation value as the variation in accuracy (amount of improvement in accuracy) increases. Then, the model selection unit 136 selects the model with the highest evaluation value calculated as the best model. Note that the method for selecting the best model is not limited to the above method. In addition, a series of processes from step S133 are repeated until the designated number of loops is reached so as to obtain a model with higher accuracy.
[0223] Therefore, the first learning unit 135 next determines whether or not the number of Loops, which is the number of times the process is repeated (looped) from step S133, has been reached (step S146). The Loop number is a hyperparameter that can be specified by the user.
[0224] Therefore, while it is determined that the designated number of loops has not been reached (step S146; No), the first learning unit 135 repeats a series of processes from step S136. This point will be described in more detail with reference to an example in FIG.
[0225] For example, when it is determined that the designated number of loops has not been reached, the first data control unit 133 performs the process of step S133, which randomly selects pairs obtained by the division in step S132 and which have not been selected until the designated number of loops is reached. Here, for example, in the process of step S133 and subsequent steps executed in the second and subsequent loops, the pair used for learning by the best model is held. Specifically, in the process of step S133 and subsequent steps executed in the second and subsequent loops, a new pair of data used for learning is added to the pair used for learning by the best model. For this reason, in the second and subsequent loops, the first data control unit 133 selects a pair of learning data to be added to the pair used for learning by the best model.
[0226] Also, as in the above example, for example, in the second Loop, “File #7”, “File #5”, and “File #4” may be selected randomly, starting with “File #10”.
[0227] Furthermore, according to the examples described above, when the designated number of loops is reached, the model selection unit 136 is able to select the model with the highest accuracy at this point in time.
[0228] [7-3-2. An example of experimental results for the third optimization algorithm] In addition, when applying the third optimization algorithm, it was verified by experiments how to effectively improve the accuracy of the model by optimizing the shuffle buffer size, that is, how to divide the data so that each set contains a certain number of learning data. Figure 12 shows a comparative example comparing the performance of the model for each shuffle buffer size.
[0229] 12 shows five graphs (graph G121, graph G122, graph G123, graph G124, and graph G125) obtained by plotting the recall rate on the horizontal axis and the number of trials on the vertical axis. In all of the graphs G121 to G125, the model, learning data, and number of trials used in the experiments are the same.
[0230] Graph G121 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "1,000K" for a certain pair including learning data. Graph G122 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "2,000K" for a similar pair. Graph G123 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "3,000K" for a similar pair. Graph G124 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "4,000K" for a similar pair. Graph G125 is a histogram showing the accuracy distribution of the model when the shuffle buffer size is set to "6,000K" for a similar pair.
[0231] Comparing graphs G121 to G125, it can be seen that the accuracy of the models is different for each. This suggests that the performance of the model can be improved by optimizing the shuffle buffer size. Therefore, it was found that the performance of the model can be improved by optimizing the shuffle buffer size by executing the third optimization algorithm. In addition, the third optimization algorithm can be said to be an idea that was conceived from the experimental results shown in FIG. 12.
[0232] The third optimization algorithm may reflect the experimental results shown in Fig. 12. Specifically, the second data control unit 134 may optimize the shuffle buffer size (the number of learning data to be included in one set) using the third optimization algorithm reflecting the experimental results shown in Fig. 12.
[0233] In this regard, in the example of FIG. 12, since the number of data records is "5,518K", it was predicted that the model would perform best when the shuffle buffer size was "6,000K", which is capable of storing all of this data. However, as shown in FIG. 12, this experiment revealed that the model performance is actually most likely to improve when the shuffle buffer size is "2,000K". Therefore, based on such experimental results, for example, the third optimization algorithm may be an algorithm that optimizes the shuffle buffer size to "2,000K". Furthermore, the third optimization algorithm may be an algorithm that optimizes the shuffle buffer size to 1 / 3 of the total size (total number) of learning data.
[0234] Furthermore, by using the example in Fig. 11, the user can appropriately consider how to divide the learning data group contained in "File #X" based on such experimental results. For example, the user can consider more optimal values for various hyperparameters such as the upper limit (maxValue), lower limit (minValue), minimumUnit, etc.
[0235] [7-4-1. About the fifth optimization algorithm] In addition, in deep learning, optimal hyperparameters are searched for by repeatedly learning a model in order to obtain the desired accuracy and generalization performance, but depending on the algorithm, amount of data, and computing environment used, one trial may take several hours. For example, in grid search, optimal parameters are selected by searching all possible ranges of hyperparameters. In such a case, as the number of types of hyperparameters increases, the number of combinations increases, which causes problems such as time and computer resource occupation. The optimization process that has been realized based on such a premise is the fifth optimization algorithm. Below, a more detailed example of the fifth optimization algorithm described above will be described with reference to FIG. 13.
[0236] 13 is a diagram showing an example of condition information related to the fifth optimization algorithm. In the learning process, trials for searching hyperparameters are repeated, and the fifth optimization algorithm is executed as trial optimization by pruning to realize efficient search. Specifically, the first learning unit 135 uses the fifth optimization algorithm to optimize trials that are not expected to produce good results by early stopping them without completing them to the end.
[0237] For example, the information processing device 100 can allow the user to set constraint conditions that condition trials that are to be subject to early stopping (targets for early termination) from the viewpoint of an evaluation value that evaluates the accuracy of the model. For example, the information processing device 100 can allow the user to set a combination of multiple such constraint conditions. FIG. 13 shows an example of a constraint condition that can be set by the user. Note that the constraint conditions shown in FIG. 13 are merely an example, and the user can set any number of combinations of any constraint conditions in the information processing device 100. Although not shown in FIG. 5, the information processing device 100 may further have a receiving unit that receives the setting of the constraint conditions.
[0238] In addition, the first learning unit 135 determines whether or not the evaluation value (evaluation value for evaluating the accuracy of the model) for the combination of hyperparameters corresponding to each trial (trials with different combinations of hyperparameters) satisfies the constraint condition, and stops the trial that is the subject of the determination when it is determined that the constraint condition is satisfied. Then, the first learning unit 135 continues only the remaining trials that were not stopped.
[0239] From here, the constraint conditions shown in Fig. 13 will be explained. Fig. 13 shows an example of a stopping condition (constraint condition) that conditions an attempt to stop (pruning) the learning process earlier than reaching all epochs. Specifically, Fig. 13 shows five stopping conditions C1 to C5.
[0240] According to the stopping condition C1, the conditions are set as follows: "function: stop_if_no_decrease_hook", "mtric_name: avarage_loss", "max_epochs_without_decrease: 3", and "min_epochs: 1". This example shows that the stopping condition C1 "conditions to stop trials where the average loss has not decreased (the accuracy has not improved) for a maximum of three epochs".
[0241] Also, according to the stopping condition C2, the conditions are set as "function: stop_if_no_decrease_hook", "mtric_name: auc", "max_epochs_without_increase: 3", and "min_epochs: 1". This example shows that the stopping condition C2 "conditions to stop trials where the auc has not increased (the accuracy has not improved) for a maximum of three epochs".
[0242] Moreover, according to the stopping condition C3, the conditions are set as "function: stop_if_lower_hook", "mtric_name: accuracy", "threshold: 0.8", and "min_epochs: 3". This example shows that the stopping condition C3 "conditions to stop trials in which the accuracy does not exceed the threshold of 0.8 after 3 epochs".
[0243] Also, according to the stop condition C4, the conditions are set as "function: stop_if_higher_hook", "mtric_name: loss", "threshold: 300", and "min_epochs: 5". This example shows that the stop condition C4 "conditions to stop trials in which the loss exceeds the threshold of 300 after the 5th epoch".
[0244] Also, according to the stopping condition C5, the condition is set with contents such as "function: stop_if_not_in_top_k_hook", "mtric_name: auc", "top_k: 10", and "epochs: 3". This example shows that the stopping condition C5 "conditions to stop trials where the auc is not in the top 10 at the time of the 3rd epoch".
[0245] [7-4-2. An example of experimental results when using the fifth optimization method] Next, an example of a process in which a trial is stopped using the fifth optimization algorithm will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the fifth optimization algorithm. In addition, the example of Fig. 14 shows a scene in which the fifth optimization algorithm is applied in a state in which the stopping conditions C6 and C7 are combined.
[0246] According to the stopping condition C6, the condition is set as "function: stop_if_not_in_top_k_hook", "mtric_name: recall", "top_k: 8", and "epochs: 3". This example shows that the stopping condition C6 "conditions to stop trials whose recall is not in the top 8 at the time of the 3rd epoch".
[0247] According to the stop condition C7, the condition is set with contents such as "function: stop_if_not_in_top_k_hook", "mtric_name: recall", "top_k: 4", and "epochs: 6". This example shows that the stop condition C7 "conditions to stop trials whose recall is not in the top four at the time of the 6th epoch".
[0248] FIG. 14 also shows an example in which individual trials with different combinations of hyperparameters are processed in parallel using a predetermined number (e.g., 16) of devices, and the first learning unit 135 monitors, for each trial, the fluctuation in recall, which is the evaluation value (evaluation value that evaluates the accuracy of the model) for the combination of hyperparameters corresponding to that trial, and determines whether the aspect based on the fluctuation in recall (in the example of FIG. 14, the ranking of the trial) satisfies the stopping conditions C6 and C7.
[0249] In this state, the first learning unit 135 stops trials whose recall is not in the top 8 at the time point of the 3rd epoch based on the stopping condition C6. Also, the first learning unit 135 stops trials whose recall is not in the top 4 at the time point of the 6th epoch.
[0250] In this way, when the fifth optimization algorithm is used to optimize trials that are not expected to improve the model's performance, the experimental results show that the processing time is improved by 45%. Specifically, the fifth optimization algorithm, which combines multiple stopping conditions that condition trials that are not expected to improve the model's performance and stops the trials early, shows that the processing time is improved by 45%. In addition, the fifth optimization algorithm can solve problems such as time and computer resource occupation.
[0251] Furthermore, a user may be required to set effective stop conditions so that computer resources can be used efficiently. For this reason, the information processing device 100 may provide information to assist the user in considering what stop conditions should be set. For example, the information processing device 100 provides a screen displaying the current optimization status for each trial so that the user can visually confirm the optimization status. For example, the information processing device 100 can deliver a screen displaying the current optimization status for each trial to the terminal device 3 in response to access from the terminal device 3 owned by the user.
[0252] According to such an information processing device 100, it becomes possible to easily visually recognize trials that are not expected to improve the performance of the model, and therefore it becomes possible to consider effective stopping conditions that should be set in order to stop trials that are not expected to improve the performance of the model early.
[0253] The screen displaying the optimization status may be provided by, for example, the providing unit 138 or by another processing unit.
[0254] [7-5-1. Optimization of mask targets] So far, the first to fifth optimization algorithms have been shown as algorithms for optimizing the learning method. In addition to these optimizations, the information processing device 100 may optimize mask target data to determine which data among the input candidate data to be input to the trained model should not be input to the model. Specifically, the information processing device 100 uses an algorithm for optimizing the mask target to select non-input target data that should not be input to the model from among the input candidate data to be input to the trained model.
[0255] For example, when predicting a certain target using a trained model, a more accurate result may be obtained by not inputting (i.e., masking) data having a specific attribute (e.g., category) from among the data to be input for prediction and inputting only the remaining data, compared to inputting all of the data. In other words, the accuracy of the trained model may be improved by not inputting (i.e., masking) data having a specific attribute (e.g., category) and inputting only the remaining data, rather than inputting all of the data.
[0256] According to this, it is considered necessary to optimize the data to be input to the trained model by determining which attribute of the input candidate data should not be input to the trained model. The optimization process that has been realized based on this premise is the mask target optimization algorithm.
[0257] For example, the attribute selection unit 139 uses an optimization algorithm for the mask target to select a target attribute, which is an attribute that is a target of non-input target data, among input candidate data to be input to the trained model, and determines which attribute data has, and which attribute is not to be input to the model. For example, for each candidate combination of target attributes, the attribute selection unit 139 measures the accuracy of the model when learning data having attributes other than the target attribute of the candidate is input to the model, and selects a combination of target attributes from the candidates according to the measurement result.
[0258] Here, for example, in the case of predicting a certain target (e.g., click-through rate of an advertisement) using the best model selected by the model selection unit 136, it is hypothesized that a better prediction result will be obtained by treating data having a specific attribute among the test data used for the prediction as non-input target data and inputting only the remaining test data excluding the non-input target data into the best model than by inputting all of the test data.
[0259] In Fig. 15, an example of optimizing the mask target is described using experimental results in which the effect of the optimization algorithm for the mask target based on the hypothesis is verified. Fig. 15 is a diagram showing an example of an optimization algorithm for optimizing the mask target.
[0260] Here, the learning data (or evaluation data) used in the optimization process has a plurality of attributes. For example, the learning data is classified into various categories such as learning data on "business", learning data on "economy", learning data on "gender", learning data on "user interests", etc. Therefore, the learning data has attributes as such categories, for example.
[0261] Therefore, for example, for each category combination that is valid for the categories contained in the learning data, the attribute selection unit 139 measures the accuracy (recall) of the model when learning data included in categories other than the category in the combination is input to the best model. Then, depending on the measurement result, for example, based on which combination of categories was excluded when the highest accuracy was obtained, the attribute selection unit 139 selects a target category (target attribute) that is a category (attribute) that is not to be input to the best model and is the target of data that is not to be input, among the test data (from FIG. 6) paired with this learning data.
[0262] For this reason, the attribute selection unit 139 automatically searches for a combination of categories (attributes) that improves the performance of the model when masked. For example, the attribute selection unit 139 can use a genetic algorithm to search for a combination of categories (attributes) that improves the performance of the model when masked.
[0263] In Fig. 15, the recall rate in each search (trial) performed by the attribute selection unit 139 is plotted. Also, Fig. 15 shows an example of a combination of attributes when the highest accuracy is obtained. For convenience of explanation, such a combination of categories is defined as "combination CB".
[0264] Then, when the highest accuracy is obtained, the attribute selection unit 139 has excluded combination CB from among the category combinations, and therefore determines that data included in the category in combination CB is not to be input to the best model as non-input target data. In other words, by selecting combination CB from among the category combinations as a target attribute, the attribute selection unit 139 determines that data included in the category in combination CB is to be masked when inputting test data to the best model.
[0265] Furthermore, the providing unit 138 can provide the best model and information indicating a category other than the category selected by the attribute selecting unit 139. The information indicating a category other than the category selected by the attribute selecting unit 139 is, for example, information regarding the accuracy of the best model when learning data included in a category other than the category selected by the attribute selecting unit 139 is input to the best model, and may be, for example, the recall shown in FIG.
[0266] In addition, by providing such information according to the optimization of the mask target, for example, when a user wants to predict a target using the best model, the user can know that, instead of inputting all the data of the test data prepared by the user, data having a specific attribute should be masked and only the remaining data should be input. As a result, the user can obtain a more accurate prediction result than when all the test data is used. In addition, the information processing device 100 having an optimization function for optimizing the mask target can help the user obtain a more accurate result using a trained model.
[0267] [7-5-2. An example of experimental results when using mask target optimization] As described above, when the optimization of the mask target is performed, a part of the test data is not input, so the number of test data actually input is smaller than the initial number when the optimization of the mask target is not performed. Therefore, an experiment was performed to verify whether the accuracy of the model is affected by the reduction in the number of test data input due to the optimization of the mask target. Figure 16 is a diagram showing a comparative example comparing the accuracy of the model when the optimization of the mask target is performed and when the optimization of the mask target is not performed.
[0268] In Fig. 16, the evaluation result (recall rate) of the model evaluated using the evaluation data used in learning is compared with the evaluation result (recall rate) of the model evaluated using the remaining data as test data, excluding data having attributes selected by optimizing the mask target for the evaluation data. According to the comparative example shown in Fig. 16, it was found from the experiment that the versatility of the model is maintained even when the optimization of the mask target is performed.
[0269] In the above example, the information processing device 100 determines which attribute of data among the input candidate data to be input to the trained model is not to be input to the trained model, and controls the data having the determined attribute to be masked and only data having attributes other than the determined attribute to be used. However, instead of controlling the data to be input to the trained model to be masked in this way, the information processing device 100 may control the learning to be performed using optimization of the mask target during the learning using the fifth optimization algorithm described above.
[0270] Specifically, the information processing device 100 further includes a determination unit that determines a plurality of new combinations of target attributes based on combinations of target attributes in a plurality of models whose accuracy satisfies a predetermined condition, and determines whether the accuracy of each model when learning data having attributes excluding the target attributes in the determined combination is input to the plurality of models satisfies the predetermined condition. Then, the first learning unit 135 trains the learning data for the model determined by the determination unit to satisfy the predetermined condition. Note that the first learning unit 135 may perform the processing of this determination unit.
[0271] 8. Configuration of the Execution Control Device Up to now, the description has focused on the information processing device 100 having the first to fifth optimization algorithms and an optimizer OP function that is a function for performing the optimization algorithms of the mask target. From here, the execution control device 200 will be described. First, the background that led to the realization of the execution control device 200 is as follows.
[0272] For example, in the case where a certain object is predicted using a trained model, a computer predicts whether or not any image data is the same as the correct image data using the trained model. Such prediction processing includes multiple processes such as a process of extracting features from an image, i.e., a two-dimensional array of pixels, a process of detecting a portion with matching features from another image, and the like.
[0273] Each process included in the prediction process is executed by a processor of a computer, but the processing time required for the entire prediction process varies depending on which device among the devices constituting the processor performs which process.
[0274] For this reason, in order to further reduce the processing time required for the entire prediction process, it is important to optimize the execution entity of the processes so that the optimal device (computing device) for executing each process included in the prediction process is assigned. However, it is impossible for a computer to dynamically determine the optimal execution entity.
[0275] Based on such a premise, the execution control device 200 performs a process of optimizing an execution subject that executes a process using a model (for example, a process of predicting a specific target). Specifically, the execution control device 200 optimizes the execution subject by determining an execution subject that executes a process using the model (for example, a process of predicting a specific target) based on the characteristics of the trained model. Therefore, the execution control device 200 has an optimization algorithm for the execution subject.
[0276] First, the execution control device 200 according to the embodiment will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of the configuration of the execution control device 200 according to the embodiment. As shown in Fig. 17, the execution control device 200 has a communication unit 210, a storage unit 220, and a control unit 230.
[0277] (Regarding the storage unit 220) The storage unit 220 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 includes a model architecture storage unit 221.
[0278] (About the model architecture storage unit 221) The model architecture storage unit 221 stores the architecture of a neural network. An example of the model architecture storage unit 221 according to the embodiment is shown in Fig. 18. In the example of Fig. 18, the model architecture storage unit 221 has items such as "model ID" and "architecture information".
[0279] "Model ID" indicates identification information for identifying a model. "Architecture information" is information indicating the characteristics of a model identified by "Model ID". Specifically, "Architecture information" is information indicating the overall structure including the learning mechanism of the model identified by "Model ID".
[0280] The example in Fig. 18 shows an example in which a model ID "MD#1" is associated with architecture information "Architecture#1". This example shows an example in which the architecture of a model identified by the model ID "MD#1" is "Architecture#1". Note that in Fig. 18, the architecture of a neural network is conceptually shown as "Architecture#1", but in reality, legitimate information indicating the architecture of a neural network is registered.
[0281] (Regarding the control unit 230) The control unit 230 is realized by a CPU, an MPU, or the like executing various programs stored in a storage device inside the execution control device 200 using a RAM as a working area. The control unit 130 is also realized by an integrated circuit such as an ASIC or an FPGA.
[0282] As shown in Fig. 17, control unit 230 has identification unit 231, decision unit 232, and execution control unit 233, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 230 is not limited to the configuration shown in Fig. 17, and may be other configurations as long as they perform information processing described below. Also, the connection relationship between each processing unit in control unit 230 is not limited to the connection relationship shown in Fig. 17, and may be other connection relationships.
[0283] (Regarding the specific part 231) The identification unit 231 identifies features of a model (a trained model) used when a plurality of arithmetic devices each having a different architecture execute a predetermined process (e.g., a process such as estimation using a model). For example, the identification unit 231 identifies features of a plurality of processes executed as the model as the model features.
[0284] (Regarding the determination unit 232) The determination unit 232 determines which of the multiple arithmetic devices is to execute a process using the model, based on the characteristics of the model identified by the identification unit 231. For example, the determination unit 232 determines which of the multiple arithmetic devices is to execute the process, for each of the multiple processes, based on the characteristics of the multiple processes identified by the identification unit 231.
[0285] For example, the determination unit 232 determines the target arithmetic device from among a plurality of arithmetic devices: a first arithmetic device that is guaranteed to output the same value when the same processing is executed using the same data, and a second arithmetic device that is not guaranteed to output the same value when the same processing is executed using the same data.
[0286] Also, for example, the determination unit 232 determines the target arithmetic unit from among the plurality of arithmetic units, a first arithmetic unit that performs scalar arithmetic operations, and a second arithmetic unit that performs vector arithmetic operations.
[0287] Also, for example, the determination unit 232 determines the target arithmetic device for execution from among a first arithmetic device that adopts an out-of-order method and a second arithmetic device that does not adopt an out-of-order method as a plurality of arithmetic devices.
[0288] That is, the determination unit 232 determines the target arithmetic device from among a central processing unit (CPU) having a branch prediction function as the first arithmetic device, and an image processing unit (GPU) not having a branch prediction function as the second arithmetic device. For example, when the model is a model for multi-class classification, the determination unit 232 determines the image processing unit as the target arithmetic device. On the other hand, when the model is a model for two-class classification, the determination unit 232 determines the central processing unit as the target arithmetic device.
[0289] (Regarding the executive control unit 233) The execution control unit 233 causes the arithmetic device determined by the determination unit 232 to execute processing using the model.
[0290] [9-1. An example of the operation of the execution control device] Hereinafter, an example of the processing performed by the execution control device 200 using the execution-subject optimization algorithm will be described.
[0291] For example, suppose that a user wishes to operate a model whose performance has been improved by the above-described fine tuning using the information processing device 100 in a production environment (for example, a server or an edge device). Specifically, suppose that the user wishes to operate a model whose performance has been improved by the fine tuning using the information processing device 100 in a server corresponding to a predetermined service.
[0292] In the following, we will explain the case where the model in question (e.g., the best model) is model MD1 (model identified by model ID "MD♯1"), which is a model for multi-class classification (pattern PT1), and the case where the model is model MD2 (model identified by model ID "MD♯2"), which is a model for two-class classification (pattern PT2).
[0293] Note that both the process using the model MD1 and the process using the model MD2 are prediction processes for predicting a predetermined target. According to the above example, the prediction process using the model MD1 and the prediction process using the model MD2 are performed by a server (e.g., an API server) corresponding to the user's production environment.
[0294] (Regarding pattern PT1) The identification unit 231 refers to the model architecture storage unit 221 using the model ID "MD♯1" and identifies the architecture of the neural network corresponding to the model MD1. In the architecture, a target arithmetic device that executes a plurality of processes (for example, a process of extracting features from an image and a process of detecting a portion with matching features from another image) executed as a model is specified. For example, in the architecture, only one of a GPU or a CPU is specified as a target arithmetic device that executes a plurality of processes executed as a model. For this reason, the identification unit 231 identifies an architecture that indicates each process included in, for example, a prediction process, from among the architectures of the neural network corresponding to the model MD1.
[0295] Furthermore, the determination unit 232 determines which of the GPU and CPU is to be used to execute the process, based on the architecture for each process identified by the identification unit 231. For example, if the architecture corresponding to process A1, which is one process identified by the identification unit 231, specifies that the process A1 is to be executed by a GPU, the determination unit 232 determines the GPU as the target calculation device for executing the process A1. For example, if the architecture corresponding to process A2, which is another process identified by the identification unit 231, specifies that the process A2 is to be executed by a CPU, the determination unit 232 determines the CPU as the target calculation device for executing the process A2.
[0296] In this state, for example, the execution control unit 233 controls the user's API server to have the GPU execute process A1 and to have the CPU execute process A2.
[0297] (Regarding pattern PT2) The identification unit 231 refers to the model architecture storage unit 221 using the model ID "MD♯2" and identifies the architecture of the neural network corresponding to the model MD2. Similarly, for the architecture, a target arithmetic device for executing the process is specified for each of a plurality of processes executed as a model (for example, a process for extracting features from an image and a process for detecting a portion with matching features from another image). That is, in the architecture, only one of a GPU or a CPU is specified as a target arithmetic device for executing the process for each of a plurality of processes executed as a model. For this reason, the identification unit 231 identifies an architecture indicating each process included in, for example, a prediction process, from among the architectures of the neural network corresponding to the model MD2.
[0298] Furthermore, the determination unit 232 determines which of the GPU and CPU is to be used to execute the process, based on the architecture for each process identified by the identification unit 231. For example, if the architecture corresponding to process B1, which is one process identified by the identification unit 231, specifies that the process is to be executed by a CPU, the determination unit 232 determines the CPU as the target calculation device for executing process B1. For example, if the architecture corresponding to process B2, which is another process identified by the identification unit 231, specifies that the process B2 is to be executed by a GPU, the determination unit 232 determines the GPU as the target calculation device for executing process B2.
[0299] The process of the determination unit 232 will be described in more detail with reference to FIG. 19. FIG. 19 is a diagram showing an example of a model architecture associated with information indicating a target arithmetic device. In FIG. 19, an architecture corresponding to the process A1 is shown among the architectures of the neural network corresponding to the model MD1. As shown in FIG. 19, among the architectures of the neural network corresponding to the model MD1, the architecture corresponding to the process A1 has information indicating a target arithmetic device that executes the process A1 incorporated therein in advance. Specifically, in the example of FIG. 19, a description that specifies that the process A1 is to be executed by a GPU is associated in advance with the architecture corresponding to the process A1. Therefore, the determination unit 232 can determine a GPU as a target arithmetic device that executes the process A1 based on such a description.
[0300] In order for the execution control device 200 to operate as described above using the execution subject optimization algorithm, it is necessary that, for each architecture associated with each process using the model among the neural network architectures corresponding to the trained model, information indicating the target arithmetic device that is to execute the process is pre-integrated. In other words, for each process, the target arithmetic device that is to execute the process needs to be given in a rule base.
[0301] Therefore, in order to realize such a rule base, experiments were conducted to verify the difference in processing time when processing using a multi-class classification model was performed on a GPU and a CPU, respectively. Also, experiments were conducted to verify the difference in processing time when processing using a two-class classification model was performed on a GPU and a CPU, respectively.
[0302] [9-2. An example of experimental results regarding the execution subject's optimization algorithm] From here, an example of the effect when the GPU and the CPU execute the processes using the model will be described with reference to Figs. 20 to 24.
[0303] (Model for multi-class classification) First, an example of the effect when processing using a multi-class classification model is executed by both the GPU and the CPU will be described with reference to Figures 20 and 21. Here, for each multi-class classification model for a given service, an experiment was conducted to see how much performance (processing time) could be improved by having the GPU execute processing for each combination of arbitrary combinations of processing that was initially performed on the CPU side. Figure 20 shows the results of this experiment.
[0304] Fig. 20 is a diagram showing the performance improvement status by an experiment targeting a model for multi-class classification. For example, Fig. 20 shows each element when the best result was obtained from the above experiment.
[0305] In the example of Figure 20, for a model (model "1") corresponding to service SV1, an experiment was conducted to see how much performance (processing time) could be improved by having the GPU execute the processes for each combination of arbitrary combinations of processes that were initially performed on the CPU side. As shown in Figure 20, it was found that by having the GPU execute some of the processes that were initially performed on the CPU side, performance improved by up to "30.8%" (processing speed was reduced by "30.8%) before and after optimization. It is also shown that the GPU usage rate changed from "28%" to "38%" before and after optimization.
[0306] In the example of Figure 20, for a model (model "2") corresponding to service SV2, an experiment was conducted to see how much performance (processing time) could be improved by having the GPU execute the processes for each combination of processes that were originally performed on the CPU side. As shown in Figure 20, it was found that by having the GPU execute some of the processes that were originally performed on the CPU side, performance improved by up to "44.2%" before and after optimization (processing speed was reduced by "44.2%). It is also shown that the GPU usage rate changed from "15%" to "42%" before and after optimization.
[0307] In the example of Figure 20, for a model (model "3") corresponding to service SV3, an experiment was conducted to see how much performance (processing time) could be improved by having the GPU execute the processes for each combination of processes that were originally performed on the CPU side. As shown in Figure 20, it was found that by having the GPU execute some of the processes that were originally performed on the CPU side, performance improved by up to "12.3%" (processing speed was reduced by "12.3%) before and after optimization. It is also shown that the GPU usage rate changed from "15%" to "18%" before and after optimization.
[0308] In the example of Figure 20, for a model (model "4") that supports service SV4, an experiment was conducted to see how much performance (processing time) could be improved by having the GPU execute the processes for each combination of processes that were originally performed on the CPU side. As shown in Figure 20, it was found that by having the GPU execute some of the processes that were originally performed on the CPU side, performance improved by up to "65.1%" (processing speed was reduced by "65.1%) before and after optimization. It is also shown that the GPU usage rate changed from "54%" to "56%" before and after optimization.
[0309] Also, as shown in Figure 20, for a model (model "5") that supports service SV5, an experiment was conducted to see how much performance (processing time) could be improved by having the GPU execute the processes for each combination of processes that were originally performed on the CPU side. As shown in Figure 20, it was found that by having the GPU execute some of the processes that were originally performed on the CPU side, performance improved by up to "39.1%" (processing speed was reduced by "39.1%) before and after optimization. It is also shown that the GPU usage rate changed from "39%" to "45%" before and after optimization.
[0310] In addition, the above experimental results show that for multi-class classification models, even if the models differ depending on the service, performance can always be improved by shifting some of the processing that was initially performed on the CPU to the GPU, and that performance can be improved by an average of 38.8%.
[0311] In addition, according to the experimental results shown in Fig. 20, it is believed that the best optimization can be achieved by making the architecture rule-based by incorporating information indicating the computing device "GPU" into the architecture associated with the processing that was executed by the GPU when the maximum performance was obtained, among the neural network architectures corresponding to the multi-class classification model.
[0312] Next, an example of the experiment contents will be shown, focusing on an experiment conducted for a model (model "1") corresponding to service SV1 among the experiments conducted for each model corresponding to each service shown in Fig. 20. Fig. 21 is a diagram showing an example of the experiment contents of an experiment conducted for a model corresponding to service SV1. Fig. 21 shows the experiment contents when the performance was improved by a maximum of "30.8%".
[0313] The example in Figure 21 shows an experiment in which, of any combination of processes that were originally performed on the CPU side, processes A11, A12, and A13 were forcibly moved to the GPU side so that these processes were performed on the GPU side.
[0314] In this way, in the model corresponding to service SV1, which is a model for multi-class classification, by incorporating information indicating a computing device "GPU" into the architecture associated with processes A11, A12, and A13, the execution control device 200 can have a higher performance optimization algorithm. As a result, for example, it becomes possible to effectively improve the performance of a user's computer (for example, a server or edge device) used to operate the model corresponding to service SV1 in a production environment.
[0315] (Model for two-class classification) Next, an example of the effect when processing using a two-class classification model is executed by the CPU and the GPU, respectively, will be described with reference to Fig. 22 and Fig. 23. Here, an experiment was conducted to see how much performance (processing time) could be improved by having the CPU execute specific processing that was initially performed by the GPU for each two-class classification model for a given service. Fig. 22 shows the results of this experiment.
[0316] Fig. 22 is a diagram showing the performance improvement status by an experiment targeting a two-class classification model. For example, Fig. 22 shows each element when the best result was obtained from the above experiment.
[0317] In the example of Figure 22, for a model (model "6") that supports service SV6, an experiment was conducted to see how much performance (processing time) could be improved by shifting specific processing that was initially performed on the GPU side to the CPU side. As shown in Figure 22, it was found that by shifting specific processing that was initially performed on the GPU side to the CPU side, performance improved by up to "50.3%" (processing speed was reduced by "50.3%") before and after optimization.
[0318] In addition, in the example of Figure 22, for a model (model "7") that supports service SV7, an experiment was conducted to see how much performance (processing time) could be improved by shifting specific processing that was initially performed on the GPU side to the CPU side. As shown in Figure 22, it was found that by shifting specific processing that was initially performed on the GPU side to the CPU side, performance improved by up to "30.2%" (processing speed was reduced by "30.2%) before and after optimization.
[0319] In addition, according to the above experimental results, we found that for two-class classification models, even if the models differ depending on the service, performance can always be improved by shifting specific processing that was initially performed on the GPU to the CPU. We also found that parallel calculation by the CPU is effective for many processes using two-class classification models.
[0320] In addition, according to the experimental results shown in Figure 22, it is believed that the best optimization can be achieved by making the architecture of the neural network corresponding to the two-class classification model, which is linked to the processing that was being executed by the CPU when the maximum performance was obtained, into a rule-based architecture by incorporating information indicating the calculation unit "CPU."
[0321] Next, an example of the experiment contents will be shown, focusing on an experiment conducted for a model (model "6") corresponding to service SV6 among the experiments conducted for each model corresponding to each service shown in Fig. 22. Fig. 23 is a diagram showing an example of the experiment contents of an experiment conducted for a model corresponding to service SV6. Fig. 23 shows the experiment contents when the performance was improved by a maximum of "50.3%".
[0322] FIG. 23 shows an example of an experiment in which a process that was originally performed on the GPU side but required a MATMUL operation was executed on the CPU side.
[0323] In this way, in a model corresponding to service SV6, which is a two-class classification model, by incorporating information indicating a calculation unit "CPU" into an architecture associated with a process requiring MATMUL calculation, the execution control device 200 can have a higher performance optimization algorithm. As a result, for example, it becomes possible to effectively improve the performance of a user's computer (for example, a server or edge device) used to operate a model corresponding to service SV6 in a production environment.
[0324] Furthermore, regardless of whether the model is compatible with Service SV6, by incorporating information indicating the calculation unit "CPU" into the architecture of the two-class classification model that is associated with processing that requires MATMUL calculations and making it rule-based, it can be said that the performance of the user's computer (e.g., server or edge device) can be effectively improved.
[0325] 10. Processing flow of information processing device So far, there has been described the algorithms of the optimization processes performed by the information processing device 100 and the execution control device 200. Next, a procedure of the process executed by the information processing device 100 will be described. Specifically, a procedure of the information processing device 100 performing a series of tuning processes (fine tuning according to the embodiment) including the first optimization process to the fifth optimization process will be described.
[0326] Fig. 24 is a flowchart showing an example of a flow of fine tuning according to the embodiment. Fig. 24 shows a part of the fine tuning according to the embodiment that is executed by an optimization function (optimizer OP) of the information processing device 100.
[0327] First, the generation unit 131 performs steps S2401 and S2402 using an algorithm (first optimization algorithm) that optimizes a random number seed used in generating a model (computation graph).
[0328] Specifically, the generation unit 131 generates multiple random number seeds for the computation graph (step S2401). For example, the generation unit 131 generates multiple random number seeds optimized so that the initial values of the weights show a uniform distribution. The generation unit 131 also generates initial values of the weights corresponding to the respective generated random number seeds (step S2402). For example, the generation unit 131 inputs the generated random number seeds to a random function, and generates weights corresponding to the respective pseudo-random numbers from the multiple pseudo-random numbers that are uniformly distributed and are obtained as outputs. The initial values of the weights obtained in this manner also show a uniform distribution.
[0329] Then, the generating unit 131 generates a plurality of models according to the initial values generated in step S2402 (step S2403). Note that, although the weight is shown as an example of the model parameter in the example of Fig. 24, the model parameter may be, for example, a weight or a bias. In this case, the generating unit 131 may generate a model having each set of model parameters (for example, a set of a weight and a bias) that are different in combination from among the initial value group of the model parameters generated in step S2402.
[0330] Next, the first data control unit 133 performs the following steps S2404 to S2406 by an algorithm (second optimization algorithm) for optimizing the learning data used in learning the model.
[0331] Specifically, the first data control unit 133 divides the learning data group, which is sorted so that the included learning data is arranged in chronological order, into a predetermined number of sets (step S2404). Then, the first data control unit 133 selects a set of learning data to be used for learning each model generated in step S2403 from the sets obtained by the division in step S2404 (step S2405). For example, the first data control unit 133 randomly selects a set to be used for model learning from all the sets obtained by the division in step S2404 until the number of selected sets reaches a predetermined number. For example, the first data control unit 133 randomly selects a set from the sets obtained by the division in step S2404 that are currently unselected until the specified number of loops is reached. Furthermore, the first data control unit 133 may randomly select pairs from among the pairs obtained by the division in step S2404 that have not yet been selected until the specified number of loops is reached, starting from the pair that contains the most recent time series of learning data, until a predetermined number (for example, a number specified by the user) is selected.
[0332] Then, the first data control unit 133 generates one learning data group by connecting the sets of learning data selected in step S2405 (step S2406). For example, the first data control unit 133 generates one learning data group by connecting the sets selected in step S2405 in the order of selection at this time.
[0333] Next, the second data control unit 134 performs the following steps S2407 and S2408 using an algorithm (third optimization algorithm) for optimizing the shuffle buffer size.
[0334] Specifically, the second data control unit 134 divides the learning data group generated by the first data control unit 133 in step S2406 (step S2407). For example, the second data control unit 134 divides the learning data group generated by the first data control unit 133 as a process of generating learning data of a size equal to the size of the shuffle buffer. For example, the second data control unit 134 can divide the learning data group generated by the first data control unit 133 into a predetermined number of sets so that each set after division equally contains a predetermined number of learning data (for example, a number specified by a user).
[0335] Then, the second data control unit 134 extracts one set according to the order (division order) obtained by the division at this time from among the sets obtained by the division at step S2407, and stores the learning data included in the extracted set in the shuffle buffer as the learning data to be learned (step S2408). For example, the second data control unit 134 extracts one set according to the division order from among the sets obtained by the division at step S2407 and unprocessed sets not currently used for learning. Then, the second data control unit 134 stores the extracted one set in the shuffle buffer as the learning data to be learned, which is the learning data to be used in the current repeated learning.
[0336] Next, the first learning unit 135 performs the following steps S2409 to S2411 using an algorithm (fourth optimization algorithm) that optimizes a random number seed (random number seed for data shuffling) when shuffling and determining the learning order when sequentially learning the learning data in the shuffle buffer.
[0337] Specifically, the first learning unit 135 generates a random number seed for a random order, which is the learning order of the learning data in the shuffle buffer (step S2409). For example, the first learning unit 135 generates a random number seed (a seed that is the basis of the random order) for the current learning for each epoch for repeated learning so that the random order associated with each learning data between epochs is not biased.
[0338] The first learning unit 135 also generates a random order according to each of the random number seeds generated in step S2409 (step S2410). For example, the first learning unit 135 generates the random order by inputting each random number seed into a random function. Then, the first learning unit 135 generates the learning data to be the final learning target in the shuffle buffer by associating the generated random order with the learning data in the shuffle buffer (step S2411).
[0339] The first learning unit 135 also causes each model to learn the features of the learning data of the final learning target in the learning order indicated by the random order determined in step S2410 (step S2412). In addition, during this learning, trials to search for hyperparameters are repeated, and in order to realize an efficient search, the first learning unit 135 executes a fifth optimization as an optimization of trials by pruning, thereby terminating trials that are not expected to produce good results early without continuing to the end.
[0340] Moreover, the first learning unit 135 performs repeated learning for the specified number of epochs on the set obtained by the division in step S2407, with steps S2408 to S2412 being one epoch. Specifically, the first learning unit 135 performs repeated learning for the user-specified number of epochs on the set obtained by the division in step S2407, with steps S2408 to S2412 being one epoch.
[0341] Therefore, next, the first learning unit 135 judges whether or not all of the pairs obtained by the third optimization (specifically, the pairs obtained by the division in step S2407) have been processed for one epoch (step S2413). Specifically, the first learning unit 135 judges whether or not all of the pairs obtained by the division in step S2407 have been used for learning with steps S2408 to S2412 as one epoch. While the first learning unit 135 judges that all of the pairs obtained by the division in step S2407 have not been processed for one epoch (step S2413; No), it repeats a series of processes from step S2408 to step S2412 until it can be judged that all of the pairs have been processed for one epoch.
[0342] On the other hand, when the first learning unit 135 determines that all of the sets obtained by the division in step S2407 have been processed for one epoch (step S2413; Yes), it then determines whether or not the specified number of epochs has been reached for the sets obtained by the division in step S2407 (step S2414). Specifically, the first learning unit 135 determines whether or not repeated learning has been performed for the specified number of epochs using the sets obtained by the division in step S2407.
[0343] While the first learning unit 135 determines that the designated number of epochs has not been reached (step S2414; No), it repeats the series of processes from step S2408 onwards until it can determine that the designated number of epochs has been reached.
[0344] On the other hand, when it is determined that the designated number of epochs has been reached (step S2414; Yes), the model selection unit 136 selects the best model at the present time based on the accuracy of each trained model at the present time (step S2415). As described in Fig. 11, a series of processes from step S2408 onwards are repeated until the designated number of loops is reached so as to obtain a model with higher accuracy.
[0345] Therefore, next, the first learning unit 135 determines whether or not the number of loops, which is the number of times designated to repeat (loop) the series of processes from step S2408 onwards, has been reached (step S2416). While the first learning unit 135 determines that the designated number of loops has not been reached (step S2416; No), it repeats the series of processes from step S2408 onwards. On the other hand, if the first learning unit 135 determines that the designated number of loops has been reached (step S2416; Yes), it ends the processing at this point.
[0346] Furthermore, at this point when the process is completed, the best model selected by the model selection unit 136 may be the one with the highest accuracy among the models selected for each loop.
[0347] Further, the second learning unit 137 corresponds to a selector function (selector SE) of the information processing device 100 in the fine tuning according to the embodiment, and subsequently performs, for example, the tuning process described in steps S21 to S24 in Fig. 3, although this is not shown in Fig. 24. Specifically, the second learning unit 137 performs this tuning process on the best model selected by the model selection unit 136.
[0348] [11. An example of experimental results regarding fine tuning] Next, an example of the effect when fine tuning according to this embodiment is performed will be described with reference to FIGS. 25A to 25C.
[0349] Fig. 25A is a diagram showing a comparative example (1) in which the accuracy of a model is compared between a case where fine tuning according to the embodiment is performed and a case where fine tuning according to the embodiment is not performed. Specifically, Fig. 25A shows a comparative example in which an evaluation result corresponding to trial A in the case where fine tuning is performed is compared with an evaluation result corresponding to trial A in the case where fine tuning is not performed.
[0350] In association with the example of FIG. 4, in the example of FIG. 25A, data from "June 16th, 17:32" to "June 17th, 7:26" in the dataset was used as evaluation data, and the accuracy of the best model was evaluated. In addition, in the example of FIG. 25A, data from "June 17th, 7:26" to "June 19th, 0:00" in the dataset was used as test data with unknown labels, and the accuracy of the best model was evaluated. And, according to the example of FIG. 25A, the evaluation result by such evaluation shows that the accuracy of the best model is improved by "4.5%" by performing fine tuning according to the embodiment.
[0351] Fig. 25B is a diagram showing a comparative example (2) in which the accuracy of the model is compared between a case where the fine tuning according to the embodiment is performed and a case where the fine tuning according to the embodiment is not performed. Specifically, Fig. 25B shows a comparative example in which an evaluation result corresponding to trial B in the case where the fine tuning is performed is compared with an evaluation result corresponding to trial B in the case where the fine tuning is not performed.
[0352] In association with the example of FIG. 4, in the example of FIG. 25B, data from "June 17th, 7:26" to "June 17th, 12:00" in the dataset was used as evaluation data, and the accuracy of the best model was evaluated. In addition, in the example of FIG. 25B, data from "June 17th, 12:00" to "June 19th, 0:00" in the dataset was used as test data with unknown labels, and the accuracy of the best model was evaluated. According to the example of FIG. 25B, the evaluation result of the evaluation shows that the accuracy of the best model is improved by "9.0%" by performing fine tuning according to the embodiment.
[0353] Fig. 25C is a diagram showing a comparative example (3) in which the accuracy of the model is compared between a case where the fine tuning according to the embodiment is performed and a case where the fine tuning according to the embodiment is not performed. Specifically, Fig. 25C shows a comparative example in which an evaluation result corresponding to trial C in the case where the fine tuning is performed is compared with an evaluation result corresponding to trial C in the case where the fine tuning is not performed.
[0354] In the example of Fig. 25C corresponding to the example of Fig. 4, data from "June 17th 12:00" to "June 19th 0:00" in the dataset was used as evaluation data to evaluate the accuracy of the best model. According to the example of Fig. 25C, the evaluation result of the evaluation showed that the accuracy of the best model was improved by "10.2%" by performing the fine tuning according to the embodiment.
[0355] In addition, according to the examples of Figures 25A to 25C, in a data set corresponding to a time series, a time range is determined as learning data, a time range is determined as evaluation data, and a time range is determined as evaluation data with unknown labels. By appropriately changing these time ranges, the effects of fine-tuning were verified from various angles.
[0356] 25A to 25B, it was found that, regardless of how the dataset is used according to the purpose, the performance improvement can be maintained by performing the fine tuning according to the embodiment, compared to the case where the fine tuning according to the embodiment is not performed. In addition, it was demonstrated that the accuracy of the model can be improved according to the information processing device 100 according to the embodiment.
[0357] [12. Others] In addition, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.
[0358] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.
[0359] Furthermore, the above-described embodiments can be appropriately combined as long as the processing contents are not contradictory.
[0360] [13. Program] Moreover, the information processing device 100 and the execution control device 200 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in Fig. 26, for example. Fig. 26 is a hardware configuration diagram showing an example of the computer 1000. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.
[0361] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.
[0362] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via the communication network 50 and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the communication network 50.
[0363] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.
[0364] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
[0365] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the function of the control unit 130. In addition, the HDD 1400 stores data in the storage unit 120.
[0366] Furthermore, for example, when the computer 1000 functions as the execution control device 200 according to the embodiment, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the function of the control unit 230. Furthermore, the HDD 1400 stores data in the storage unit 220.
[0367] The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be obtained from another device via the communication network 50.
[0368] 14. Effects (Effect (1) of one aspect of the information processing device 100 according to the embodiment) As described above, the information processing device 100 according to the embodiment (an example of a learning device) includes the generation unit 131, the first learning unit 135, the model selection unit 136, and the second learning unit 137. The generation unit 131 generates a plurality of models each having different parameters. The first learning unit 135 causes each of the plurality of models generated by the generation unit 131 to learn features possessed by a portion of predetermined learning data. The model selection unit 136 selects one of the models according to the accuracy of the model learned by the first learning unit 135. The second learning unit 137 causes the model selected by the model selection unit 136 to learn features possessed by the predetermined learning data.
[0369] According to such an information processing device 100, it is possible to provide a user with a model with improved accuracy and performance, thereby effectively supporting the user in putting the model into practical use for a specific service.
[0370] In addition, the generation unit 131 generates multiple input values to be input to a predetermined first function that calculates a random number value based on the input value, and generates multiple models for each generated input value having parameters corresponding to the random number value output by the predetermined first function when the input value is input.
[0371] According to such an information processing device 100, the accuracy of the model can be improved.
[0372] Furthermore, the generating unit 131 generates a plurality of input values, as input values to be input to a predetermined first function, such that the random number value output by the predetermined first function satisfies a predetermined condition.
[0373] According to such information processing device 100, it is possible to control the variation in the initial values of the model parameters, and therefore it is possible to improve the accuracy of the model.
[0374] Furthermore, the generating unit 131 generates a plurality of input values such that the random number value falls within a predetermined range.
[0375] According to such information processing device 100, the variation in the initial values of the model parameters can be controlled so as to exhibit a uniform distribution, and therefore the accuracy of the model can be improved.
[0376] Moreover, the generating unit 131 generates a plurality of input values such that the distribution of the random numbers exhibits a predetermined probability distribution.
[0377] According to such information processing device 100, the variation in the initial values of the model parameters can be controlled so as to exhibit a uniform distribution, and therefore the accuracy of the model can be improved.
[0378] Furthermore, the generating unit 131 generates a plurality of input values whose average value of the random numbers is a predetermined value.
[0379] According to such information processing device 100, the variation in the initial values of the model parameters can be controlled so as to exhibit a uniform distribution, and therefore the accuracy of the model can be improved.
[0380] In addition, the generation unit 131 selects, as a predetermined first function, a function such that the distribution of random number values output when an input value is input exhibits a predetermined probability distribution, and generates a plurality of models having parameters according to the random number values output by the selected function.
[0381] According to such information processing device 100, the variation in the initial values of the model parameters can be controlled so as to exhibit a uniform distribution, and therefore the accuracy of the model can be improved.
[0382] In addition, the first learning unit 135 (an example of a selection unit) selects, from the models that have been trained, multiple models whose evaluation values for evaluating accuracy satisfy predetermined conditions, and causes the multiple selected models to learn characteristics possessed by a portion of predetermined training data.
[0383] According to such an information processing device 100, among the trials searching for hyperparameters, those that satisfy a stopping condition defined using the evaluation value of the model can be terminated early, while those that do not satisfy the stopping condition (multiple models whose evaluation value for evaluating accuracy satisfies a predetermined condition) can be continued, thereby solving problems related to time and computer resource occupation, and also improving the accuracy of the model by pruning early trials that are not expected to produce good results.
[0384] Moreover, the first learning unit 135 selects a plurality of models whose appearance based on the change in evaluation value while repeatedly learning the features of a portion of the predetermined training data a predetermined number of times satisfies a predetermined appearance.
[0385] According to such an information processing device 100, while applying each trial with a different combination of hyperparameters to a model and repeating learning, trials that satisfy a stopping condition can be terminated early, while trials that do not satisfy the stopping condition (multiple models whose evaluation value for evaluating accuracy satisfies a predetermined condition) can be continued, thereby solving problems related to time and computer resource occupation, and also improving the accuracy of the model by pruning trials that are not expected to produce good results early.
[0386] In addition, the first learning unit 135 selects a model that satisfies a plurality of conditions designated by the user as the predetermined conditions.
[0387] According to such an information processing device 100, by combining a plurality of stopping conditions that condition trials that are unlikely to improve the performance of the model to be stopped early, and that are defined using the evaluation value of the model, it is possible to improve the accuracy of the model compared to the case where a general early stopping algorithm is used.
[0388] In addition, the first learning unit 135 may generate a plurality of input values to be input to a predetermined second function that calculates a random value based on an input value, and generate a portion of the predetermined learning data based on the random value output by the predetermined second function when the input value is input for each generated input value. In this manner, the first learning unit 135 may be an example of a learning data generating unit.
[0389] Furthermore, according to such an information processing device 100, it is possible to solve the problem of bias in the learning order in which the model is trained with the learning data, which results in poor learning, and therefore it is possible to improve the accuracy of the model.
[0390] In addition, the first learning unit 135 generates multiple input values to be input to a predetermined second function for each repeated learning, thereby generating learning data to be learned in the learning, and learns a model using this learning data generated for the learning for each repeated learning.
[0391] According to such information processing device 100, for each epoch for repeated learning, the learning order for the current epoch can be determined so that the learning order associated with each piece of learning data between epochs is not biased.
[0392] Moreover, the first learning unit 135 generates, as a part of the predetermined learning data, learning data in which the random numbers are associated as learning orders.
[0393] According to such an information processing device 100, for example, an optimized learning order can be associated with each piece of learning data in the shuffle buffer, thereby solving the problem of bias in the learning order in which the learning data is used to learn the model, resulting in poor learning.
[0394] Furthermore, the model selection unit 136 selects one of the models according to the accuracy of the model trained by the first training unit 135 for each combination of models having different parameters and predetermined training data.
[0395] According to such information processing device 100, it becomes possible to select a model with improved performance as the best model from among models with different parameters and provide the model to the user.
[0396] (Effect (2) of one aspect of the information processing device 100 according to the embodiment) As described above, the information processing device 100 (an example of a learning device) according to the embodiment has the second data control unit 134. The second data control unit 134 divides predetermined learning data, the features of which are to be learned by a model, into a plurality of groups in chronological order, and controls the first learning unit 135 to learn the features of the learning data included in each divided group in a predetermined order into the model. For this reason, the second data control unit 134 is a processing unit corresponding to an example of a dividing unit and a learning unit.
[0397] According to such information processing device 100, the shuffle buffer size can be optimized based on the fact that the accuracy of the model changes depending on the shuffle buffer size, and the learning data can be divided in accordance with the optimized shuffle buffer size, thereby improving the accuracy of the model.
[0398] Furthermore, the second data control unit 134 performs control so that, for each divided group, the features of the learning data included in that group are learned by the model in a random order.
[0399] According to such an information processing device 100, the accuracy of the model can be improved.
[0400] Furthermore, the second data control unit 134 performs control so that, among the divided groups, the features of the learning data included in the group are learned by the model in order starting from the group corresponding to the time series.
[0401] According to such an information processing device 100, by learning from older learning data in a time series to newer learning data in a time series, the tendency of the characteristics of the learning data can be calculated with high accuracy, and the accuracy of the model can be improved.
[0402] In addition, the second data control unit 134 divides the predetermined learning data into groups each having a number of learning data designated by the user.
[0403] According to such information processing device 100, a user who has verified how the accuracy of a model changes depending on the shuffle buffer size can divide training data based on the results of this verification, thereby improving usability in optimizing the shuffle buffer size.
[0404] In addition, the second data control unit 134 divides the specified learning data into a plurality of sets so that the number of learning data pieces included in each set into which the specified learning data is divided falls within a range specified by the user.
[0405] According to such information processing device 100, when a user finds it difficult to specify an appropriate number, the user can specify a rough range, thereby improving usability in optimizing the shuffle buffer size.
[0406] (Effect (3) of one aspect of the information processing device 100 according to the embodiment) As described above, the information processing device 100 according to the embodiment (an example of a learning device) has the first data control unit 133. The first data control unit 133 divides predetermined learning data, from which a model learns features, into a plurality of groups in chronological order, and selects a group to be used for model learning from the divided groups. The first data control unit 133 also controls the first learning unit 135 to learn features of the learning data included in each group into the model by using the groups in order of age of the included learning data from the selected groups. For this reason, the first data control unit 133 is a processing unit corresponding to an example of a dividing unit, a selecting unit, and a learning unit.
[0407] According to such information processing device 100, it is possible to optimize the learning data that is actually used for learning, among the data set, and therefore it is possible to improve the accuracy of the model.
[0408] Moreover, the first data control unit 133 divides the predetermined learning data into groups each having a predetermined number of learning data.
[0409] According to such an information processing device 100, the data set can be divided so that each group obtained by the division contains a predetermined number of pieces of learning data, and therefore it becomes possible to optimize each group containing the learning data actually used for learning.
[0410] Furthermore, the first data control unit 133 randomly selects, from the divided sets, a set to be used for model training.
[0411] According to such information processing device 100, it is possible to fairly select, from among the groups obtained by division, which group will contain the learning data to be actually used for learning.
[0412] Furthermore, the first data control unit 133 selects, from among the divided groups, a group that includes newer time series learning data.
[0413] According to such an information processing device 100, it is possible to control so that features of more recent learning data are learned, thereby improving the accuracy of the model.
[0414] Furthermore, the first data control unit 133 selects a number of sets designated by the user from among the divided sets.
[0415] According to such an information processing device 100, it is possible to improve the usability when dividing a data set.
[0416] In addition, the first data control unit 133 selects, from among the divided sets, in chronological order, sets that contain newer chronologically ordered learning data, until the number of selected sets reaches the number specified by the user.
[0417] According to such information processing device 100, it is possible to learn the features of the learning data designated by the user so as to improve the accuracy of the model to the maximum extent possible.
[0418] (Effect (4) of one aspect of the information processing device 100 according to the embodiment) As described above, the information processing device 100 (an example of a classification device) according to the embodiment includes the first learning unit 135 (or the second learning unit 137), the attribute selection unit 139, and the provision unit 138. The first learning unit 135 causes a model to learn features of learning data having a plurality of attributes. The attribute selection unit 139 selects a target attribute, which is an attribute that is a target of data that is not to be input, or is a target attribute of data that is not to be input, among the input candidate data to be input to the model learned by the first learning unit 135. The provision unit 138 provides the model and information indicating attributes other than the target attribute selected by the attribute selection unit 139.
[0419] According to such an information processing device 100, when a user wants to use a trained model, the user does not need to input all the test data prepared by the user, but can learn that data having a specific attribute should be masked and only the remaining data should be input. As a result, the user can obtain a more correct output result than when all the test data is used. Therefore, the information processing device 100 can support the user in obtaining a more correct result by using the trained model.
[0420] Furthermore, the attribute selection unit 139 selects a combination of target attributes.
[0421] Such an information processing device 100 can measure the accuracy of the model for all possible combinations of target attributes and can compare the accuracy of the model between combinations, making it possible to determine with high accuracy what combinations of learning data should not be input to the model in order to achieve the highest accuracy.
[0422] In addition, for each candidate combination of target attributes, the attribute selection unit 139 measures the accuracy of the model when learning data having attributes excluding the target attribute for that candidate is input to the model, and selects a combination of target attributes from among the candidates based on the measurement results.
[0423] Such an information processing device 100 makes it possible to compare the accuracy of the model between possible combinations of target attributes, and therefore makes it possible to determine with high accuracy which combinations of learning data should not be input to the model in order to obtain the highest accuracy.
[0424] The first learning unit 135 also determines multiple new combinations of target attributes based on combinations of target attributes in multiple models whose model accuracy satisfies a predetermined condition, and determines whether the accuracy of each model satisfies the predetermined condition when learning data having attributes excluding the target attributes in the determined combinations is input to the multiple models. Then, the first learning unit 135 trains the learning data for the models determined to satisfy the predetermined condition.
[0425] According to such an information processing device 100, multiple models whose evaluation value for evaluating accuracy satisfies a predetermined condition are selected, and when the selected multiple models are made to learn the characteristics of a portion of the training data, it is possible to control so that training data that may degrade the performance of the model is not learned, thereby improving the accuracy of the model.
[0426] In addition, the providing unit 138 provides information indicating attributes other than the target attributes selected by the attribute selection unit 139, that is, information regarding the accuracy of the model when learning data having attributes other than the target attributes selected by the attribute selection unit 139 is input to the model.
[0427] According to such an information processing device 100, it is possible to support a user in obtaining a more accurate result by using a trained model.
[0428] (Effect (5) of one aspect of the information processing device 100 according to the embodiment) As described above, the execution control device 200 according to the embodiment includes the identification unit 231, the determination unit 232, and the execution control unit 233. The identification unit 231 identifies the characteristics of a model used when a plurality of arithmetic units having different architectures execute a predetermined process. The determination unit 232 determines which of the plurality of arithmetic units is to execute the process using the model, based on the characteristics of the model identified by the identification unit 231. The execution control unit 233 causes the arithmetic unit determined by the determination unit 232 to execute the process using the model.
[0429] According to such an information processing device 100, it is possible to optimize the target arithmetic device based on the characteristics of the model so that each process using the model is executed by an appropriate arithmetic device. Also, according to such an information processing device 100, it is possible to further reduce the processing time spent on the process using the model. Also, according to such an information processing device 100, it is possible to indirectly improve the accuracy of the model from the viewpoint of the computer on which the user intends to execute the process using the model.
[0430] Furthermore, the identification unit 231 identifies the characteristics of a plurality of processes executed as a model as the characteristics of the model, and the determination unit 232 determines, for each of the plurality of processes, which of the plurality of arithmetic devices is to be used to execute the process, based on the characteristics of the plurality of processes identified by the identification unit 231.
[0431] According to such an information processing device 100, each of the multiple processes executed as a model can be executed by a calculation device that is more skilled in that process, thereby making it possible to further reduce the processing time spent on processing using the model.
[0432] Furthermore, the determination unit 232 determines the target arithmetic device from among a plurality of arithmetic devices: a first arithmetic device that is guaranteed to output the same value when the same processing is executed using the same data, and a second arithmetic device that is not guaranteed to output the same value when the same processing is executed using the same data.
[0433] According to such an information processing device 100, the accuracy of the model can be improved.
[0434] Furthermore, the determination unit 232 determines, as the plurality of arithmetic units, an execution target arithmetic unit from among a first arithmetic unit that performs scalar arithmetic operations and a second arithmetic unit that performs vector arithmetic operations.
[0435] According to such an information processing device 100, among the multiple processes executed as a model, the processes requiring scalar operations can be executed by a first arithmetic device, and the processes requiring vector operations can be executed by a second arithmetic device, thereby making it possible to further reduce the processing time spent on processing using the model.
[0436] Furthermore, the determination unit 232 determines the target arithmetic device for execution from among the multiple arithmetic devices: a first arithmetic device that employs the out-of-order method, and a second arithmetic device that does not employ the out-of-order method.
[0437] According to such an information processing device 100, the accuracy of the model can be improved.
[0438] The determination unit 232 determines the target arithmetic unit from among a central processing unit having a branch prediction function as the first arithmetic unit, and an image arithmetic unit not having a branch prediction function as the second arithmetic unit.
[0439] According to such an information processing device 100, among multiple processes executed as a model, the CPU can be assigned to processes for which the CPU excels, and the GPU can be assigned to processes for which the GPU excels, thereby making it possible to further reduce the processing time spent on processes using the model.
[0440] Furthermore, when the model is a model for multi-class classification, the determination unit 232 determines the image processing device as the processing device to be executed.
[0441] According to such information processing device 100, it is possible to further reduce the processing time required for processing using a model.
[0442] Furthermore, when the model is a two-class classification model, the determination unit 232 determines the central processing unit as the processing unit to be executed.
[0443] According to such information processing device 100, it is possible to further reduce the processing time required for processing using a model.
[0444] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.
[0445] Moreover, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, a generating section can be read as a generating means or a generating circuit. [Explanation of symbols]
[0446] 1 Information provision system 2. Model Generation Server 3 Terminal Equipment 10 Information provision device Sy Information Processing System 100 Information processing device 120 Storage section 121 Learning data storage unit 122 Model Memory Section 130 Control section 131 Generation part 132 Acquisition Department 133 First data control section 134 Second data control section 135 First Study Section 136 Model Selection Section 137 Second Learning Section 138 Provision Department 139 Attribute Selection Section 200 Execution control device 220 Storage section 221 Model Architecture Memory Unit 230 Control Unit 231 Specific section 232 Decision Section 233 Execution Control Unit
Claims
1. A storage unit that stores architecture information of a neural network in association with each of a plurality of models, the architecture information defining, for each of a plurality of processes to be executed as the model, a target arithmetic device that is to execute the process; A determination unit that determines a target arithmetic device for each of a plurality of processes to be executed as the target model based on architecture information of the neural network corresponding to the target model; an execution control unit that causes the computing device determined by the determination unit to execute a corresponding process among a plurality of processes that are executed as the model of the target of use; having The target arithmetic unit includes a central arithmetic unit and an image arithmetic unit, The plurality of models includes a model for multi-class classification and a model for two-class classification.
2. An execution control device comprising:
2. The target arithmetic units further include a first arithmetic unit that performs scalar operations and a second arithmetic unit that performs vector operations.
2. The execution control device according to claim 1 .
3. The execution target arithmetic units further include a third arithmetic unit in which the out-of-order method is adopted, and a fourth arithmetic unit in which the out-of-order method is not adopted.
3. The execution control device according to claim 1 or 2.
4. The target arithmetic unit includes a central arithmetic unit having a branch prediction function as the central arithmetic unit, and an image arithmetic unit not having a branch prediction function as the image arithmetic unit.
4. The execution control device according to claim 1, wherein the execution control device is a processor.
5. When the model to be used is a model for multi-class classification, the determination unit determines the image processing device as a processing device to be executed for some of a plurality of processes executed as the model.
5. The execution control device according to claim 4.
6. When the model to be used is a model for two-class classification, the determination unit determines the central processing unit as a processing unit to be executed for a specific process among a plurality of processes to be executed as the model.
6. The execution control device according to claim 5,
7. An execution control method executed by an execution control device, comprising: a storage step of storing architecture information of a neural network in association with each of a plurality of models, the architecture information defining, for each of a plurality of processes to be executed as the model, a target arithmetic device that is to execute the process; determining a target arithmetic device for each of a plurality of processes to be executed as the target model based on architecture information of the neural network corresponding to the target model; an execution control step of causing the arithmetic device determined by the determination step to execute a corresponding process among a plurality of processes executed as the model of the target of use; Including, The target arithmetic unit includes a central arithmetic unit and an image arithmetic unit, The plurality of models includes a model for multi-class classification and a model for two-class classification.
23. An execution control method comprising:
8. A storage procedure for storing architecture information of a neural network in association with each of a plurality of models, the architecture information defining, for each of a plurality of processes to be executed as the model, a target arithmetic device for executing the process; a determination step of determining a target arithmetic device for each of a plurality of processes to be executed as the target model based on architecture information of the neural network corresponding to the target model; an execution control program for causing a computer to execute an execution control procedure for causing a computing device determined by the determination procedure to execute a corresponding process among a plurality of processes executed as a model of the target of use, The target arithmetic unit includes a central arithmetic unit and an image arithmetic unit, The plurality of models includes a model for multi-class classification and a model for two-class classification.
2. An execution control program comprising:
Citation Information
Patent Citations
Data processing device
JP2013186770A
Dynamic adaptation of deep neural networks
JP2019164793A
Prefetch for systems with heterogeneous architectures
US20100153934A1
Electronic device and operating method thereof of processing neural network model by using plurality of processors
US20200065671A1
Scheduling heterogenous execution on heterogeneous hardware
US20200218969A1
