Generation device, method, and program

Abstract

An acquisition unit (29) acquires, for each of a plurality of learning models that have performed machine learning using a plurality of types of feature quantities, the accuracy of the result of the prediction by the learning model and the degree of contribution of at least one type of feature quantity specified by a user to the prediction result; and a generation unit (30) generates, from the accuracy and the degree of contribution, an index for selecting a prescribed learning model from among the plurality of learning models.

Description

[Technical field]

[0001] The disclosed technology relates to a generation device, a generation method, and a generation program. [Background technology]

[0002] Machine learning technology that automatically learns patterns of input data and creates learning models that predict classification of video and audio, trends in time-series data, etc., is becoming widespread. In addition, learning models can be created in various patterns by setting items related to the behavior of the learning model, such as the base structure, parameters, and the input data shape. Therefore, when using machine learning technology, it is necessary to select a learning model of an appropriate pattern so that the prediction accuracy meets a predetermined standard.

[0003] As a technology for selecting a learning model, a technology has been proposed in which various learning models that predict the same objective variable but have different structures or features used are generated, and the generated learning models are listed in descending order of prediction accuracy (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Satoshi Hara, Masakazu Ishibata, Takanori Maehara, "Enumeration of Machine Learning Models", Artificial Intelligence Basic Issues Research Group, 105:P.1-5, 2018. Summary of the Invention [Problem to be solved by the invention]

[0005] In machine learning technology, not only the prediction accuracy of the learning model but also the basis of the prediction can be important. For example, in tasks that support human decision-making, such as diagnosing a disease based on the prediction results of a learning model, the reliability of the prediction result is judged not only by the prediction result of the learning model but also by looking at the basis of the prediction. Therefore, it is necessary to select an appropriate learning model so that not only the prediction accuracy but also the basis of the prediction meets a specified standard.

[0006] In conventional technology, learning models are listed based only on the accuracy of prediction results estimated based on predetermined indicators, so when it is desired to also place importance on the basis of the prediction, as described above, there is a problem in that the learning models are not necessarily listed in the most optimal order.

[0007] The disclosed technology has been made in consideration of the above points, and aims to generate an index for appropriately selecting a learning model that can achieve the desired prediction accuracy and prediction basis. [Means for solving the problem]

[0008] A first aspect of the present disclosure is a generation device that generates an index for selecting a specific learning model from among multiple learning models that have been machine-learned using multiple types of features, and is configured to include an acquisition unit that acquires, for each of the multiple learning models, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature specified by a user to the prediction result, and a generation unit that generates the index from the accuracy and the contribution.

[0009] A second aspect of the present disclosure is a method for generating an index for selecting a specific learning model from among multiple learning models machine-learned using multiple types of features, in which an acquisition unit acquires, for each of the multiple learning models, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature specified by a user to the prediction result, and the generation unit generates the index from the accuracy and the contribution.

[0010] A third aspect of the present disclosure is a generation program for generating an index for selecting a specific learning model from among multiple learning models machine-learned using multiple types of features, the program causing a computer to function as an acquisition unit that acquires, for each of the multiple learning models, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature specified by a user to the prediction result, and a generation unit that generates the index from the accuracy and the contribution. Effect of the Invention

[0011] According to the disclosed technology, it is possible to generate an index for appropriately selecting a learning model that can obtain the desired prediction accuracy and prediction basis. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining an overview of a first embodiment. [Diagram 2] FIG. 11 is a schematic diagram for explaining an example of a SHAP value. [Diagram 3] FIG. 2 is a block diagram showing a hardware configuration of the learning model selection device. [Figure 4] 1 is a block diagram showing an example of a functional configuration of a learning model selection device according to a first embodiment. [Diagram 5] FIG. 11 is a sequence diagram showing the flow of a learning model selection process. [Figure 6] FIG. 11 is a block diagram showing an example of the functional configuration of a learning model selection device according to a second embodiment. [Figure 7] FIG. 11 is a diagram showing an example of material data. [Figure 8] FIG. 13 is a diagram illustrating an example of a learning pattern set. [Figure 9] FIG. 13 is a diagram illustrating an example of a column created as a response variable of learning data. [Figure 10] FIG. 13 is a diagram illustrating an example of a sequence column created as an explanatory variable of learning data. [Figure 11] 11 is a diagram for explaining calculation of a contribution degree for each feature identifier; FIG. [Figure 12] FIG. 13 is a diagram illustrating an example of a contribution weight vector and an overall contribution weight. [Figure 13] FIG. 11 is a diagram illustrating an example of a calculation result by an acquisition unit. [Figure 14] 11A and 11B are diagrams illustrating an example of an acquisition result by an acquisition unit and a generation result by a generation unit. [Figure 15] 13 is a flowchart showing an example of an evaluation dataset creation process. [Figure 16] 13 is a flowchart illustrating an example of a learning model evaluation process. [Figure 17] FIG. 13 is a diagram for explaining differences in model evaluation functions due to differences in overall contribution weights. [Figure 18] FIG. 13 is a diagram for explaining adjustment of overall contribution weights. [Figure 19] FIG. 13 is a diagram showing an example of a prediction result when a learning pattern that maximizes the model evaluation function is selected when W2=1.0. [Figure 20] FIG. 13 is a diagram showing an example of a prediction result when a learning pattern that maximizes a model evaluation function is selected when W2=1.5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0014] First Embodiment Fig. 1 is a diagram for explaining an overview of the first embodiment. As shown in Fig. 1, input data is input to each of a plurality of learning models (learning models A and B in the example of Fig. 1), a prediction process is performed in each learning model, and output data that is a prediction result is output. The prediction accuracy of the learning model is obtained by comparing the output data that is the prediction result with the correct answer of the output data.

[0015] Conventionally, the learning model is evaluated using this prediction accuracy. On the other hand, as described above, when it is desired to evaluate the learning model by placing importance not only on the prediction accuracy but also on the grounds for the prediction, it is possible to use the contribution of the desired feature to obtain the prediction result by the learning model. The desired feature may be selected based on prior knowledge. For example, when trying to predict the probability of developing lung cancer using a learning model, since it is known that the probability of developing lung cancer is related to the amount of cigarettes consumed, the amount of cigarettes consumed or a value extracted from the amount of cigarettes consumed may be used as the desired feature.

[0016] In light of the above, in the first embodiment, in addition to the prediction accuracy, the contribution of the feature is also used to generate an index for selecting a learning model. The contribution of the feature is a value indicating the contribution of each type of feature used in the learning model to the prediction result. In other words, the greater the influence of a certain feature value on the prediction result, the greater the contribution of the feature. Here, the type of feature is information that distinguishes each column present in the learning data used for machine learning of the learning model. In other words, the types of feature are different between different columns. However, in the second embodiment described later, time series data is used as the learning data, and the columns of the learning data also include columns in which the time series of a certain column is shifted. In this case, the types of feature are the same in the original column and the column in which the time series is shifted. In addition, even in the case of a column in which the time series is shifted, if the feature of the original column is the objective variable and the feature of the column in which the time series is shifted is the explanatory variable, the original column and the column in which the time series is shifted are treated as having different types of feature.

[0017] An example of the degree of contribution is the SHAP (SHapley Additive exPlanations, Reference 1) value.

[0018] Reference 1: Scott M. Lundberg and Su-In Lee, "A Unified Approach to Interpreting Model Predictions", Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.

[0019] FIG. 2 shows an example of a schematic diagram of the SHAP value. In the example of FIG. 2, the horizontal axis represents the magnitude of the SHAP value, with the contribution increasing in the positive direction toward the right and the contribution increasing in the negative direction toward the left. In addition, in the example of FIG. 2, the SHAP value for each data of each feature (each point in FIG. 2) is shown in a histogram format for each type of feature. Note that in the example of FIG. 2, even if the type of feature is the same, features whose time series are shifted are shown separately. Also, the darker the color of each point, the larger the value of the feature indicated by that point.

[0020] In the first embodiment, an index for selecting a learning model is generated so that a learning model having a high contribution of a desired type of feature among the contributions of each type of feature, such as the above-mentioned SHAP value, is selected. The type of the desired feature is specified by a user. Then, using the generated index, a learning model is selected so that the prediction accuracy and the contribution of the specified type of feature satisfy a predetermined standard. Hereinafter, a learning model selection device including a function for generating an index for selecting a predetermined learning model from among multiple learning models machine-learned using multiple types of feature, as described above, will be described.

[0021] Fig. 3 is a block diagram showing a hardware configuration of a learning model selection device 10 according to the first embodiment. As shown in Fig. 3, the learning model selection device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0022] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In the first embodiment, the ROM 12 or the storage 14 stores a learning model selection program for executing a learning model selection process described later.

[0023] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of storage devices such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system and various data.

[0024] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0025] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0026] The communication I / F 17 is an interface for communicating with other devices external to the learning model selection device 10, and uses, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0027] Next, the functional configuration of the learning model selection device 10 will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the learning model selection device 10. As shown in FIG. 4, the learning model selection device 10 includes, as functional configurations, a material data collection unit 21, a material data storage unit 22, a learning pattern transmission unit 23, an evaluation data set creation unit 24, and an evaluation data set storage unit 27. The evaluation data set creation unit 24 further includes a verification data creation unit 25 and a learning model creation unit 26. The learning model selection device 10 also includes a learning model evaluation unit 28 and a selected learning model storage unit 32. The learning model evaluation unit 28 further includes an acquisition unit 29, a generation unit 30, and a selection unit 31. The acquisition unit 29 and the generation unit 30 are examples of the generation device of the disclosed technology. Each functional configuration is realized by the CPU 11 reading out a learning model selection program including a generation program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0028] The material data collection unit 21 collects material data that is used to construct a learning dataset to be used in creating a learning model. The material data is data that can be used as a target variable and an explanatory variable, and has a unique data identifier for distinguishing each piece of data. The material data collection unit 21 accepts an input of a data identifier designated by a user, and collects material data that has the accepted data identifier. The material data collection unit 21 may collect, for example, a sensor value output from a sensor, data stored in an external or internal storage device, and the like, as material data. The material data collection unit 21 stores the collected material data in the material data storage unit 22.

[0029] The learning pattern transmission unit 23 receives an input of a learning pattern set designated by a user, and transmits the received learning pattern set to the evaluation data set creation unit 24 .

[0030] The behavior of output data of a learning model changes depending on differences in the base structure, parameters, input data shape, etc. Here, a "learning pattern set" refers to a collection of multiple learning patterns, which are settings of information that affect the behavior of output data of a learning model, that are created in advance by a user. An index is assigned to each learning pattern to identify it. In addition, components of each learning pattern include, for example, a base model identifier, hyperparameters, and a learning data construction method.

[0031] The base model identifier is an identifier that identifies a model structure that is the base of the learning model, and is, for example, the API (Application Programming Interface) name of the learning model. The hyperparameter is a parameter related to the machine learning method that corresponds to the base model identifier. The value of each hyperparameter may be uniquely determined, or there may be multiple candidates. The learning data construction method is a method of processing material data and constructing learning data including explanatory variables and objective variables to be input to the learning model. As the learning data construction method, multiple items are set, such as feature identifiers that are identifiers indicating the type of feature for each of multiple feature amounts that constitute the learning data, and a calculation method for the material data to obtain the feature amount for each feature identifier. As described above, when time series data is used as the learning data and a column of the learning data includes a column in which the time series of a certain column is shifted, the type of feature amount in the original column is the same as that in the column in which the time series is shifted. Therefore, the same feature identifier is used for the feature amount of the original column and the feature amount of the column in which the time series is shifted. However, when the feature quantities of the original column are the objective variables and the feature quantities of the time-shifted column are the explanatory variables, different feature identifiers are used for the feature quantities of the original column and the feature quantities of the time-shifted column.

[0032] The evaluation dataset creation unit 24 uses the validation data created by the validation data creation unit 25 and the learning model created by the learning model creation unit 26 as validation data. The evaluation dataset creation unit 24 collects the validation data for the number of learning patterns, i.e., the number of indexes included in the learning pattern set, as an evaluation dataset and stores it in the evaluation dataset storage unit 27.

[0033] The verification data creation unit 25 acquires material data from the material data storage unit 22, and creates learning data from the material data according to the learning data construction method included in the learning pattern. The verification data creation unit 25 also extracts a part of the learning data as verification data, and outputs the remaining learning data to the learning model creation unit 26. The verification data is data used to verify the created learning model, and is data that is not used for machine learning of the learning model.

[0034] The learning model creation unit 26 creates a learning model using the learning data output from the verification data creation unit 25, in accordance with the base model identifier and the hyperparameters included in the learning pattern.

[0035] The learning model evaluation unit 28 evaluates each of the multiple learning models stored in the evaluation dataset storage unit 27 based on the indexes generated by the acquisition unit 29 and the generation unit 30, and selects a desired learning model by the selection unit 31. Then, the learning model evaluation unit 28 stores the learning model selected by the selection unit 31 in the selected learning model storage unit 32.

[0036] The acquiring unit 29 acquires the prediction accuracy of each learning model and the contribution of at least one type of feature quantity specified by the user to the prediction result by the learning model. Specifically, the acquiring unit 29 acquires the prediction accuracy based on the error between the output data of the learning model when the explanatory variables included in the validation data are input to the learning model and the objective variables included in the validation data. The prediction accuracy may be a value based on RMSE (Root Mean Square Error), for example. The acquiring unit 29 also acquires the contribution of each type of feature quantity constituting the learning data using the validation data. The contribution may be a value based on the SHAP value, for example. The acquiring unit 29 outputs the prediction accuracy and the contribution of each acquired learning model to the generating unit 30.

[0037] The generating unit 30 generates an index for selecting a learning model for each learning model using the prediction accuracy and the contribution degree output from the acquiring unit 29. The generating unit 30 may generate an index using only the contribution degree for the type of feature quantity specified by the user. The generating unit 30 may also generate an index using the contribution degree for each type of feature quantity to which a weight specified by the user is added. This makes it possible to generate an index that takes into account the prediction accuracy and the contribution degree of a desired feature quantity, i.e., the basis for a desired prediction. When a weight is added to the contribution degree, it is further possible to adjust which type of feature quantity is given more importance in generating an index.

[0038] The generating unit 30 may generate an index after matching the scale of the contribution degree for each type of feature. The scale here refers to a scale of the magnitude of the value. This allows the generation of an index that can fairly evaluate the relative contribution degree for each type of feature. The generating unit 30 may generate an index by matching the scale of the prediction accuracy and the contribution degree. This allows the generation of an index that can fairly evaluate the magnitude of the prediction accuracy and the contribution degree. The generating unit 30 may generate an index by adding a weight to at least one of the prediction accuracy and the contribution degree. This allows the adjustment of which of the prediction accuracy and the contribution degree is emphasized when selecting a learning model. Note that, as described above, by matching the scale of the prediction accuracy and the contribution degree and adding a weight to at least one of the prediction accuracy and the contribution degree, it is possible to more appropriately adjust which of the prediction accuracy and the contribution degree is emphasized. The generating unit 30 outputs the index generated for each learning model to the selecting unit 31.

[0039] The selection unit 31 selects the learning model with the highest index from the multiple learning models stored in the evaluation dataset storage unit 27 based on the index of each learning model output from the generation unit 30. The selection unit 31 may select one or more learning models with an index equal to or greater than a predetermined value, or may select a predetermined number of learning models with the top indexes. When multiple learning models are selected, the selection unit 31 may present the selected learning models together with the indexes to the user and accept the selection of the learning model to be adopted from the user.

[0040] Next, a description will be given of the operation of the learning model selection device 10. Fig. 5 is a sequence diagram showing the flow of the learning model selection process by the learning model selection device 10. The CPU 11 reads out the learning model selection program including the generation program from the ROM 12 or the storage 14, expands it in the RAM 13, and executes it, thereby performing the learning model selection process.

[0041] In step S11, the CPU 11, functioning as the material data collection unit 21, accepts input of a data identifier designated by the user, collects material data having the accepted data identifier, and stores the collected material data in the material data storage unit 22. Next, in step S12, the CPU 11, functioning as the learning pattern transmission unit 23, accepts input of a learning pattern set designated by the user, and transmits the accepted learning pattern set to the evaluation dataset creation unit 24.

[0042] Next, in step S13, the CPU 11, as the evaluation dataset creation unit 24, acquires material data from the material data storage unit 22. Next, in step S14, the CPU 11, as the evaluation dataset creation unit 24, executes an evaluation dataset creation process. Specifically, the CPU 11, as the verification data creation unit 25 of the evaluation dataset creation unit 24, creates learning data from the material data for each learning pattern included in the learning pattern set, according to a learning data construction method included in the learning pattern. Also, the CPU 11, as the verification data creation unit 25, extracts a part of the created learning data as verification data, and outputs the remaining learning data to the learning model creation unit 26. Furthermore, the CPU 11, as the learning model creation unit 26 of the evaluation dataset creation unit 24, creates a learning model using the learning data output from the verification data creation unit 25 according to the base model identifier and hyperparameters included in the learning pattern.

[0043] Next, in step S15, CPU 11, as evaluation dataset creation unit 24, sets the validation data created by validation data creation unit 25 and the learning model created by learning model creation unit 26 as validation data. Then, CPU 11, as evaluation dataset creation unit 24, compiles the validation data for the number of learning patterns, i.e., the number of indexes included in the learning pattern set, as an evaluation dataset, and stores it in evaluation dataset storage unit 27.

[0044] Next, in step S16, the CPU 11, functioning as the learning model evaluation unit 28, acquires the evaluation dataset stored in the evaluation dataset storage unit 27. Also, in step S17, the CPU 11, functioning as the learning model evaluation unit 28, accepts an input of a type of feature specified by a user ("designation of feature" in FIG. 5). Note that the CPU 11, functioning as the learning model evaluation unit 28, may accept a weight for each type of feature with respect to the contribution degree ("weight of contribution degree" in FIG. 5) instead of designation of a feature. Furthermore, the CPU 11, functioning as the learning model evaluation unit 28, may accept a weight to be assigned to at least one of the prediction accuracy and the contribution degree when generating an index ("overall weight" in FIG. 5).

[0045] Next, in step S18, the CPU 11 executes a learning model evaluation process as the learning model evaluation unit 28. Specifically, the CPU 11, as the acquisition unit 29 of the learning model evaluation unit 28, acquires prediction accuracy for each learning model included in the evaluation data set using the verification data. Also, the CPU 11, as the acquisition unit 29, acquires the contribution degree for each type of feature amount using the verification data for each learning model. Then, the CPU 11, as the generation unit 30 of the learning model evaluation unit 28, generates an index for selecting a learning model for each learning model using the prediction accuracy acquired by the acquisition unit 29 and the contribution degree for the type of feature amount specified by the user. Note that, when the weight of the contribution degree is accepted instead of the designation of the feature amount, the CPU 11, as the generation unit 30, generates an index by adding the weight of the contribution degree to the contribution degree for each type of feature amount. Furthermore, when the overall weight is accepted, the CPU 11, as the generation unit 30, generates an index by adding the overall weight to at least one of the prediction accuracy and the contribution degree. Then, the CPU 11, as the selection unit 31 of the learning model evaluation unit 28, selects the learning model with the highest index from the multiple learning models stored in the evaluation dataset storage unit 27 based on the index of each learning model generated by the generation unit 30.

[0046] Next, in step S19, the CPU 11, functioning as the learning model evaluation unit 28, stores the learning model selected by the selection unit 31 ("selected learning model" in FIG. 5) in the selected learning model storage unit 32, and the learning model selection process ends.

[0047] In step S18, the process executed by the CPU 11 as the acquisition unit 29 and the generation unit 30 is an example of a generation process executed by the CPU 11 executing a generation program included in the learning model selection program. The generation process is also an example of a generation method of the disclosed technology.

[0048] Conventionally, the evaluation of learning models does not take into account the magnitude or positive / negative of the contribution of each type of feature to the prediction result. Therefore, when it is clearly stated that the learning model is evaluated using two indices, "prediction accuracy" and "contribution of each type of feature," it may not be possible to select an appropriate learning model. For example, as in the technology described in Non-Patent Document 1, even if multiple learning models with different structures and features used are listed in descending order of prediction accuracy, they are not necessarily listed in the optimal order that satisfies the desired prediction accuracy and grounds for prediction. This is because, in the conventional technology, model candidates are listed based only on the prediction accuracy estimated based on a predetermined index. The contribution of each type of feature changes depending on the nature of the learning data and the learning model, the cost function designed after selecting the learning model, etc. This point is not taken into consideration in the conventional technology, making it difficult to select a learning model that satisfies both the high prediction accuracy of the learning model and the distribution of the contribution of the model's features being close to the distribution of the contribution of the desired features. Therefore, in the conventional technology, it is necessary to manually search for the optimal learning model from the listed learning models. In particular, as the number of types of features increases, the number of candidate learning models also becomes enormous, making the task of selecting a learning model itself difficult.

[0049] On the other hand, the learning model selection device according to the first embodiment generates an index for each of a plurality of learning models from the prediction accuracy of the learning model and the contribution of at least one type of feature quantity specified by the user. In other words, this index takes into account both the contribution of the type of feature quantity specified by the user and the prediction accuracy. Then, the learning model selection device uses this index to select a predetermined learning model from among a plurality of learning models machine-learned using a plurality of types of feature quantities. This allows the learning model selection device according to the first embodiment to appropriately select a learning model that can obtain a desired prediction accuracy and basis for prediction.

[0050] <Second embodiment> Next, a second embodiment will be described. In the second embodiment, an embodiment that is a more specific version of the first embodiment will be described. Specifically, this is an example in which reinforcement learning is used to calculate an optimal control method for the temperature setting of an air conditioning controller with the goal of optimizing thermal comfort in an air conditioning utilization section that is separated from the outside by a door or the like inside a building. The learning model to be selected is an environmental model that predicts temperature changes according to the temperature setting of the air conditioner, which is the action of the reinforcement learning agent. Note that the hardware configuration of the learning model selection device according to the second embodiment is similar to the hardware configuration of the learning model selection device 10 according to the first embodiment shown in FIG. 3, and therefore a description thereof will be omitted.

[0051] The functional configuration of the learning model selection device 110 according to the second embodiment will be described. FIG. 6 is a block diagram showing an example of the functional configuration of the learning model selection device 110. As shown in FIG. 6, the learning model selection device 110 includes, as a functional configuration, a material data collection unit 121, a material data storage unit 122, a learning pattern transmission unit 123, an evaluation data set creation unit 124, and an evaluation data set storage unit 127. The evaluation data set creation unit 124 further includes a verification data creation unit 125 and a learning model creation unit 126. In addition, the learning model selection device 110 includes a learning model evaluation unit 128 and a selected learning model storage unit 132. The learning model evaluation unit 128 further includes an acquisition unit 129, a generation unit 130, and a selection unit 131. The acquisition unit 129 and the generation unit 130 are examples of a generation device of the disclosed technology. Each functional configuration is realized by the CPU 11 reading out a learning model selection program including a generation program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0052] In addition, detailed explanations of the common contents between the functional configuration of the learning model selection device 110 in the second embodiment and the functional configuration of the learning model selection device 10 in the first embodiment, where the last two digits of the symbols are in common, will be omitted.

[0053] The material data collection unit 121 collects material data related to air conditioning control. For example, the material data collection unit 121 collects each of the material data having a data identifier represented by the room temperature, the outside temperature, the flow of people, the air conditioning setting value, and the open flag. The room temperature is the temperature measured in the air conditioning utilization unit. The outside temperature is the temperature measured outdoors. The flow of people is the number of unique people present in the air conditioning utilization unit. The unique number of people is the number of people present in the air conditioning utilization unit per unit time, and is the number of people that are not counted twice for the same person even if they enter or leave the air conditioning utilization unit within the unit time (i.e., it is not the total number of people). The unit time may be, for example, the sampling interval of the data. The air conditioning setting value is the temperature setting value of the air conditioner present in the air conditioning utilization unit. The open flag is a flag indicating whether people can enter or leave the building including the air conditioning utilization unit. For example, if entry and exit to the building is possible, it may be set to "1", and if entry and exit is not possible, it may be set to "0".

[0054] Of the material data, room temperature is material data used as the objective variable, and outside temperature, foot traffic, air conditioning setting value, and open flag are material data used as explanatory variables. When the learning model is intended to be used as an environmental model for reinforcement learning, as in the second embodiment, it is necessary to include data that corresponds to the action of the reinforcement learning agent, or processed data, among the explanatory variables. This is because the output of the environmental model needs to change depending on the action of the reinforcement learning agent. Here, an explanatory variable using the air conditioning setting value is essential.

[0055] The material data collection unit 121 may collect, for example, room temperature, outside temperature, air conditioning setting value, and open flag from a BEMS (Building and Energy Management System) 201. The material data collection unit 121 may also collect people flow from a people flow detection sensor 202 installed in the air conditioning utilization unit. All of these material data are time-series data. Specifically, each material data is time-series data in which the date and time of the data sampling point is used as an index, and the index is associated with the data value at the date and time indicated by the index. The material data collection unit 121 stores the collected material data in the material data storage unit 122. FIG. 7 shows an example of material data stored in the material data storage unit 122.

[0056] The learning pattern transmission unit 123 accepts an input of a learning pattern set designated by a user, and transmits the accepted learning pattern set to the evaluation data set creation unit 124. FIG. 8 shows an example of a learning pattern set. In the example of FIG. 8, a learning pattern (p) is an index of a learning pattern. Here, p=1, 2, 3. In addition, hereinafter, a learning pattern with an index p (p=1, 2, 3) is expressed as "learning pattern p". In addition, in the example of FIG. 8, the base model identifier of learning pattern 1 is Light GBM (Gradient Boosting Machine, Reference 2). In addition, the base model identifier of learning patterns 2 and 3 is XGBoost (eXtreme Gradient Boosting, Reference 3). There is a partial difference between the learning pattern 2 and the learning pattern 3 in the learning data construction method (details will be described later).

[0057] Reference 2: Ke et al., "LightGBM: A Highly Efficient Gradient Boosting Decision Tree", 2017. Reference 3: Tianqi Chen, Carlos Guestrin, "XGBoost: A Scalable Tree Boosting System", 2016.

[0058] Here, in the second embodiment, the material data is time-series data, and can be expressed as table data including a plurality of columns as shown in FIG. 7 described above. Therefore, it is possible to create new columns that become feature quantities constituting the learning data by performing calculations between different columns of table data, acquiring time differences of the same column, and the like. Furthermore, in the case of explanatory variables, it is possible to create new columns by shifting the index of the new column created as described above, that is, by shifting the time series. Hereinafter, such a time-series-shifted column is referred to as a "series column." A method for creating this new column is defined in the learning data construction method for learning patterns.

[0059] In the example of FIG. 8, the learning data construction method uses the feature identifier F i p , formula E i p , and sequence parameter S1 i p , S2 i p , S3 i p is defined. Feature identifier F i p is an identifier indicating the type of feature. i is the index of the feature identifier, where i = 0, 1, 2, 3, 4, 5. Therefore, the feature identifier F i p represents the i-th type of feature of the learning pattern p. In the case of the sequence column, the feature of the material data column that was the source of the shift or the newly created column, and the sequence column are essentially the same type of feature, and therefore the same feature identifier F i p For example, a column A and a new column A' created by shifting the data of column A 30 minutes backward will both have the same feature identifier A. In the following, in order to distinguish between the series columns, for example, F i pThe feature identifier is expressed by adding a notation indicating the shifted time, such as -30 min. Note that the feature identifier F i p The feature value indicated by is the objective variable, and the feature identifier F i p The feature quantity indicated by is used as the explanatory variable. For example, the room temperature difference (F0 1 ) is the feature identifier F0 in learning pattern 1. 1 The type of feature represented by is the objective variable "room temperature difference". For example, the room temperature difference (F1 1 ) is the feature identifier F1 in learning pattern 1. 1 This indicates that the type of feature represented by is the explanatory variable “room temperature difference”.

[0060] Calculation formula E i p is a formula for calculating a new column that is a feature from material data, and is specified using the data identifier of the material data. For example, the formula in the first line of Figure 8 is the room temperature difference (F0 1 ) Formula for calculating E0 1 and E0 1 specifies that the room temperature 60 minutes before time t is subtracted from the room temperature at time t. In what follows, the <data identifier> or <feature identifier> at time t will be expressed as "<data identifier> or <feature identifier>(t)", for example, "room temperature (t)" or "room temperature difference (t)".

[0061] Series parameter S1 i p , S2 i p , S3 i p is the formula E i p This is a parameter for creating a sequence column based on the new column X created in S1. i p is the number of series, S2 i p is the start time, S3 i pis a parameter that specifies the end point. Specifically, S2 i p ~S3 i p The intervals are equally spaced S1 i p This represents the division into the number of parts and the creation of a sequence column by shifting X to each point in time. p In the case of , i.e., the objective variable, there is no need to create a sequence column, so no sequence parameters are specified.

[0062] The verification data creation unit 125 acquires the material data from the material data storage unit 122, and creates learning data from the material data according to the learning data construction method included in the learning pattern. A specific example of learning data creation will be described using the example of the learning pattern set shown in FIG. 8. The verification data creation unit 125 creates learning data from E0 1 = Room temperature (t) - Room temperature (t-60min), feature identifier F0 1 For example, the verification data creation unit 125 calculates the room temperature difference (t) shown by [F0 1 ] = [Room temperature at 2020-01-01 09:00:00] - [Room temperature at 2020-01-01 08:00:00]. Figure 9 shows the feature identifier F0 1 An example of the room temperature difference (t) is shown below.

[0063] Also, for example, the verification data creation unit 125 calculates the formula E1 1 and the series parameter S11 1 / S21 1 / S31 1 Using =6 / -60min / -360min, feature identifier F1 1 Specifically, the verification data creation unit 125 calculates a series column of the room temperature difference (t) represented by the formula E1 1 = Room temperature (t) - F1 calculated at room temperature (t-60min) 1 F1 shifted by -60min 1-60min is calculated as room temperature (t-60min) - room temperature (t-120min). For example, the verification data creation unit 125 calculates [F1 1 -60min] = [Room temperature at 2020-01-01 08:00:00] - [Room temperature at 2020-01-01 07:00:00]. Figure 10 shows the feature identifier F1 created as an explanatory variable of the learning data. 1 ~F5 1 An example of the sequence column of F1 in FIG. 10 is shown below. 1 is the room temperature difference, F2 1 is the outside temperature, F3 1 is the flow difference, F4 1 is the difference in air conditioning settings, F5 1 is a feature identifier indicating each of the open flags.

[0064] Furthermore, the verification data creation unit 125 extracts a part of the learning data as verification data, and outputs the remaining learning data to the learning model creation unit 126. For example, when the verification data creation unit 125 acquires one month's worth of material data, it creates one month's worth of learning data. Then, the verification data creation unit 125 extracts a certain week's worth of learning data that is specified in advance from the one month as verification data.

[0065] The learning model creation unit 126 sets hyperparameters in the model structure indicated by the base model identifier included in the learning pattern, inputs explanatory variables of the learning data to the learning model in which the parameters to be adjusted are set to initial values, and obtains output data. The learning model creation unit 126 then executes machine learning of the learning model by updating the parameters so that the output data and the objective variable are closer to each other. Note that, if the hyperparameters are not uniquely determined but are specified as ranges, the learning model creation unit 126 may create the learning model while searching for hyperparameters that will maximize the performance of the learning model.

[0066] The acquiring unit 129 converts the SHAP values ​​of the same number of elements as the explanatory variables of the validation data (the number of columns of the explanatory variables × the number of indexes of the validation data) into absolute values ​​for each learning pattern using the validation data. i p Then, as shown in FIG. 11, the feature identifier F i p The SHAP of each element in the validation data period for all columns belonging to i p Average of (SHAP i p )￣(In Figures 11 and 13, and in the formula, "SHAP i p The acquisition unit 129 calculates "(overline)" on top of all (SHAP i p ) ￣ is scaled to fall within the range of 0 to 100. i p In addition, the acquisition unit 129 calculates c i p The contribution evaluation vector c p (written in bold in the formula), i.e., c i p is the feature identifier F i p corresponds to the contribution of the type of feature indicated by

[0067]

number

[0068] The acquisition unit 129 also acquires a contribution weight vector W1 (indicated in bold in the formula) designated by the user. The contribution weight vector W1 is, for example, as shown in FIG. p ,w2 p ,w3 p ,w4 p ,w5 p ] T (T stands for transpose), each element of which is the contribution evaluation vector c pThe contribution weight vector W1 is a vector representing the weight corresponding to each element of the contribution evaluation vector W1. The contribution weight vector W1 is created in advance by the user, and the weight of each element may be set to a value according to the degree of importance to be attached when selecting a learning model. For example, as in the second embodiment, in the context of "constructing an environmental model for reinforcement learning", the weight of the "feature value related to the action of the reinforcement learning agent" may be set large. In this case, this corresponds to the "air conditioning setting value". Note that by using two values, 0 and 1, as the weight, it is possible to realize the specification of the feature value as explained in the first embodiment. The acquisition unit 129 calculates the contribution evaluation vector c as shown in the following formula (3). p and the contribution weight vector W1 to obtain the contribution evaluation α p Get the.

[0069]

number

[0070] In addition, the acquisition unit 129 calculates the root mean square error (RMSE) using the validation data. p Calculate the RMSE as shown in the following formula (4), for example. p The reciprocal of the accuracy evaluation β p The acquisition unit 129 acquires the acquired contribution evaluation α p and accuracy evaluation β p to the generation unit 130.

[0071]

number

[0072] The generation unit 130 calculates the contribution evaluation α p and accuracy evaluation β p Using the above, for example, the contribution evaluation α p and accuracy evaluation β p Calculate the contribution scaling constant K to match the scale with

[0073]

number

[0074] The generating unit 130 also obtains an overall contribution weight W2 as shown in FIG. 12. The overall contribution weight W2 is a value set in advance by the user, and may be set to a value according to how much importance is attached to the contribution when the prediction accuracy is set to 1. The generating unit 130 calculates the contribution evaluation α p ,Accuracy evaluation β p , the contribution scaling constant K, and the overall contribution weight W2 are used to calculate the model evaluation function L, for example, according to the following equation (6): p Generate.

[0075]

number

[0076] Instead of the contribution scaling constant K, a prediction accuracy scaling constant K' obtained by reciprocal of equation (5) may be used. In this case, in equation (6), K' is replaced by β p In addition, instead of the overall contribution weight W2, an overall prediction accuracy weight W2', which is a weight according to how much importance is attached to the prediction accuracy when the contribution is set to 1, may be used. In this case, in formula (6), W2' is multiplied by β p Multiply the term.

[0077] FIG. 13 shows the (SHAP i p ) ￣, contribution evaluation vector c p , and RMSE p 14 shows an example of the contribution evaluation α p and accuracy evaluation β p , the contribution degree scaling constant K calculated by the generation unit 130, the contribution degree evaluation after scale conversion α p ×K, and the model evaluation function L generated by the generation unit 130 p An example is shown below.

[0078] The selection unit 131 selects a model evaluation function L generated by the generation unit 130 from the multiple learning models stored in the evaluation data set storage unit 127. p Select the learning model that maximizes

[0079] Next, the operation of the learning model selection device 110 will be described. In the second embodiment, as in the first embodiment, the CPU 11 reads out a learning model selection program including a generation program from the ROM 12 or storage 14, expands it in the RAM 13, and executes it to perform the learning model selection process shown in Fig. 5. Here, more detailed processes of the evaluation data creation process in step S14 in Fig. 5 and the learning model evaluation process in step S18 will be described with reference to the flowcharts shown in Figs. 15 and 16.

[0080] First, the evaluation data set creation process shown in FIG. 15 will be described.

[0081] In step S101, the CPU 11, functioning as the evaluation dataset creation unit 124, initializes a learning model set and a validation dataset. Specifically, the CPU 11, functioning as the evaluation dataset creation unit 124, prepares an empty set to which the created learning model is to be added, and an empty set to which the created validation data is to be added. Furthermore, the CPU 11, functioning as the evaluation dataset creation unit 124, sets a variable p indicating an index of a learning pattern to 1.

[0082] Next, in step S102, the CPU 11, functioning as the evaluation data set creation unit 124, determines whether or not p is equal to or less than the maximum index value pmax (here, pmax=3). If p≦pmax, the process proceeds to step S103.

[0083] In step S103, the CPU 11, functioning as the evaluation dataset creation unit 124, initializes a learning dataset. Specifically, the CPU 11, functioning as the evaluation dataset creation unit 124, prepares an empty set to which the created learning data is to be added. Then, the CPU 11, functioning as the verification data creation unit 125, sets a variable i indicating an index of a feature identifier to 0.

[0084] Next, in step S104, the CPU 11, functioning as the verification data creation unit 125, determines whether or not i is equal to or less than the maximum index value imax (here, imax=5). If i≦imax, the process proceeds to step S105.

[0085] In step S105, the CPU 11, functioning as the verification data generating unit 125, calculates the formula E i p The material data used in is obtained from the material data storage unit 122, and the obtained material data is applied to the formula E i p to create a new column X.

[0086] Next, in step S106, the CPU 11, functioning as the verification data creation unit 125, checks whether i is 0, that is, whether the feature identifier F corresponding to column X is i p is a response variable. If i=0, the process proceeds to step S107, and the CPU 11, functioning as the validation data creation unit 125, adds column X to the learning data set. On the other hand, if i≧1, the process proceeds to step S108. In step S108, the CPU 11, functioning as the validation data creation unit 125, adds sequence parameter S i p , S2 i p , S3 i p Read S2 i p ~S3 i p The intervals are equally spaced S1 i p, and creates a sequence column by shifting X to each time point. Then, the CPU 11, functioning as the verification data creation unit 125, adds the created sequence column to the learning data set.

[0087] Next, in step S109, the CPU 11, functioning as the verification data creation unit 125, increments i by 1, and the process returns to step S104. If it is determined in step S104 that i>imax, the process proceeds to step S110.

[0088] In step S110, the CPU 11, functioning as the validation data creation unit 125, extracts a part of the learning data included in the learning data set as validation data and adds it to the validation data set. The CPU 11, functioning as the validation data creation unit 125, outputs the remaining learning data to the learning model creation unit 126.

[0089] Next, in step S111, the CPU 11, as the learning model creation unit 126, acquires the model structure specified by the base model identifier of the learning pattern p by calling an API from the base model identifier, etc., and sets the hyperparameters of the learning pattern p. Then, the CPU 11, as the learning model creation unit 126, executes machine learning of the learning model in which the hyperparameters are set to the model structure indicated by the base model identifier, using the remaining learning data output from the validation data creation unit 125. The CPU 11, as the learning model creation unit 126, executes machine learning while evaluating the learning model by, for example, grid search cross validation.

[0090] Next, in step S112, the CPU 11, functioning as the learning model creation unit 126, adds the completed learning model to the learning model set. Next, in step S113, the CPU 11, functioning as the evaluation dataset creation unit 124, increments p by 1, and the process returns to step S102. If it is determined in step S102 that p>pmax, the evaluation dataset creation process ends.

[0091] The validation dataset and the learning model set created by the evaluation dataset creation process are stored as an evaluation dataset in the evaluation dataset storage unit 127 (S17 in FIG. 5).

[0092] Next, the learning model evaluation process shown in FIG. 16 will be described.

[0093] In step S121, the CPU 11, functioning as the learning model evaluation unit 128, sets a variable p indicating an index of a learning pattern to 1. Next, in step S122, the CPU 11, functioning as the learning model evaluation unit 128, determines whether p is equal to or less than a maximum index value pmax (here, pmax=3). If p≦pmax, the process proceeds to step S123.

[0094] In step S123, the CPU 11, as the acquisition unit 129, uses the verification data of the learning pattern p to obtain a SHAP value obtained by converting the SHAP value having the same number of elements as the explanatory variables of the verification data into absolute values. i p Next, in step S124, the CPU 11, functioning as the acquisition unit 129, sets 1 to a variable i indicating an index of the feature identifier.

[0095] Next, in step S125, the CPU 11, functioning as the acquisition unit 129, determines whether or not i is equal to or smaller than the maximum index value imax (here, imax=5). If i≦imax, the process proceeds to step S126.

[0096] In step S126, the CPU 11, functioning as the acquisition unit 129, acquires the feature identifier F i p The SHAP of each element in the validation data period for all columns belonging to i p Average of (SHAP i p) Next, in step S127, the CPU 11 functioning as the acquisition unit 129 increments i by 1, and the process returns to step S125. If it is determined in step S125 that i>imax, the process proceeds to step S128.

[0097] In step S128, the CPU 11, functioning as the acquisition unit 129, calculates all (SHAP i p ) is scaled to fall within the range of 0 to 100. i p Next, in step S129, the CPU 11, functioning as the acquisition unit 129, calculates c i p The contribution evaluation vector c p In addition, the CPU 11, as the acquisition unit 129, acquires a contribution weight vector W1 designated by the user, and creates a contribution evaluation vector c p and the contribution weight vector W1 to obtain the contribution evaluation α p Get the.

[0098] Next, in step S130, the CPU 11, functioning as the acquisition unit 129, calculates the root mean square error (RMSE) using the verification data of the learning pattern p. p Calculate the RMSE as shown in equation (4), for example. p The reciprocal of the accuracy evaluation β p Next, in step S131, the CPU 11, functioning as the learning model evaluation unit 128, increments p by 1, and the process returns to step S122. If it is determined in step S122 that p>pmax, the process proceeds to step S132.

[0099] In step S132, the CPU 11, functioning as the generation unit 130, calculates the contribution evaluation α p and accuracy evaluation β p Using the formula (5), for example, the contribution evaluation α p and accuracy evaluation β pCalculate the contribution scaling constant K to match the scale with

[0100] Next, in step S133, the CPU 11 as the generation unit 130 acquires the overall contribution weight W2 designated by the user. Then, the CPU 11 as the generation unit 130 calculates the contribution evaluation α p ,Accuracy evaluation β p , the contribution scaling constant K, and the overall contribution weight W2 are used to calculate the model evaluation function L, for example, using equation (6). p Then, the CPU 11, as the selection unit 131, selects the model evaluation function L generated by the generation unit 130 from the multiple learning models stored in the evaluation data set storage unit 127. p The learning model that maximizes is selected, and the learning model evaluation process ends.

[0101] The learning model selected by the learning model evaluation process is stored in the selected learning model storage unit 132 (S19 in FIG. 5).

[0102] As described above, the learning model selection device of the second embodiment can also appropriately select a learning model that provides the desired prediction accuracy and basis for prediction, similar to the learning model selection device of the first embodiment.

[0103] In addition, the scale of contribution of each type of feature calculated for each learning model does not necessarily match, so if it is used as is, it is not possible to make a fair comparison between learning models. For example, i p The sum of ￣ does not match, (SHAP i p In the second embodiment, the contribution of each type of feature is scaled from 0 to 100 as shown in equation (1), for example, to determine the contribution of each learning pattern. p The sum of these is 100, allowing for a fair evaluation of the relative contribution of each type of feature.

[0104] In the second embodiment, the contribution of a desired feature can be emphasized in the evaluation by using a "contribution weight vector" that indicates the individual weight for each type of feature. For example, by designing a contribution weight vector W1 as shown in FIG. 12, the fourth feature identifier F4, which has the highest weight, can be p Therefore, as shown in Figs. 13 and 14, the air conditioning setting value or the air conditioning room temperature difference indicated by feature identifier F4 p Corresponding C4 p C i p The largest contribution evaluation α of learning pattern 1 p is at its maximum.

[0105] In addition, since the scale of the contribution degree and the accuracy do not necessarily match, there are cases where either the contribution degree evaluation or the accuracy evaluation is too large or too small. For example, in the example of FIG. p is the accuracy evaluation β p Therefore, if K=1 and the model evaluation function L p When generating a model, the contribution evaluation becomes too large, and the difference in accuracy evaluation is caused by the model evaluation function L p In the second embodiment, for example, as shown in the formula (5), a contribution scaling constant K is calculated to match the scale of the contribution evaluation with that of the accuracy evaluation, and the contribution evaluation α p As shown in Figure 14, by multiplying p and accuracy evaluation β p The scale is roughly the same, allowing for a fair evaluation.

[0106] Furthermore, by designing K as in the second embodiment, the user can intuitively design the overall contribution weighting W2. For example, when the contribution evaluation and accuracy evaluation are to be equally emphasized, W2=1 can be set; when the contribution evaluation is to be emphasized, W2>1; and when the accuracy evaluation is to be emphasized, W2<1. For example, the second embodiment shows an example where W2=1.5, but when W2=0.5, the accuracy evaluation is emphasized, and as shown in FIG. 17, the accuracy evaluation β p The model evaluation function L of the learning pattern 3 with the largest p is the largest.

[0107] Here, a method for adjusting W2 will be described using a specific example similar to that of the second embodiment. As a premise, it is desirable that the room temperature predicted by the learning model fluctuates according to the air conditioning setting value. In addition, after a certain time has passed since the air conditioner is started, the room temperature becomes steady due to thermal equilibrium. It is desired to reproduce that behavior with the prediction of the learning model. Specifically, if the heating is started one hour earlier, the predicted room temperature also starts one hour earlier, and after four hours have passed since the air conditioner is started, the predicted room temperature converges to the room temperature when the air conditioner is not started one hour earlier, that is, the true room temperature value. After the air conditioner is started, the speed at which the predicted room temperature converges to the true room temperature value depends on the contribution degree of the air conditioning setting value, and in the example of the second embodiment, it depends on the contribution degree evaluation. In order to confirm that the air conditioning setting value contributes to the prediction of the room temperature as expected, the behavior of the predicted room temperature is observed when the air conditioning setting value is shifted one hour earlier from the original data.

[0108] The procedure for adjusting W2 under the above assumptions is as follows. First, FIG. 19 shows the prediction results when learning pattern 3 that maximizes the model evaluation function is selected when W2=1.0 as shown in FIG. 18. In FIG. 19 and FIG. 20 described later, temperature is the true room temperature value, and shifted_predict is the predicted room temperature when the air conditioning setting value is shifted to one hour earlier. Also, air control temperature is the air conditioning setting value or the air conditioning setting value difference, and shifted air control temperature is the air conditioning setting value or the air conditioning setting value difference shifted to one hour earlier.

[0109] The learning model for learning pattern 3 has the highest accuracy, but even if the air conditioner is started at around 7:00, the predicted room temperature converges to the true room temperature by around 10:00 (the area circled in dashed line in Figure 19), which deviates from the ideal behavior. In this case, it is considered that the contribution of the air conditioning setting to the learning model's predictions is insufficient, so W2 is increased to 1.5 and the learning model is selected again.

[0110] Figure 20 shows the prediction results when learning pattern 1, which maximizes the model evaluation function, is selected when W2=1.5, as shown in Figure 18. The accuracy of this learning model is slightly lower than that of the learning model of learning pattern 3, but after the air conditioner is started at around 7:00, the predicted room temperature converges to the true room temperature value at around 11:00 (the area surrounded by the dashed circle in Figure 20), which is close to ideal behavior. Therefore, the contribution of the air conditioning setting value is considered to be sufficient, and the learning model selected with W2=1.5 is adopted. Note that if the time at which the predicted room temperature converges to the true room temperature value is late, such as 12:00, the contribution is considered to be too high, so W2 is reduced and the learning model is selected again.

[0111] <Modification> In the second embodiment, a case has been described in which time-series data such as room temperature and air conditioning settings, and a regression problem using a decision tree model such as LightGBM are handled, but the disclosed technology is also applicable to other types of data and problems. Below, the differences from the second embodiment will be mainly described for each modification.

[0112] As a first modified example, a regression problem of stock price prediction using a deep learning model and time series data of stock transactions can be given. In this problem, a regression is performed with a future "current price" as a target variable based on data on stock transactions of a certain stock. As data identifiers for material data, "current price", "highest purchase price", "lowest sale price", "highest number of sale prices", "highest number of purchase prices", "total number of purchase orders", "total number of sell orders", etc. are used. The base model identifier may be selected from two types, for example, LSTM (Long Short-Term Memory, Reference 4) and QRNN (Quasi-Recurrent Neural Networks, Reference 5). The hyperparameters are the number of hidden layer nodes, number of steps, batch size, dropout rate, and number of layers for the layer of the base model identifier, the number and total number of nodes for the fully connected layer, the number of nodes and activation function for the activation function layer, and an optimization function. In addition, since the size of data related to transactions varies greatly depending on the stock, a normalization process may be added to all feature identifiers in the calculation formula of the learning data construction method. In addition, in the evaluation dataset creation unit, the explanatory variables of the training data are a two-dimensional matrix of [number of data × (number of features × number of steps)], but a process is added to convert this into a three-dimensional matrix of [number of data × number of steps × number of features].

[0113] As a second variation, a classification problem of the subscription status of a paid membership service using a machine learning model and time-series data of membership information can be given. In this problem, the future "subscription status of a service (subscribed or unsubscribed)" is predicted for a certain paid membership service based on time-series data such as the customer's usage history. As data identifiers for the material data, "customer ID", "gender", "service subscription date", "service unsubscription date (Nan value for those who have not unsubscribed)", "service usage time on that day", "service subscription status", etc. are used. When the customer ID is different, the same index may be duplicated for the index of the time-series data. In addition, in the learning data construction method, "number of days of service usage in the current month" may be created as a new feature identifier. The calculation formula for this feature identifier may be to divide the data by "customer ID", group the data by the year and month of the index of the time-series data, and count the number of data whose "service usage time on that day" is 0 or more. In addition, the calculation formula for the "service subscription date" may include a process of converting the year and month to days. In addition, a process of converting the Nan value of the "service unsubscription date" to -1 may be included. Since "Customer ID" and "Gender" are categorical variables, it may be specified to perform label encoding in these calculation formulas. When creating series columns using series parameters in the evaluation dataset creation section, data is divided by "Customer ID" before processing.

[0114] A third variant is a regression problem of house prices using a machine learning model and house feature data. In this problem, the "price" of a house is regressed from various information about the house. The data identifiers of the material data include "house type (apartment, house, etc.)", "prefecture", "city / town", "walking time to the nearest station", "year of construction", "floor plan (1K, 2LDK, etc.)", "exclusive area", "whether renovated", and "price". In addition, in the formula of the learning data construction method, categorical variables including characters such as "house type", "prefecture", "city / town", "floor plan", and "whether renovated" may be specified to perform label encoding. Since the material data in this problem is not time series data, there is no need to specify series parameters in the learning data construction method, and no process of creating a series column is performed.

[0115] A fourth modified example is a classification problem of iris varieties using a machine learning model and feature data of iris flowers. In this problem, the "variety" of iris flowers is classified based on various feature data of the flowers. As data identifiers of the material data, "sepal length", "sepal width", "petal length", "petal width", "variety", etc. are used. The base model identifier may be selected from two types, support vector machine (Reference 6) and logistic regression (Reference 7). In the case of a support vector machine, the hyperparameters are the kernel type, regularization method, evaluation function, whether or not to solve the dual problem, algorithm termination condition, strictness of soft margin, etc. In the case of logistic regression, the hyperparameters are the regularization method, strength of regularization, etc. In addition, in the learning data construction method, "sepal information" calculated by the formula ("sepal length" x "sepal width") and "petal information" calculated by the formula ("petal length" x "petal width") may be added as new feature identifiers. In the formula of the training data construction method, since "variety" is a categorical variable, label encoding may be specified in the formula. Since the material data in this problem is not time-series data, there is no need to specify series parameters in the training data construction method, and the process of creating a series column is not performed.

[0116] Reference 4: S. Hochreiter, J. Schmidhuber, "Long short-term memory", Neural Computation 9 (8), pp. 1735-1780, 1997. Reference 5: J. Bradbury, et al, "Quasi-Recurrent Neural Networks", ICLP, 2016. Reference 6: Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, Yasemin Altun, "Large Margin Methods for Structured and Interdependent Output Variables", The Journal of Machine Learning Research 6 (9), pp. 1453-1484, 2005. Reference 7: DR Cox, "The regression analysis of binary sequences (with discussion)", Journal of the Royal Statistical Society, Series B (Methodological), Vol. 20, No. 2, pp. 215-242, 1958.

[0117] In addition, the learning model selection process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. In addition, the learning model selection process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0118] In addition, in each of the above embodiments, the learning model selection program including the generation processing program is pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0119] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0120] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, For each of a plurality of learning models machine-learned using a plurality of types of feature quantities, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature quantity designated by a user to the prediction result are obtained; An index for selecting a predetermined learning model from among the plurality of learning models is generated based on the accuracy and the contribution degree. A generating device configured as follows.

[0121] (Additional note 2) A non-transitory recording medium storing a program executable by a computer to execute a generation process, The generation process includes: For each of a plurality of learning models machine-learned using a plurality of types of feature quantities, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature quantity designated by a user to the prediction result are obtained; An index for selecting a predetermined learning model from among the plurality of learning models is generated based on the accuracy and the contribution degree. Non-transitory recording media, including [Explanation of symbols]

[0122] 10, 110 Learning model selection device 11 CPU 12 ROM 13 RAM 14. Storage 15 Input section 16 Display section 17 Communication I / F 19 Bus 21, 121 Materials Data Collection Division 22, 122 Material Data Storage Unit 23, 123 Learning pattern transmission unit 24, 124 Evaluation Data Set Creation Department 25, 125 Verification Data Creation Department 26, 126 Learning Model Creation Department 27, 127 Evaluation Data Set Storage Section 28, 128 Learning model evaluation section 29, 129 Acquisition Department 30, 130 generation section 31, 131 Selection section 32, 132 Selection learning model storage section 201 BEMS 202 People flow detection sensor

Claims

1. A generation device for generating an index for selecting a predetermined learning model from a plurality of learning models machine-learned using a plurality of types of feature amounts, the device comprising: an acquisition unit that acquires, for each of the plurality of learning models, an accuracy of a prediction result by the learning model and a contribution degree to the prediction result of at least one type of feature value designated by a user; A generation unit that generates the index from the accuracy and the contribution degree; A generating device comprising:

2. The generating device according to claim 1 , wherein the generating unit generates the index using a contribution degree for each type of the feature amount to which a weight designated by a user is added.

3. The generating device according to claim 2 , wherein the generating unit generates the index using a contribution degree for each type of the feature amount, the contribution degree being determined on a consistent scale.

4. 4. The generating device according to claim 1, wherein the generating unit generates the index by matching a scale of the degree of accuracy with a scale of the degree of contribution.

5. 5. The generating device according to claim 1, wherein the generating unit generates the index by adding a weight to at least one of the accuracy and the contribution degree.

6. The learning model is an environment model that predicts a change in temperature according to the temperature setting of an air conditioner, which is an action of a reinforcement learning agent, when an optimal control method for the temperature setting of an air conditioning controller is calculated by reinforcement learning with the goal of optimizing the temperature of an air conditioning utilization unit, The feature amount is a feature amount related to air conditioning control including a temperature of the air conditioning utilization unit and a temperature setting value of the air conditioner. The generating device according to any one of claims 1 to 5.

7. A generation method executed by a generation device including an acquisition unit and a generation unit, which generates an index for selecting a predetermined learning model from a plurality of learning models machine-learned using a plurality of types of features, the acquiring unit acquires, for each of the plurality of learning models, an accuracy of a prediction result by the learning model and a contribution degree to the prediction result of at least one type of feature value designated by a user; The generating unit generates the index from the accuracy and the contribution degree. Generation method.

8. A generation program for generating an index for selecting a predetermined learning model from a plurality of learning models machine-learned using a plurality of types of features, Computer, an acquisition unit that acquires, for each of the plurality of learning models, an accuracy of a prediction result by the learning model and a contribution degree to the prediction result of at least one type of feature value designated by a user; and A generating unit for generating the index from the accuracy and the contribution degree. A generator to function as a.

Images