Learning device, learning method, and learning program

Abstract

This invention provides a learning device, learning method, and learning program capable of realizing a binary classification learning model optimized using the Ising model. [Solution] The solution includes a regularization method that performs L0 regularization of features included in a binary classification learning model constructed using logistic regression and a Factorization Machine, using an Ising model.

Description

[Technical Field]

[0001] This disclosure relates to a learning device, a learning method, and a learning program. [Background technology]

[0002] In recent years, approaches using the Ising model have attracted attention as a method for solving combinatorial optimization problems involving 0-1 binary variables (see Non-Patent Document 1) and for constructing learning models using Factorization Machines (FM) (see Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Lucas, A. Ising formulations of many NP problems. Front. Phys., 12, 00005 (2014). [Non-Patent Document 2] Kitai, K., Guo, J., Ju, S., Tanaka, S., Tsuda, K., Shiomi, J. & Tamura, R. Designing metamaterials with quantum annealing and factorization machines. Phys. Rev. Research 2, 013319 (2020). [Overview of the project] [Problems that the invention aims to solve]

[0004] Learning models can be broadly divided into regression models and classification models. Common loss functions for classification models include cross-entropy error and hinge loss. However, since these loss functions are not quadratic, directly optimizing learning models using the Ising model is difficult, and the two are not well-suited to each other.

[0005] This disclosure has been made in view of these problems. One of the objectives of this disclosure is to provide a learning device, a learning method, and a learning program that can realize a learning model for binary classification optimized using an Ising model.

Means for Solving the Problems

[0006] The learning device according to this disclosure includes regularization means for performing L0 regularization of feature amounts included in a learning model for binary classification constructed using logistic regression and a factorization machine, using an Ising model.

[0007] The learning method according to this disclosure is such that a computer performs L0 regularization of feature amounts included in a learning model for binary classification constructed using logistic regression and a factorization machine, using an Ising model.

[0008] The learning program according to this disclosure causes a computer to execute a regularization process for performing L0 regularization of feature amounts included in a learning model for binary classification constructed using logistic regression and a factorization machine, using an Ising model.

Effects of the Invention

[0009] According to this disclosure, a learning model for binary classification optimized using an Ising model can be realized.

Brief Description of the Drawings

[0010] [Figure 1] It is a block diagram illustrating a learning device. [Figure 2] It is a flowchart illustrating the operation of the learning device. [Figure 3] It is an explanatory diagram illustrating an overview of the operation of the learning device. [Figure 4] It is an explanatory diagram illustrating the influence on the correct classification rate when adjusting the hyperparameter α of the sigmoid function. [Figure 5]This is an explanatory diagram illustrating the relationship between the number of features (M) selected by L0 regularization and the accuracy rate. [Figure 6] This is an explanatory diagram illustrating the effect of the correction term in equation (11) on the accuracy rate. [Figure 7] This is a block diagram illustrating the hardware configuration of a computer that implements a learning device. [Figure 8] This is a block diagram illustrating the main components of a learning device. [Modes for carrying out the invention]

[0011] The application of the annealing method is limited to Ising models composed of binary variables. Regression models have been widely used in this field. However, in practical terms, classification models are considered to be just as important, if not more so, than regression models. This disclosure addresses methods for applying Ising models composed of binary variables to classification models.

[0012] One binary classification method using the Ising model is QBoost for QUBO (Quantum Boosting for Quadratic Unconstrained Binary Optimization), which employs ensemble learning of weak learners (see reference 3 below). However, the classification accuracy of this method is highly dependent on the accuracy of the weak learners and is generally difficult to improve. Therefore, a different approach is needed. Reference 3: "6. Machine Learning with Quantum Annealing (QBoost)", [online], OpenJij Tutorial 0.3.0 documentation, [Retrieved December 4, 2024], Internet<URL:https: / / tutorial.openjij.org / build / html / ja / 006-Machine_Learning_by_QA.html>

[0013] One of the objectives of this disclosure is to propose a binary classification solution using the Ising model that achieves accuracy comparable to standard binary classification methods. The learning device in this disclosure performs L0 regularization using the Ising model. This regularization is expected to improve generalization performance. Furthermore, since L0 regularization directly controls the number of features, it clearly distinguishes between important and unimportant features. As a result, improved interpretability of the learning model is also expected.

[0014] Embodiments of this disclosure will be described below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary for clarity. Unless otherwise specified, predetermined values ​​such as set values ​​and thresholds are stored in advance in a storage device accessible from the device that uses those values. Unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0015] In the following embodiments, even if the superscript and subscript of a variable are aligned within a mathematical formula, they may be offset within the text. In such cases, however, if the variable symbol, the superscript symbol, and the subscript symbol are the same, they represent the same variable.

[0016] Embodiment 1. [Explanation of the structure] The learning device of this embodiment will now be described. Figure 1 is a block diagram illustrating the learning device. The learning device 100 of this embodiment includes a data adjustment unit 110, a learning model construction unit 120, a binary variable introduction unit 130, an objective function generation unit 140, and a solution unit 150.

[0017] The data adjustment unit 110 has the function of taking data containing multiple features to be classified as input and converting it into a format suitable for optimization using FM learning or the Ising model. Specifically, the data adjustment unit 110 converts categorical variables into one-hot vectors for optimization using FM learning or the Ising model. The data adjustment unit 110 also randomly divides the entire dataset into training data and test data.

[0018] The learning model construction unit 120 has the function of constructing a binary classification learning model. The learning model construction unit 120 constructs a binary classification learning model using, for example, logistic regression and FM. The learning model construction unit 120 can uniformly adjust all the arguments of the exponential function of logistic regression using hyperparameters (for example, hyperparameter α described later).

[0019] The binary variable introduction unit 130 has the function of introducing binary variables into the constructed learning model. For example, the binary variable introduction unit 130 introduces a 0-1 binary variable representing selection or non-selection to each of the first-order and second-order terms of the learning model constructed by the learning model construction unit 120. By utilizing this 0-1 binary variable, it is possible to control the coefficients of the first-order and second-order terms and achieve feature selection (when the value of the 0-1 binary variable is 1) / non-selection (when the value of the 0-1 binary variable is 0).

[0020] The objective function generation unit 140 has the function of generating an objective function, which is the target of optimization (minimization) using the Ising model, based on a learning model in which binary variables have been introduced. Specifically, the objective function generation unit 140 converts the problem of regularizing the constructed learning model into a combinatorial optimization problem described in QUBO format. This combinatorial optimization problem is the problem of selecting a combination of 0-1 binary variables that minimizes the objective function.

[0021] The objective function generation unit 140 introduces a penalty function into the objective function to set the total number of features (terms) to be selected. The strength of the penalty function is adjusted by a hyperparameter (for example, hyperparameter A, described later). The objective function generation unit 140 also introduces a correction term into the objective function that reflects the classification result (success / failure) of the L0 regularized learning model. The strength of the correction term is adjusted by a hyperparameter (for example, hyperparameter B, described later).

[0022] The problem-solving unit 150 has the function of solving combinatorial optimization problems. Furthermore, the problem-solving unit 150 has the function of outputting the solution to the solved combinatorial optimization problem. For example, the problem-solving unit 150 outputs the solution to the solved combinatorial optimization problem as the feature selection result for the features included in the constructed learning model.

[0023] In this embodiment, the objective function generation unit 140 generates an objective function based on a learning model into which binary variables are introduced, and converts the problem of regularizing the constructed learning model into a combinatorial optimization problem, which is the problem of selecting a combination of binary variables that minimizes the objective function described in QUBO format. The solution unit 150 searches for a combination of binary variables that minimizes the objective function as a solution method for the combinatorial optimization problem, using methods such as simulated annealing or quantum annealing.

[0024] The learning device 100 of this embodiment has the following features A to D. A: The learning device 100 constructs a classification model within the general framework of logistic regression using the sigmoid function and the cross-entropy method. B: The learning device 100 uses a quadratic learning model f of the argument part of the sigmoid function via FM. FM (x i Set this f FM (x i Classification accuracy is improved by introducing a hyperparameter α (hereinafter also called "gain") that adjusts the overall size. C: The learning device 100 introduces 0-1 binary variables corresponding to selection or non-selection for each of the primary and secondary terms included in the learning model (hereinafter also referred to as the FM model) constructed using FM, and determines the optimal combination thereof using the Ising model. This process corresponds to L0 regularization. D: The learning device 100 introduces a correction term reflecting the classification result (success / failure) by the L0-regularized classification model into the objective function.

[0025] Regarding the details of item A, consider the case where the mean squared error (MSE: Mean Squared Error) is defined as the loss function in the evaluation of the classification problem. Here, the label (correct label) of the i-th data is y l i ={0,1}, and the predicted value predicted by some classification model is y p i Let it be. The loss function (Loss) for N data is defined by the following equation (1) using the mean squared error (MSE).

[0026]

Equation

[0027] The problem with using equation (1) is that when the predicted value y p i is much larger than 1, this label should originally be predicted as "1" with a high probability, but instead, the loss increases as it gets larger than 1. Similarly, when the predicted value y p i is much smaller than 0, this label should originally be predicted as "0" with a high probability, but the problem occurs that the loss increases instead as the predicted value gets smaller from 0 due to the characteristics of the squared error.

[0028] Therefore, in this embodiment, the learning model construction unit 120 applies the cross-entropy error as the loss function in the binary classification problem, and further, as the output function, the sigmoid function S that converts the input value x i of the i-th data into the interval [0,1] and outputs it.g (x i This method uses the sigmoid function and cross-entropy error, which eliminates the loss increase problem that occurs when using MSE.

[0029] The sigmoid function used in this embodiment is defined by equation (2) below.

[0030]

number

[0031] In equation (2), f FM (x i ) is a quadratic learning model constructed using FM, which will be described later. Also, α (α>0) is called the "gain" and is a hyperparameter that adjusts the steepness or slopiness of the sigmoid function. The loss function using cross-entropy error is defined by equation (3) below.

[0032]

number

[0033] By considering the partial derivative of equation (3) with respect to θ (a general term for the coefficient parameters included in the FM model), we derive the following equation (4).

[0034]

number

[0035] Let's explain item B in detail. When the input vector is x, the quadratic learning model F can be formally expressed by the following equation (5).

[0036]

number

[0037] Note that in equation (5), i and j are not data numbers, but rather numbers used to identify features. In other words, each data point is described by m features.

[0038] In FM, the intersection term in equation (5) (i.e., the third term on the right-hand side) is approximated by the inner product of latent vectors v, so this part is expressed by equation (6) below. The latent vector has K elements.

[0039]

number

[0040] f (2) (x) v a q Taking the partial derivative with respect to this, we obtain equation (7) below.

[0041]

number

[0042] Therefore, the partial derivatives of the loss function with respect to each parameter are calculated as shown in equation (8) below.

[0043]

number

[0044] The update formula for each parameter is expressed by equation (9) below, where r is the learning rate.

[0045]

number

[0046] In the learning process of the embodiment described later, mini-batch learning (batch size 32) was used, and the learning rate was set to 0.02, and the stochastic gradient descent method was applied. However, the learning process by the learning device 100 of this disclosure is not limited to this method.

[0047] Let's explain the details of item C. The quadratic learning model represented by equation (5) includes an intersection term between m features and m(m-1) / 2 features. The intersection term is x i x j to X k By substituting this, it can be considered as a type of feature. Thus, the quadratic learning model represented by equation (5) can be considered to consist of a total of m(m+1) / 2 effective features.

[0048] Incidentally, when the number of features is large, there is a concern about overfitting, where the model overfits the training data, so regularization is necessary. Generally, L1 or L2 regularization is used, but in the Ising model, L0 regularization, which directly controls the selection / deselection of each feature, can be easily introduced. This L0 regularization is expected to improve generalization performance.

[0049] Furthermore, L0 regularization allows for the explicit specification of the number of features to select, making it clear which features are dominant in the learned model. This has the advantage of improving the interpretability of the model.

[0050] Therefore, in this embodiment, the binary variable introduction unit 130 is w in equation (5). i (1) and w ij (2) A 0-1 binary variable (I) to control i ,I ij ) is introduced. This 0-1 binary variable (I i ,I ij The model incorporating ) is represented by equation (10) below.

[0051]

number

[0052] The objective function generation unit 140 then generates an objective function, which is the target of optimization (minimization) using the Ising model, based on the learning model in which binary variables have been introduced. This regularization objective function F is expressed by equation (11) below.

[0053]

number

[0054] In equation (11), the second term on the right-hand side is a penalty function that represents the constraint of selecting M features from m(m+1) / 2 effective features. Also, A in the second term on the right-hand side is a hyperparameter that adjusts the strength of the penalty.

[0055] Let's explain the details of item D. The third term on the right-hand side of equation (11) has the following meaning. Since the range of the sigmoid function in equation (2) is [0,1], the argument f of its exponential function FM If (x) is positive, the output of the sigmoid function will be greater than 1 / 2, predicting the label "+1". Conversely, if the argument f FM If (x) is negative, the output of the sigmoid function will be less than 1 / 2, predicting the label "0". Therefore, the third term on the right-hand side is (2y l k -1)f FM (x,I i ,I ij This term takes a positive value when the ground truth label and the predicted label match, and decreases the value of the objective function F. On the other hand, when the ground truth label and the predicted label do not match, this term takes a negative value, and increases the value of the objective function F. In other words, the third term on the right-hand side of equation (11) works to improve the prediction accuracy of the model.

[0056] [Explanation of operation] Next, the operation of the learning device will be described. Figure 2 is a flowchart illustrating the operation of the learning device. Note that the operation example shown in Figure 2 is not limited to the operation of the learning device 100 according to this disclosure.

[0057] The learning device 100 takes the data to be classified as input and divides it into training data and test data (step S110). Specifically, the data adjustment unit 110 takes the data to be classified as input and converts the categorical variables into one-hot vectors for optimization using FM learning or the Ising model. After that, the data adjustment unit 110 randomly divides the entire data into training data and test data.

[0058] Next, the learning device 100 constructs a learning model using FM (step S120). Specifically, the learning model construction unit 120 uses the training data to construct a binary classification learning model by combining logistic regression and FM.

[0059] Next, the learning device 100 constructs an Ising model including correction terms (step S130). Specifically, the binary variable introduction unit 130 introduces 0-1 binary variables (I) representing selection or non-selection for each of the first and second quadratic terms of the constructed learning model. i ,I ij The objective function generation unit 140 then generates an objective function, which is the target of optimization (minimization) using the Ising model, based on the learning model in which binary variables have been introduced. At this time, the objective function generation unit 140 introduces a correction term into the objective function that reflects the classification result (success / failure) of the L0 regularized learning model. Through the processing of step S130, the problem of regularizing the constructed learning model is converted into a combinatorial optimization problem described in QUBO format, which corresponds to the Ising model.

[0060] Next, the learning device 100 performs optimization by simulated annealing (step S140). Specifically, the solution unit 150 solves the transformed combinatorial optimization problem and outputs the solution. For example, the solution unit 150 outputs the solution to the solved combinatorial optimization problem as the feature selection result (i.e., a selection result indicating the selection or non-selection of a feature (term)) for the features included in the constructed learning model.

[0061] Next, an overview of the learning device related to this disclosure will be described. Figure 3 is an explanatory diagram illustrating an overview of the operation of the learning device. Note that Figure 3 is an explanatory diagram intended to facilitate understanding of the overview of the operation of the learning device. Therefore, the configuration and operation of the learning device are not limited to those shown in Figure 4. Also, the arrows in Figure 3 simply indicate the direction of signal (data) flow, but do not exclude bidirectionality. This is also true for other figures.

[0062] As shown in Figure 3, the processing performed by the learning device 100 can be divided into three blocks: a data adjustment block, an FM learning block, and an L0 regularization block.

[0063] In the data adjustment block, the data adjustment unit 110 receives the data to be classified. Next, the data adjustment unit 110 converts the categorical variables into one-hot vectors for building a learning model using FM or optimization using the Ising model. The data adjustment unit 110 also randomly divides the entire dataset into training data and test data.

[0064] In the FM learning block, the learning model construction unit 120 uses the above training data to determine a sigmoid function with quadratic arguments. Through learning based on FM, the learning model construction unit 120 determines the coefficients (w) of each term in the quadratic equation. (0) , w (1) The , v) is obtained. Here, the learning model construction unit 120 introduces a hyperparameter α (gain) that adjusts the overall size of the arguments.

[0065] In the L0 regularization block, the binary variable introduction unit 130 introduces a 0-1 binary variable (I) corresponding to the selection or non-selection of each first- and second-order term included in the FM model. i ,I ijThe objective function generation unit 140 generates an objective function, which is the target of optimization (minimization) using the Ising model, based on the learning model into which binary variables have been introduced. At this time, the objective function generation unit 140 introduces a penalty function to set the total number of features (terms) to be selected to M, and adjusts its strength with hyperparameter A. Furthermore, the objective function generation unit 140 introduces a correction term that reflects the classification result (success / failure) of the L0 regularized classification model, and sets hyperparameter B to adjust the strength of this correction term. The solution unit 150 calculates the combination of binary variable values ​​that minimizes the generated objective function and outputs the calculation result (i.e., the selection result indicating the selection or non-selection of features (terms)).

[0066] [Explanation of effects] Next, the effects of this embodiment will be described. In this embodiment, the binary variable introduction unit 130 introduces a 0-1 binary variable (I) representing selection or non-selection for each of the first and second terms of the constructed learning model. i ,I ij The objective function generation unit 140 introduces an objective function that is the target of optimization (minimization) using the Ising model, based on the learning model in which binary variables have been introduced. Specifically, the objective function generation unit 140 converts the problem of regularizing the constructed learning model into a combinatorial optimization problem described in QUBO format. This combinatorial optimization problem involves minimizing the objective function using 0-1 binary variables (I i ,I ij This is a problem of selecting a combination of ). The solution unit 150 solves the combinatorial optimization problem using methods such as simulated annealing or quantum annealing, and outputs the solved solution to the combinatorial optimization problem. For example, the solution unit 150 outputs the solved solution to the combinatorial optimization problem as the result of feature selection. With this configuration, a binary classification learning model optimized using the Ising model can be realized.

[0067] Furthermore, in this embodiment, the learning model construction unit 120 constructs a binary classification learning model using logistic regression and FM. The learning model construction unit 120 also uniformly adjusts all arguments of the exponential function of logistic regression with the hyperparameter α (gain). With this configuration, classification accuracy can be improved, as shown in Example 2 described later.

[0068] Furthermore, in this embodiment, the objective function generation unit 140 generates an objective function that includes a correction term that reflects the classification result of the L0-normalized learning model. This correction term also includes a hyperparameter B whose strength can be adjusted. With this configuration, classification accuracy can be improved, as shown in Example 4 described later.

[0069] In this embodiment, the data adjustment unit 110 receives data to be classified that has multiple features, converts the categorical variables into one-hot vectors, and splits the data into training data and test data. The solution unit 150 outputs the solution to the solved combinatorial optimization problem as the feature selection result for the features included in the learning model constructed using the training data. With this configuration, a binary classification learning model can be constructed using the input data to be classified, and optimized using the Ising model. [Examples]

[0070] In this embodiment, the Titanic dataset, widely known as a Kaggle tutorial (see, for example, reference 4 below), was used. The dependent variable of this dataset is the survival status of passengers (0: deceased, 1: alive), and it has 13 explanatory variables (features). However, one of the features, the port of departure (Cherbourg, Queenstown, Southampton), was converted into a one-hot vector, so the number of features became 18 at this point. After removing missing values, the total number of data points became 358. This data was randomly split into training data and test data in a 7:3 ratio. In the learning process, mini-batch learning (batch size of 32) was used, and stochastic gradient descent was applied with a learning rate of 0.02. Note that the learning process by the learning device 100 of this disclosure is not limited to this method. Reference 4: "Titanic - Machine Learning from Disaster", [online], [Retrieved December 4, 2024], Internet<URL: https: / / www.kaggle.com / c / titanic / data>

[0071] Example 1. In Example 1, we compare the proposed method (i.e., the processing performed by the learning device 100) with a general method. In this example, logistic regression was used as a general method for solving a two-class classification task using machine learning. When we examined the classification accuracy, the accuracy for the training data was 84.00%, and the accuracy for the test data was 77.78%. This result is with L2 regularization applied. On the other hand, without L2 regularization, the accuracy for the training data was 83.60%, and the accuracy for the test data was 76.85%. Here, if we define generalization performance as the accuracy on the test data, since the number of features is a relatively small value of 18, it can be seen that although an improvement in generalization performance due to L2 regularization was observed, the effect was not significant. As an evaluation criterion for the proposed method in this disclosure, we use the accuracy on the test data when L2 regularization is considered.

[0072] Example 2. In Example 2, we adjust the hyperparameters. The proposed method in this disclosure includes several hyperparameters, but the most representative are α (gain) in equation (2) and A and B in equation (11). Hyperparameter A adjusts the strength of the constraint term that fixes the number of features to be selected to a specific value M. Normally, setting A=1 satisfies this constraint. However, if this condition is not met, A is increased by a factor of two each time (for example, increasing A in the order of A=1 → 2 → 4). The value of M will be examined in Example 3. Hyperparameter B adjusts the strength of the correction term that is introduced with the expectation of improving the accuracy. The value of this term will be examined in Example 4, and in this example, we will focus on α.

[0073] Figure 4 is an illustrative diagram illustrating the effect of adjusting the hyperparameter α of the sigmoid function on accuracy. The table in Figure 4 shows the accuracy obtained using the FM model before applying L0 regularization and the accuracy obtained after applying L0 regularization using the Ising model, for both the training data and the test data, for each case where α = {0.25, 0.5, 1, 2, 4}. Note that parameters other than α were fixed at M=10, A=1 (except when α=0.25, in which case A=4) and B=1.

[0074] Comparing the accuracy of the test data between the original FM model and the model after L0 regularization, it can be seen that L0 regularization significantly improved the accuracy, except for α=1. In Example 1, the improvement effect of regularization was small, but the reason for the significant improvement here is thought to be that because FM includes the intersection term as a feature, the number of features increased from the original 18 to 153, making the effect of regularization more pronounced. In the results shown in Figure 4, the cases of α=0.25 and α=0.5 are particularly noteworthy. The accuracy rates were 80.56% and 79.63%, respectively, exceeding the accuracy of the test data for a general logistic regression considering L2 regularization in Example 1 (77.78%). These results demonstrate the effectiveness of the hyperparameter α in the method proposed in this disclosure.

[0075] Example 3. In Example 3, we examined the relationship between the number of features (M) selected by L0 regularization and the accuracy rate. Here, we examined the cases where α=0.25 and α=0.5 yielded good accuracy rates, as shown in the table in Figure 4, which was explained in Example 2. The value of A was set to A=4 (α=0.25), A=1 (α=0.5, M≧10), or A=2 (α=0.5, M=8 or 6). The value of B was always fixed at 1.

[0076] Figure 5 is an explanatory diagram illustrating the relationship between the number of features (M) selected in L0 regularization and the accuracy. The table in Figure 5 shows the accuracy for the training data and the test data. The results obtained for α=0.25 and α=0.5 were similar, and the accuracy for the test data was particularly good when M was 30 or less. Furthermore, as mentioned in Example 2, it was confirmed that the accuracy exceeded that of a general logistic regression model (77.78%). Since the original data has 18 features without considering the intersection term, when M=10 or less, it was possible to construct a model with accuracy equivalent to or better than a general classification model using approximately half the number of features. As a result, it is expected that the interpretability of the model will improve as the number of features decreases.

[0077] Example 4. In Example 4, the effect of the correction term was examined. Figure 6 is an explanatory diagram illustrating the effect of the correction term in equation (11) on the accuracy rate. The last term on the right-hand side of equation (11) is the correction term introduced with the expectation of improving the accuracy rate. In this example, the cases of not considering this correction term (B=0) and the cases of varying the strength of the correction term (B=0.5 or 1) were examined. The accuracy rates for training data and test data were compared using several representative combinations of α and M. Figure 6 shows the comparison results. The quantity of particular note is the accuracy rate for the test data. The higher accuracy rate when the correction term is considered (B=0.5 or 1) is equal to or higher than the accuracy rate when the correction term is not considered (B=0). From this result, the effectiveness of the correction term was confirmed.

[0078] As described above, the learning device 100 of the above embodiment can realize a binary classification learning model optimized using the Ising model. Therefore, in the above embodiment, it was confirmed that a binary classification learning model optimized using the Ising model can be realized, which has accuracy comparable to or better than that of general methods.

[0079] Each component in the above embodiments and examples can be configured with one piece of hardware, but can also be configured with one piece of software. Furthermore, each component can be configured with multiple pieces of hardware, or with multiple pieces of software. In addition, some parts of each component can be configured with hardware, and other parts with software.

[0080] Each function (each process) in the above embodiment can be implemented by a computer having a processor, memory, etc. For example, a program for implementing the method (process) in the above embodiment may be stored in a storage device (storage medium), and each function may be implemented by executing the program stored in the storage device with a processor.

[0081] Figure 7 is a block diagram illustrating the hardware configuration of computer 1000. Computer 1000 is any computer. For example, computer 1000 is a stationary computer such as a personal computer or a server machine. Alternatively, computer 1000 is a portable computer such as a smartphone or a tablet device. Computer 1000 may be a dedicated computer designed to implement a signal processing device or signal processing system, or it may be a general-purpose computer.

[0082] Computer 1000 has a processor 1001, a storage device 1002, memory 1003, a bus 1004, an input / output interface 1005, and a network interface 1006.

[0083] Processor 1001 is a variety of processing units, including CPUs (Central Processing Units), GPUs (Graphics Processing Units), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0084] The storage device 1002 is, for example, a non-transitory computer-readable medium. Non-transitory computer-readable media include various types of tangible storage media. Specific examples of non-transitory computer-readable media include semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM).

[0085] Memory 1003 is a main memory system implemented using RAM (Random Access Memory) or similar technologies. Memory 1003 temporarily stores data when the processor 1001 executes processing.

[0086] Bus 1004 is a data transmission path for the processor 1001, memory 1003, storage device 1002, input / output interface 1005, and network interface 1006 to send and receive data to and from each other. However, the method of connecting the processor 1001 and the others to each other is not limited to bus connection.

[0087] The input / output interface 1005 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1005.

[0088] The network interface 1006 is an interface for connecting computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0089] The storage device 1002 stores a program that implements each of the functional components in the above-described embodiments and examples. The processor 1001 reads this program into the memory 1003 and executes it to implement each of the functional components in the above-described embodiments and examples.

[0090] The learning device 100 may be implemented using one computer 1000 or multiple computers 1000. In the latter case, the configuration of each computer 1000 does not need to be the same and can be different.

[0091] Each functional component in the above embodiments and examples may be implemented by a combination of the hardware and software described above, or by hardware (for example, a hardwired electronic circuit).

[0092] Next, an overview of this disclosure will be given. Figure 8 is a block diagram illustrating the main components of the learning device. The learning device 10 shown in Figure 8 (corresponding to, for example, the learning device 100) includes a regularization means 11 (in this embodiment, realized by, for example, a binary variable introduction unit 130, an objective function generation unit 140, and a solution unit 150) that performs L0 regularization of features included in a binary classification learning model constructed using logistic regression and a Factorization Machine, using an Ising model. With this configuration, a binary classification learning model optimized using an Ising model can be realized.

[0093] The learning device 10 shown in Figure 8 can be configured to include a learning means (in this embodiment, for example, implemented by a learning model construction unit 120) that constructs a binary classification learning model using logistic regression and a factorization machine. This learning means uniformly adjusts the entire set of arguments of the exponential function of logistic regression using hyperparameters (in this embodiment, corresponding to hyperparameter α). With this configuration, classification accuracy can be improved, as shown in Example 2.

[0094] The regularization means 11 can perform L0 regularization of features using an objective function that includes a correction term that reflects the classification result of the L0-normalized learning model. This correction term also includes a hyperparameter (for example, corresponding to hyperparameter B) whose strength can be adjusted. With this configuration, classification accuracy can be improved, as shown in Example 4.

[0095] Furthermore, the learning device 10 shown in Figure 8 can be configured to include an adjustment means (implemented, for example, by a data adjustment unit 110 in this embodiment) that takes data to be classified having multiple features as input, converts the categorical variables into one-hot vectors, and splits the data into training data and test data. The regularization means 11 introduces binary variables representing selection or non-selection to each of the first and second terms of the binary classification learning model constructed using the training data. Then, the regularization means 11 generates an objective function based on the learning model into which the binary variables have been introduced. Furthermore, the regularization means 11 solves a combinatorial optimization problem, which is the problem of selecting a combination of binary variables that minimizes the objective function described in QUBO format. After that, the regularization means 11 outputs the solution to the solved combinatorial optimization problem as the feature selection result for the features included in the constructed learning model. With this configuration, a binary classification learning model can be constructed using the input data to be classified and optimized using the Ising model.

[0096] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the embodiments and examples described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Each embodiment and example can be combined with other embodiments and examples as appropriate.

[0097] Each drawing is merely illustrative to illustrate one or more embodiments or examples. Each drawing may be associated with one or more other embodiments or examples, rather than being associated with only one specific embodiment or example. As those skilled in the art will understand, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create embodiments that are not explicitly illustrated or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0098] The above embodiments and examples, in whole or in part, may also be described as follows, but are not limited to the following.

[0099] (Note 1) This system includes a regularization method that performs L0 regularization of features included in a binary classification learning model constructed using logistic regression and a factorization machine, using an Ising model. A learning device characterized by the following features.

[0100] (Note 2) The regularization means introduces binary variables representing selection or non-selection for each of the primary and quadratic terms of the constructed binary classification learning model, and performs L0 regularization of the features by determining the optimal combination of binary variables using the Ising model. The learning device described in Appendix 1.

[0101] (Note 3) The regularization means is, Based on a learning model that incorporates binary variables, an objective function is generated. We solve a combinatorial optimization problem, which is the problem of selecting a combination of binary variables that minimizes the objective function described in the QUBO (Quadratic Unconstrained Binary Optimization) format. Output the solution to the solved combinatorial optimization problem. The learning device described in Appendix 2.

[0102] (Note 4) It includes an adjustment mechanism that takes data to be classified, which has multiple features, as input, converts the categorical variables into one-hot vectors, and splits the data into training data and test data. The regularization means outputs the solution to the solved combinatorial optimization problem as the feature selection result for the features included in the binary classification learning model constructed using the training data. The learning device described in Appendix 3.

[0103] (Note 5) It includes a learning method for constructing a binary classification learning model using logistic regression and factorization machine. The learning method uniformly adjusts all arguments of the exponential function of logistic regression using hyperparameters. A learning device as described in any of the appendices 1 through 4.

[0104] (Note 6) The learning method sets the argument portion of the exponential function of logistic regression into a quadratic learning model using a Factorization Machine, and uniformly adjusts the size of the entire quadratic learning model using hyperparameters. The learning device described in Appendix 5.

[0105] (Note 7) The regularization means performs L0 regularization of features using an objective function that includes a correction term that reflects the classification result of the L0-normalized learning model. A learning device as described in any of the appendices 1 through 4.

[0106] (Note 8) The correction term includes a hyperparameter that allows adjustment of the strength of the correction term. The learning device described in Appendix 7.

[0107] (Note 9) Computers L0 regularization of features in a binary classification learning model constructed using logistic regression and a factorization machine is performed using an Ising model. A learning method characterized by the following:

[0108] (Note 10) On the computer, This regularization process uses the Ising model to perform L0 regularization on the features included in a binary classification learning model constructed using logistic regression and a Factorization Machine. A learning program to execute.

[0109] (Note 11) A non-temporary computer-readable recording medium on which a regularization program is stored, The regularization program is provided to the computer, This process performs L0 regularization of features in a binary classification learning model constructed using logistic regression and a Factorization Machine, using an Ising model. A non-temporary, computer-readable recording medium.

[0110] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 8 that are dependent on Appendice 1 may also be dependent on Appendices 9, 10, and 11 in the same way as those described in Appendices 2 to 8. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software. [Explanation of symbols]

[0111] 10,100 Learning devices 11 Regularization means 110 Data Adjustment Unit 120 Learning Model Construction Department 130 Introduction to Binary Variables 140 Objective Function Generation Unit 150 Solving section 1000 computers 1001 Processor 1002 Storage device 1003 memory 1004 Bus 1005 Input / Output Interface 1006 Network Interface

Claims

1. This system includes a regularization method that performs L0 regularization of features included in a binary classification learning model constructed using logistic regression and a factorization machine, using an Ising model. A learning device characterized by the following features.

2. The regularization means introduces binary variables representing selection or non-selection for each of the primary and quadratic terms of the constructed binary classification learning model, and performs L0 regularization of the features by determining the optimal combination of binary variables using the Ising model. The learning device according to claim 1.

3. The regularization means is, Based on a learning model that incorporates binary variables, an objective function is generated. We solve a combinatorial optimization problem, which is the problem of selecting a combination of binary variables that minimizes the objective function described in the form of QUBO (Quadratic Unconstrained Binary Optimization). Output the solution to the solved combinatorial optimization problem. The learning device according to claim 2.

4. It includes an adjustment mechanism that takes data to be classified, which has multiple features, as input, converts the categorical variables into one-hot vectors, and splits the data into training data and test data. The regularization means outputs the solution to the solved combinatorial optimization problem as the feature selection result for the features included in the binary classification learning model constructed using the training data. The learning device according to claim 3.

5. It includes a learning method for constructing a binary classification learning model using logistic regression and factorization machine. The learning method uniformly adjusts all arguments of the exponential function of logistic regression using hyperparameters. A learning device according to any one of claims 1 to 4.

6. The learning means sets the argument portion of the exponential function of logistic regression into a quadratic learning model using a Factorization Machine, and uniformly adjusts the size of the entire quadratic learning model using hyperparameters. The learning device according to claim 5.

7. The regularization means performs L0 regularization of features using an objective function that includes a correction term that reflects the classification result of the L0-normalized learning model. A learning device according to any one of claims 1 to 4.

8. The correction term includes a hyperparameter that allows adjustment of the strength of the correction term. The learning device according to claim 7.

9. Computers L0 regularization of features in a binary classification learning model constructed using logistic regression and a factorization machine is performed using an Ising model. A learning method characterized by the following:

10. On the computer, This regularization process uses the Ising model to perform L0 regularization on the features included in a binary classification learning model constructed using logistic regression and a Factorization Machine. A learning program to execute.

Images