Information processing device, prediction method, and program

Abstract

One aspect of the present invention pertains to an information processing device that includes: a prediction model that predicts a response to input data; and a learning unit that learns parameters of the prediction model. The prediction model comprises: an embedding unit that performs conversion into a latent vector on the basis of each feature of the input data; a latent vector updating unit that updates the latent vector in consideration of at least one of a plurality of test results and a plurality of the features; and a prediction unit that predicts a response to the input data on the basis of the updated latent vector.

Description

Information processing device, prediction method, and program

[0001] This invention relates to information processing equipment, prediction methods, and programs.

[0002] Numerous methods have been proposed for training predictive models from input data (see, for example, Non-Patent Document 1). A predictive model is, for example, a model that predicts the response to input data.

[0003] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A., “Revisiting deep learning models for tabular data', Advances in Neural Information Processing Systems, 34, 18932-18943,2021

[0004] However, conventional technology has a problem in that it cannot predict when there is data on multiple test results that include missing values, because it is not possible to train the predictive model. Note that "tests" refer to, for example, health checkups for users. Furthermore, "multiple test results" refer to the results of health checkups for the same person over several years. In view of the above circumstances, the present invention aims to provide a technology that can predict with high accuracy even when there is data on multiple test results that include missing values.

[0005] One aspect of the present invention is an information processing device comprising: a prediction model for predicting the response of input data; a learning unit for learning the parameters of the prediction model, wherein the prediction model comprises: an embedding unit for converting each feature of the input data into a latent vector; a latent vector update unit for updating the latent vector considering at least one of a plurality of the features and a plurality of test results; and a prediction unit for predicting the response of the input data from the updated latent vector.

[0006] One aspect of the present invention is a prediction method for an information processing device comprising a prediction model for predicting the response of input data and a learning unit for learning the parameters of the prediction model, wherein the prediction model converts each feature of the input data into a latent vector, updates the latent vector considering at least one of a plurality of such features and a plurality of test results, and predicts the response of the input data from the updated latent vector.

[0007] One aspect of the present invention is a program for an information processing device comprising a predictive model for predicting the response of input data and a learning unit for learning the parameters of the predictive model, which causes the computer of the information processing device to convert each feature of the input data into a latent vector, update the latent vector considering at least one of a plurality of the features and a plurality of test results, and predict the response of the input data from the updated latent vector.

[0008] This invention makes it possible to make predictions with high accuracy even when there is data on multiple test results that include missing values.

[0009] This figure shows an example configuration of the information processing device of the embodiment. This figure shows an example configuration of the prediction model of the embodiment. This is a flowchart of the learning procedure of the embodiment. This is a flowchart of the response prediction procedure of the embodiment. This figure shows a schematic of the hardware configuration example of the information processing device 1 applied to the embodiment. This figure shows an example of evaluation results.

[0010] Embodiments of the present invention will be described in detail with reference to the drawings. Figure 1 is a diagram showing an example of the configuration of an information processing device of this embodiment. As shown in Figure 1, the information processing device 1 includes, for example, an acquisition unit 2, a prediction model 3, and a learning unit 4. The prediction model 3 includes, for example, an embedding unit 31, a latent vector update unit 32, and a prediction unit 33.

[0011] The information processing device 1 trains a prediction model 3 using the input data. The information processing device 1 uses the trained prediction model 3 to predict the response to the input data.

[0012] The acquisition unit 2 acquires the input data, which consists of a feature vector x and a mask vector m, from an external device. Alternatively, the acquisition unit 2 may acquire the data and generate the feature vector x and mask vector m from the acquired data.

[0013] Predictive model 3 is trained by the learning unit 4. Predictive model 3 predicts the response to the input data.

[0014] The embedding unit 31 converts the data for each feature of the input data into a latent vector.

[0015] The latent vector update unit 32 updates the latent vector while considering at least one of multiple features and multiple test results. The test results refer to, for example, the results of performing multiple tests on each of the multiple input data.

[0016] The prediction unit 33 predicts the response to the input data from the updated latent vector. The procedures for the learning method and the prediction method will be described later.

[0017] The learning unit 4 trains the prediction model 3 by training it with the parameters of the prediction model 3.

[0018] Next, an example of the configuration of the prediction model 3 will be described. Figure 2 is a diagram showing an example of the configuration of the prediction model in this embodiment. As shown in Figure 2, the latent vector update unit 32 and the prediction unit 33 of the prediction model 3 are composed of, for example, multiple (e.g., L) layers. The latent vector update unit 32 and the prediction unit 33 of the prediction model 3 are, for example, neural networks.

[0019] Next, an example of a learning method using the information processing device 1 will be explained. The learning data provided is learning data D (in cursive) concerning N users (equation (1) below).

[0020]

[0021] In equation (1), x ｎｔ The following equation (2) is the feature vector for the t-th test of the n-th user. Also, y ｎｔ The following equation (3) is the response of the tth test of the nth user. Also, Tｎ represents the number of inspections of the n-th user, I represents the feature number, and J represents the number of responses.

[0022]

[0023]

[0024] This data contains missing values. Information regarding the missing value of the t-th inspection of the n-th user is m if it is an observed location, ｎｔｉ m = 1, and m ｎｔｉ = 0 if it is a missing value. Also, the mask vector m ｎｔ is expressed by the following equation (4).

[0025]

[0026] The learning unit 4 uses this data to train the prediction model 3. Also, the learning unit 4 uses the learning data D (handwritten) to learn the parameters Θ of the prediction model 3 so that the prediction performance of the response becomes high. Examples of the parameters Θ of the prediction model 3 include the linear projection matrix of the embedding unit 31, the bias, the Transformer of the latent vector update unit 32, and the parameters of the neural network of the prediction unit 33.

[0027] The learning unit 4 obtains the parameters Θ so that the prediction performance in the learning data becomes high. For example, when L(y ｎｔ , y^ ｎｔ ) is defined as a quantity representing the difference between the observed location y ｎｔ and the predicted value y^ ｎｔ , the parameters Θ are obtained by minimizing L as shown in the following equation (5).

[0028]

[0029] For example, when the response is a continuous value, squared error or the like can be used as L, and when the response is a categorical value, cross-entropy loss or the like can be used. Also, when there are missing values in the response, the loss may be calculated excluding the missing values. Note that by pseudo-creating missing data and using it as learning data during learning, robust prediction against missing values becomes possible. Also, by learning so that the prediction accuracy for responses other than the target response is also high, the prediction accuracy for the target response can be increased. For example, when the target response is y ｎｔ and the response other than the target (y ｎｔ ) Ｏ is defined by the following equation (6), equation (6) can be used instead of equation (5).

[0030] [[ID=IO]]

[0031] In equation (6), λ is given by the following equation (7) and is a weight parameter. As the response other than the target, pseudo-created missing values can be used. This optimization can be executed, for example, by using the stochastic gradient method in the process of FIG. 4 described below.

[0032]

[0033] Next, an example of the learning procedure by the information processing apparatus 1 will be described. FIG. 3 is a flowchart of the learning procedure of the present embodiment.

[0034] (Step S1) The learning unit 4 acquires learning data D (notebook).

[0035] (Step S2) The learning unit 4 initializes the parameters Θ of the prediction model

[0036] (Step S3) The learning unit 4 randomly selects mini-batch data from the learning data D (notebook). Note that mini-batch data is data used for training during learning. For example, the data is randomly divided into several groups, the loss function for each group is calculated, and the average is calculated, which is the data used when proceeding with learning.

[0037] (Step S4) The learning unit 4 randomly generates missing values.

[0038] (Step S5) The learning unit 4 predicts the response for each data point in the mini-batch using the prediction model 3.

[0039] (Step S6) The learning unit 4 calculates the loss of the predicted response and the observed response using equation (6).

[0040] (Step S7) The learning unit 4 updates the parameters Θ of the prediction model 3 using the gradient of the loss.

[0041] (Step S8) The learning unit 4 determines whether the update termination conditions are met. The update termination conditions include, for example, that steps S3 to S7 have been performed a predetermined number of times, or that the loss has fallen within a predetermined value. If the update termination conditions are met (Step S8; Yes), the learning unit 4 terminates the learning process. If the update termination conditions are not met (Step S8; No), the learning unit 4 returns to step S3.

[0042] The learning procedure described above is just one example, and other processes may also be used.

[0043] Next, we will explain how to predict the response using the pre-trained prediction model 3. Given a feature vector x containing missing values ​​and a mask vector m, prediction model 3 predicts the response y^.

[0044] Although the term "user" is used for simplicity in explanation during learning and prediction, it is not limited to user data. The method of this embodiment can be applied to learning data or prediction targets that only have one test. Furthermore, the method of this embodiment can be applied to non-vector features and responses by converting them to vectors using a neural network or the like. In addition, time information for each test can be included in the features. Also, missing data is acceptable in the response. The response of the prediction target may be a future state of that user. In this case, for example, by inputting the future time to be predicted into the features, future predictions at any time become possible.

[0045] First, let's explain the processing performed by the embedding unit 31. The embedding unit 31 converts the data for each feature for each iteration into a latent vector, while also considering information about missing data. For example, in the case of a continuous feature, the embedding unit 31 converts the following equation (8) and feature xｎｔi (Equation (9) below) and mask m ｎｔi The vector obtained by concatenating (Equation (10)) is the linear projection matrix W. ｉ (Equation (10) below) and bias term wO i By performing the transformation using (equation (11)) below, it is converted into a latent vector.

[0046]

[0047]

[0048]

[0049]

[0050] Furthermore, in the case of categorical features, the embedding section 31 converts them into latent vectors by assigning a vector to each category, including missing data, using the following equation (12).

[0051]

[0052] Note that in equation (12), e ｉｃ This corresponds to the latent vector in the case of missing values. Also, e ｉｃ (Equation (13) below) is the latent vector corresponding to category c. Note that the embedding section 31 may embed the response as a missing feature in the latent vector.

[0053]

[0054] Next, the processing performed by the latent vector update unit 32 will be explained. The latent vector update unit 32 updates the latent vector while considering multiple features and the results of multiple checks. When updating while considering the results of multiple checks, a Transformer can be used, for example. A Transformer is a deep learning model that uses an encoder and a decoder (see, for example, Reference 1).

[0055] Reference 1: Ashish Vaswani, Noam Shazeer, et al., “Attention Is All You Need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017.

[0056] Here, (z ｎｔｉ ) ｌ Let (equation (14)) be the latent vector of the i-th feature of the t-th test for the n-th user in the l-th layer (g12 in Figure 2).

[0057]

[0058] Then, the latent vectors of the l-th feature for the n-th user, multiple times, are compiled into a matrix (Z ｎ・ｉ ) ｌ (Equation (15) below)

[0059]

[0060] The latent vector update unit 32 updates the previous layer (for example, layer l-1 (g11) in Figure 2) using Transformer as shown in equation (16).

[0061]

[0062] Furthermore, the latent vector update unit 32 can be configured not to use information regarding future test results. The latent vector update unit 32 can also similarly use Transformer as a method for updating while considering multiple features. For example, the latent vector update unit 32 might combine the latent vectors of multiple features from the t-th test of the n-th user into a matrix (Z ｎ・ｔ ) ｌ (Equation (17) below) Then, the latent vector update unit 32 updates the previous layer using Transformer as shown in Equation (18) below.

[0063]

[0064]

[0065] The latent vector update unit 32 may use any function that can transform the output according to the latent vector being considered, such as a feedforward neural network, a recurrent neural network, or a neural network that can input and output sets, in addition to Transformer. In this case, the latent vector update unit 32 alternately repeats updates that consider the results of multiple tests and updates that consider multiple features. The latent vector update unit 32 may consider them in any order, not just alternately, or simultaneously. Although the dimension K of the latent vector was set to be the same for all layers, it may be set to a different dimension for each layer.

[0066] Next, the processing performed by the prediction unit 33 will be explained. The prediction unit 33 predicts the response to the input data from the updated latent vectors. For example, the prediction unit 33 predicts the response to the input data using a feedforward neural network f that takes as input the latent vectors of all features of the final layer (the Lth layer) (g14 in Figure 2), as shown in equation (19).

[0067]

[0068] In equation (19), y^ ｎｔ f is the predicted value of the response to the t-th test for the n-th user. Note that f may be a neural network other than a feedpropagating type or a linear model. Also, latent vectors of other tests may be included as input to f. Furthermore, if there is a latent vector corresponding to the response, that may also be included. In addition, if there is information about the response, that may also be included. For example, when making a future prediction, information about that time can be included.

[0069] By performing the above-described processing, the embedding unit 31, the latent vector update unit 32, and the prediction unit 33 can predict the predicted value y^ of the response when given a feature vector x containing missing values ​​and a mask vector m, for example, using the processing method shown in Figure 4. Figure 4 is a flowchart of the response prediction procedure in this embodiment.

[0070] (Step S11) The acquisition unit 2 acquires the input data, which is the feature vector x and the mask vector m, from the external device, and inputs the acquired feature vector x and mask vector m to the prediction model 3.

[0071] (Step S12) The embedding unit 31 obtains a latent vector (equation (20)) from the feature vector x and the mask vector m.

[0072]

[0073] (Step S13) The latent vector update unit 32 sets l to 1.

[0074] (Step S14) The latent vector update unit 32 obtains the latent vector of the next layer (equation (22)) from the latent vector of the previous layer (equation (21)) by considering the results of multiple tests (equation (17)) or by considering multiple features (equation (18)).

[0075]

[0076]

[0077] (Step S15) The latent vector update unit 32 determines whether l = L (processing completed up to the final layer). If l = L (processing completed up to the final layer) (Step S15; Yes), the latent vector update unit 32 proceeds to step S17. If l = L (processing completed up to the final layer) is not the case (Step S15; No), the latent vector update unit 32 proceeds to step S16.

[0078] (Step S16) The latent vector update unit 32 adds 1 to l and returns to the process in step S14.

[0079] (Step S17) The prediction unit 33 calculates the predicted response value y^ from the latent vector of the final layer, for example, using equation (19).

[0080] The prediction procedure described above is just one example, and other processes may also be used.

[0081] The information processing device 1 is composed of a processor such as a CPU (Central Processing Unit) and memory. The information processing device 1 functions as an acquisition unit 2, a prediction model 3, and a learning unit 4 when the processor executes a program. All or part of each function of the information processing device 1 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor memory devices (e.g., SSDs: Solid State Drives), and memory devices such as hard disks and semiconductor memory devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0082] Figure 5 is a schematic diagram of an example hardware configuration of the information processing device 1 applied to this embodiment. The information processing device 1 includes, for example, a processor 101, a main memory 102, a communication interface 103, an auxiliary storage device 104, an input / output interface 105, and an internal bus 106. The processor 101, the main memory 102, the communication interface 103, the auxiliary storage device 104, and the input / output interface 105 are connected to each other via the internal bus 106 so as to be able to communicate with each other. For example, the acquisition unit 21 may be configured using the communication interface 103 or the input / output interface 105. For example, the prediction model 3 may be configured using the auxiliary storage device 104. In addition, the acquisition unit 21 and the learning unit 4 may be configured using the processor 101 and the main memory 102.

[0083] Next, we will explain an example of the evaluation results. Figure 6 shows an example of the evaluation results. The evaluation was performed using artificial data. For the evaluation, data with 100 users, 20 features, 10 tests, 5 responses, and a missing data rate of 0.7 was created using a matrix decomposition model with a latent dimension of 3. In the evaluation, 70% of the user data was used for training, 10% for validation, and 20% for use. Furthermore, linear models, feedforward neural networks, FR-Transformer (see Non-Patent Literature 1), and gradient boosting were used as comparative examples in the evaluation. Note that each value in Figure 6 is the squared error value, which is the evaluation result; a smaller value is better.

[0084] As shown in Figure 6, the prediction method of this embodiment can achieve a lower test squared error compared to the linear model, feedforward neural network, FT-Transformer, and gradient boosting of the comparative examples.

[0085] In the information processing device 1 configured in this way, the data for each feature from each test is converted into a latent vector, taking into account whether or not the data is missing, and the latent vector is updated while considering multiple features and multiple test results. Furthermore, the information processing device 1 outputs a predicted response value from the updated latent vector and uses the training data to learn the model parameters so that the response prediction performance is improved. In this embodiment, by performing the above processing, it becomes possible to train a predictive model even when there is data on multiple test results that include missing values. As a result, this embodiment can achieve high predictive performance when there is data on multiple test results that include missing values. In addition, according to this embodiment, for example, it is possible to estimate a person's condition (which is not included in the health checkup) from the results of health checkups over the past several years. Note that in the example above, a health checkup was described as an example of an examination, but it is not limited to this. Other examinations may also be used.

[0086] (Modification) In the information processing device 1, the prediction model 3 may be located in the cloud or run on the cloud. Furthermore, the prediction model 3 may be updated even after training.

[0087] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0088] The present invention is applicable to information processing devices, learning devices, prediction devices, and the like.

[0089] 1... Information processing device, 2... Acquisition unit, 3... Prediction model, 4... Learning unit, 31... Embedding unit, 32... Latent vector update unit, 33... Prediction unit

Claims

1. An information processing device comprising: a prediction model for predicting the response of input data; a learning unit for learning the parameters of the prediction model, wherein the prediction model comprises: an embedding unit for converting each feature of the input data into a latent vector; a latent vector update unit for updating the latent vector considering at least one of a plurality of the features and a plurality of test results; and a prediction unit for predicting the response of the input data from the updated latent vector.

2. The information processing apparatus according to claim 1, wherein the embedding unit converts the input data into a latent vector while considering whether or not there is missing data.

3. A prediction method for an information processing device comprising a prediction model for predicting the response of input data and a learning unit for learning the parameters of the prediction model, wherein the prediction model converts each feature of the input data into a latent vector, updates the latent vector considering at least one of a plurality of the features and a plurality of test results, and predicts the response of the input data from the updated latent vector.

4. A program for an information processing device comprising a predictive model for predicting the response of input data and a learning unit for learning the parameters of the predictive model, which causes the computer of the computer to convert each feature of the input data into a latent vector, update the latent vector considering at least one of a plurality of the features and a plurality of test results, and predict the response of the input data from the updated latent vector.

Images