Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program that accurately correct predicted values of items with high predicted values to reduce bias.SOLUTION: A processor is configured to: acquire verification data for verifying the relation between a predicted value output from a predictor that predicts a user's rating on a candidate item and the true value of the user's rating on the candidate item; input the verification data to the predictor, and acquire higher level items, which are candidate items with relatively higher rankings of the output predicted values; extract higher level data corresponding to the higher level items from the verification data; and train a corrector to correct the input predicted value so that the input predicted value becomes closer to the true value based on the higher level data.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to a technique for reducing bias in a predicted value. [Background technology]

[0002] 2. Description of the Related Art Recommendation systems are known that predict the rating value of an item based on a user and a context, and recommend items with high predicted rating values ​​(predicted values) to the user.

[0003] Patent document 1 describes an information processing device that includes a prediction index derivation unit that derives a prediction index that predicts an index related to each of a plurality of pieces of content that are candidates for content to be provided to a terminal device, a ranking processing unit that ranks the plurality of pieces of content based on a ranking index that is based on the prediction index derived by the prediction index derivation unit, and a prediction index correction unit that corrects the prediction index so as to reduce bias associated with the ranking. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-154488 Summary of the Invention [Problem to be solved by the invention]

[0005] Naturally, there is a prediction error in the evaluation value prediction, and some items are predicted to be higher than the true value (overestimated), while other items are predicted to be lower than the true value (underestimated).

[0006] There are many overrated items among the items with high predicted values. Because recommended items are items with high predicted values, a bias occurs that makes them prone to overrating. This leads to problems such as (1) when a predicted rating value is presented to a user, the user's expectations are raised and they are disappointed, and (2) when inventory is purchased based on a predicted purchase rate, sales are lower than expected, resulting in excess inventory.

[0007] The present invention has been made in view of the above circumstances, and aims to provide an information processing device, an information processing method, and a program that accurately correct the predicted values ​​of items with high predicted values ​​to reduce bias. [Means for solving the problem]

[0008] In order to achieve the above object, an information processing device according to a first aspect of the present disclosure is an information processing device that includes one or more processors and one or more memories that store instructions to be executed by the one or more processors, and the processor acquires verification data for verifying the relationship between a predicted value output from a predictor that predicts a user's rating for a candidate item and the true value of the user's rating for the candidate item, inputs the verification data to the predictor, acquires top items that are candidate items that are relatively higher in the ranking of the output predicted values, extracts top data corresponding to the top items from the verification data, and learns a corrector that corrects the input predicted value based on the top data so that the input predicted value is closer to the true value.

[0009] According to this aspect, the corrector that corrects the predicted value based on the higher-ranked data is trained, so that the predicted values ​​of items with higher-ranked predicted values ​​can be corrected with high accuracy, thereby reducing bias.

[0010] In an information processing device according to a second aspect of the present disclosure, in the information processing device according to the first aspect, it is preferable that the processor uses a predictor to predict a user's rating for each of a plurality of candidate items, selects a candidate item with a relatively high ranking in predicted value from the plurality of candidate items as a recommended item to be recommended to the user, corrects the predicted value of the recommended item using a corrector, and outputs the recommended item and the corrected predicted value.

[0011] An information processing device according to a third aspect of the present disclosure is the information processing device according to the first or second aspect, wherein the predictor preferably predicts the user's rating using collaborative filtering.

[0012] An information processing device according to a fourth aspect of the present disclosure is an information processing device according to any one of the first to third aspects, wherein the predictor includes a trained model, and the processor divides multiple data sets and uses a portion of the data sets for training the predictor and a remaining portion of the data sets as validation data.

[0013] In an information processing device according to a fifth aspect of the present disclosure, in an information processing device according to any one of the first to fourth aspects, it is preferable that the processor limits the top items to candidate items for which user ratings exist in the verification data.

[0014] An information processing device according to a sixth aspect of the present disclosure is the information processing device according to any one of the second to fifth aspects, wherein the processor preferably extracts top items in a number equal to the number of recommended items.

[0015] An information processing device according to a seventh aspect of the present disclosure is the information processing device according to any one of the first to sixth aspects, wherein the corrector preferably includes a parametric model or a non-parametric model in which the predicted value is an explanatory variable and the true value is a target variable.

[0016] An information processing device according to an eighth aspect of the present disclosure is the information processing device according to any one of the first to seventh aspects, wherein the corrector preferably performs correction according to the ranking of the predicted value of the top item.

[0017] An information processing device according to a ninth aspect of the present disclosure is the information processing device according to the seventh aspect, wherein the corrector preferably includes a rank of the predicted value as an explanatory variable.

[0018] In an information processing device according to a tenth aspect of the present disclosure, in the information processing device according to any one of the first to ninth aspects, it is preferable that the processor corrects the predicted value by assigning a relatively larger weight to the predicted value having a relatively higher ranking.

[0019] An information processing device according to an eleventh aspect of the present disclosure is the information processing device according to the tenth aspect, wherein the corrector is preferably trained by assigning a relatively larger weight to a predicted value having a relatively higher ranking.

[0020] In order to achieve the above object, an information processing method according to a twelfth aspect of the present disclosure is an information processing method executed by one or more processors, and includes the steps of: acquiring verification data for verifying the relationship between a predicted value output from a predictor that predicts a user's rating for a candidate item and the true value of the user's rating for the candidate item; inputting the verification data into the predictor and acquiring top items that are candidate items that have relatively higher rankings for the output predicted values; extracting top data corresponding to the top items from the verification data; and learning a corrector that corrects the input predicted value based on the top data so that the input predicted value is closer to the true value.

[0021] The information processing method of the twelfth aspect can have a configuration including the same specific aspects as those of the information processing device described above.

[0022] In order to achieve the above object, a program according to a thirteenth aspect of the present disclosure is a program that causes a computer to realize the information processing method according to the twelfth aspect. The present disclosure also includes a non-transitory computer-readable storage medium that stores the program according to the thirteenth aspect.

[0023] The program of the thirteenth aspect and the storage medium storing the program according to the thirteenth aspect may have a configuration including the same specific aspects as the information processing device described above. [Effects of the Invention]

[0024] According to the present disclosure, the predicted values ​​of items with high predicted values ​​can be accurately corrected to reduce bias. [Brief explanation of the drawings]

[0025] [Figure 1] Figure 1 is a conceptual diagram of a typical recommendation system. [Figure 2] FIG. 2 is a diagram for explaining the division of data for model construction. [Figure 3] Figure 3 is an image diagram of the calibration. [Figure 4] FIG. 4 is a diagram of a typical recommendation system. [Figure 5] FIG. 5 is a diagram showing the configuration of a typical calibration-based recommendation system. [Figure 6] FIG. 6 is a diagram showing the relationship between predicted values ​​and true values. [Figure 7] FIG. 7 is a configuration diagram of a recommendation system according to the present disclosure. [Figure 8] FIG. 8 is a diagram for explaining data used in the recommendation system. [Figure 9] FIG. 9 is a diagram showing calibration data and extracted calibration data. [Figure 10] FIG. 10 is a block diagram illustrating an example of a hardware configuration of an information processing device according to the embodiment. [Figure 11] FIG. 11 is a functional block diagram showing the functional configuration of the information processing device during model learning. [Figure 12] FIG. 12 is a functional block diagram showing the functional configuration of the information processing device when recommending an item. [Figure 13] FIG. 13 is a flowchart showing the process of the recommendation method performed by the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Note that the same components are designated by the same reference numerals and redundant explanations will be omitted.

[0027] [Item recommendation based on predictions] FIG. 1 is a conceptual diagram of a typical recommendation system 10. The recommendation system 10 receives user information and context information as input, predicts the rating value of items for each user based on the context, and outputs and recommends to the user the items with the top N predicted values, where N is an integer. Context refers to various "situations," such as the day of the week, the time of day, or the weather. Items can be various objects, such as products, videos, or stores. Rating values ​​can include, for example, the probability of purchase, viewing, or visiting, and rating values ​​after purchase, viewing, or visiting.

[0028] In the example shown in Figure 1, the recommendation system 10 obtains predicted values ​​for each of 100 items, item 1 to item 100, and recommends the top three items with predicted values: item 28, item 79, and item 4. The predicted values ​​for item 28, item 79, and item 4 are 0.56, 0.46, and 0.35, respectively.

[0029] Figure 1 shows an example in which the predicted value is relatively larger for items with relatively higher user ratings. Therefore, the larger the predicted value, the higher the predicted value will be ranked. On the other hand, depending on the definition of the predicted value, it may be the case that the predicted value is relatively smaller for items with relatively higher user ratings. In this case, the smaller the predicted value, the higher the predicted value will be ranked.

[0030] A recommendation is generally considered successful when the user responds positively to an item recommended by the recommendation system 10. A positive response may be, for example, a purchase, a viewing, or a visit. Such recommendation technology is widely used, for example, in electronic commerce (EC) sites and gourmet sites that introduce restaurants.

[0031] [Importance of predicted values] The predicted values ​​of the items are primarily used to select recommended items, but the predicted values ​​of the selected recommended items have further uses, so their accuracy is important.

[0032] The predicted value of an item may be presented to the user, especially in the case of a rating evaluation value after purchase, viewing, or visiting. For example, it may be presented as "This item is predicted to have a satisfaction rating of 4.3 on a 5-point scale." Because there is a cost involved in purchasing the item and the time cost of watching the video, whether or not a recommended item is predicted to have a high level of satisfaction is important information for the user.

[0033] Additionally, item predictions, especially purchase, view, and visit probabilities, may be used for service operations: for example, adjusting inventory preparation based on purchase rates, scaling servers based on view rates, or adjusting employee shifts based on visit rates.

[0034] [Calibration of predicted values] The recommendation system 10 includes a prediction model that outputs a predicted value, which is a value that predicts a user's rating. The prediction model is constructed using, for example, machine learning technology. We want to convert the output score, which is the predicted value of this prediction model, so that the true value and the expected value match. This can be expressed as the following formula.

[0035] Let X be the feature, f(X) be the prediction model, and s be the output score of the prediction model f(X). s=f(X)

[0036] Also, the true value of the evaluation value is y, and the calibration model that converts the output score s is g(s). y=g(s)

[0037] If the expected value is E, it can be expressed by the following formula: E[Y|g(s)=y]=y Note that Y is the objective variable.

[0038] Fig. 2 is a diagram for explaining the division of data for model construction. As shown in Fig. 2, data D_dev used for model construction is randomly divided into data D_tra for model learning and data D_cal for calibration so that the data distributions are the same. That is, the data distributions P_dev(X,Y), P_tra(X,Y), and P_cal(X,Y) of each data D_dev, D_tra, and D_cal satisfy the following formula 4. P_dev(X,Y) = P_tra(X,Y) = P_cal(X,Y) …(Equation 4)

[0039] Calibration is performed using the calibration data D_cal acquired in this manner, and the output score of the prediction model is converted.

[0040] [Calibration image] Figure 3 is an image diagram of calibration. The horizontal axis of Figure 3 is the predicted value, and the vertical axis is the probability that the true value is 1, that is, the frequency of y=1 in the calibration data D_cal. Here, it is assumed that the relationship between the predicted value and the probability of y=1 is as follows:

[0041] Predicted value is -10 → Probability of y=1 is 0.2 Predicted value is 0 → The probability of y=1 is 0.4 Predicted value is +10 → The probability of y=1 is 0.9 In this case, the calibration converts a predicted value of 10 to 0.2, a predicted value of 0 to 0.4, and a predicted value of +10 to 0.9.

[0042] [Typical calibration methods] Representative calibration methods include Platt scaling and isotonic regression. Platt scaling is a method of performing logistic regression with the model predicted value as the feature x and the true value as the objective variable y. Isotonic regression is a calibration method using a step function.

[0043] [Typical Recommender System Configuration] 4 is a configuration diagram of a typical recommendation system 10. The recommendation system 10 includes prediction target data D1, a prediction model 12, and a top ranking extraction means 14.

[0044] The prediction target data D1 is data for outputting a predicted value for each item, and includes user information and context information.

[0045] The prediction model 12 uses the prediction target data D1 to obtain a predicted value for each item for a combination of a user and a context.

[0046] The top ranking extraction means 14 ranks the items for each user based on the predicted value for each item acquired by the prediction model 12, and extracts items with relatively high predicted value rankings for each user.

[0047] The recommendation system 10 recommends to the user the items extracted by the top ranking extraction means 14. In the example shown in Fig. 4, the recommendation system 10 recommends item 28, item 79, and item 4. The predicted values ​​of item 28, item 79, and item 4 are 0.56, 0.46, and 0.35, respectively, which are overestimated.

[0048] Typical calibration configuration 5 is a diagram showing the configuration of a typical calibration-performing recommendation system 10. The recommendation system 10 includes prediction target data D1, calibration data D2, a prediction model 12, and a calibration construction means 16.

[0049] The calibration data D2 is a behavior history data set that indicates past behaviors of users with respect to items for each combination of a plurality of users and a plurality of items.

[0050] The calibration construction means 16 uses the calibration data D2 to construct a calibration model (not shown) that corrects the predicted value for each item predicted by the prediction model 12.

[0051] The recommender system 10 calibrates the predictions of the prediction model 12 using the calibration model to obtain calibrated predictions for all subject data.

[0052] [Issue: Overvaluation bias in recommended items] FIG. 6 is a diagram showing the relationship between predicted values ​​and true values ​​in the recommendation system 10. The vertical axis of FIG. 6 represents predicted values, and the horizontal axis represents true values. Multiple items are plotted in FIG. 6. The dashed lines in FIG. 6 indicate plot positions in the absence of bias. In the example shown in FIG. 6, of the top three items, items I1, I2, and I3, the predicted values ​​of two items, I1 and I2, are greater than the true values. In other words, items I1 and I2 are overrated.

[0053] In this way, even if the overall prediction is calibrated, there is still an issue of overestimation bias for items with high predicted values.

[0054] In reality, due to differences in various characteristics between all items and the top items, not only overestimation bias but also underestimation bias occurs.

[0055] [Configuration of the recommendation system of the present disclosure] 7 is a configuration diagram of a recommendation system 20 according to the present disclosure. The recommendation system 20 is a system that recommends to a user an item (an example of a "recommended item") selected from a plurality of items (an example of a "candidate item"). The recommendation system 20 includes prediction target data D11, calibration data D12, extracted calibration data D13, a prediction model 22, a top ranking extraction means 24, and a calibration construction means 26.

[0056] The prediction target data D11 includes user information and context information.

[0057] The prediction model 22 (an example of a "predictor") may be a trained model that uses the prediction target data D11 to predict a user's rating for each item for a combination of a user and a context, and obtains a predicted value. The prediction model 22 predicts the user's rating using, for example, collaborative filtering.

[0058] Collaborative filtering is a method of predicting user preferences using accumulated data such as the behavioral history of many users. Collaborative filtering is classified into memory-based and model-based methods. Memory-based methods predict user preferences by directly using data such as the user's behavioral history when making recommendations. Model-based methods build a model that represents the regularities between user and item preferences in advance, and make recommendations based on the model and user preference data, without using data such as behavioral history.

[0059] Memory-based methods are classified into user-to-user memory-based methods and item-to-item memory-based methods. User-to-user methods first find other users with similar preference patterns to the user, and then rate items that other users like relatively highly. On the other hand, item-to-item methods rate items that are similar to the items the user likes relatively highly.

[0060] Memory-based collaborative filtering does not have a "trained model" in the narrow sense, but in the broad sense, if the entire data used for prediction (and the prediction method based on it) is considered to be a model, predictions can be made using the same processing as a "trained model."

[0061] The calibration data D12 is a behavior history data set that indicates past behaviors of users with respect to items for each combination of a plurality of users and a plurality of items.

[0062] The top ranking extraction means 24 extracts items with high rankings of predicted values ​​based on the predicted values ​​for each item acquired by the prediction model 22 using the prediction target data D11. The extracted top items are recommended items that the recommendation system 20 will recommend to the user.

[0063] Furthermore, the top ranking extraction means 24 extracts data of items with high rankings in predicted values ​​from the calibration data D12 based on the predicted values ​​acquired by the prediction model 22, and sets the extracted calibration data D13. That is, the extracted calibration data D13 is a data set of items with high rankings in predicted values ​​from among the data sets of the calibration data D12.

[0064] The calibration construction means 26 uses the extracted calibration data D13 to construct a calibration model 28 (an example of a "corrector," see FIG. 9 ) that corrects the predicted value for each item predicted by the prediction model 22. The calibration model 28 may include a parametric machine learning model (parametric model) in which the predicted value is an explanatory variable and the true value is a response variable, or a nonparametric machine learning model (nonparametric model). In this case, construction of the calibration model 28 means learning of the calibration model 28.

[0065] The recommender system 20 uses the calibration model to calibrate and correct the predictions of the predictive model 12 to obtain corrected predictions for the recommended items.

[0066] The recommendation system 20 presents recommended items and the corrected predicted values ​​of the recommended items to the user. In the example shown in Fig. 7, the recommendation system 20 recommends item 28, item 79, and item 4. The corrected predicted values ​​of item 28, item 79, and item 4 are 0.42, 0.37, and 0.29, respectively.

[0067] [Calibration for top predicted values] FIG. 8 is a diagram illustrating data used in the recommendation system 20. The recommendation system 20 uses model-building data D21. The model-building data D21 is a behavior history data set that indicates past user behaviors with respect to items for each combination of multiple users and multiple items (examples of "candidate items"). Note that the model evaluation data may be divided in advance from the model-building data D21 as needed.

[0068] The model construction data D21 is randomly divided into training data D22 for the prediction model 22 and calibration data D12. The prediction model 22 is trained using the training data D22. That is, the recommendation system 20 inputs combinations of users and contexts from the training data D22 into the prediction model 22 and trains the prediction model 22 so as to reduce prediction errors. For example, the recommendation system 20 trains the prediction model 22 using items purchased by the user as positive examples and items not purchased as negative examples. Machine learning is performed until the prediction error converges, and the target prediction performance is achieved.

[0069] The calibration data D12 is verification data for verifying the relationship between the predicted value output from the prediction model 22 and the true value of the user's rating for the item. The calibration data D12 is divided into extracted calibration data D13 and other data D14. The items in the calibration data D12 are ranked by their predicted value, and data of the highest-ranked items is extracted to become the extracted calibration data D13. The extracted calibration data D13 may include the predicted value of the item.

[0070] 9 is a diagram showing calibration data D12 and extracted calibration data D13. As shown in FIG. 9, in the calibration data D12, all items for each of M users U1 to UM are ranked in order of predicted values, where M is an integer. The extracted calibration data D13 is data obtained by extracting the behavioral history of the top N items (top 3 in the example shown in FIG. 9) for each of users U1 to UM. The extracted calibration data D13 is used to train a calibration model 28.

[0071] The higher the ranking of an item, the greater the overestimation bias in the predicted value. Therefore, the calibration construction means 26 may learn a calibration model 28 that calibrates for each rank, such as calibration for the first rank, calibration for the second rank, etc. In this way, the calibration model 28 corrects the predicted value according to the ranking of the predicted value of the item.

[0072] Furthermore, the calibration constructing means 26 may learn a calibration model 28 that performs calibrations collectively within a certain range of rank, such as calibrations ranked 1st to 2nd, calibrations ranked 3rd to 4th, and so on.

[0073] Calibrating for each rank requires more calibration data. In this case, the calibration construction means 26 trains the calibration model 28 using both the predicted values ​​and the ranks of the prediction model 22 as features (explanatory variables).

[0074] The higher the ranking of an item, the more likely it is that a user will view it, so calibration accuracy becomes more important. In response to this, the calibration construction means 26 may assign a relatively larger weight to a higher ranking item when training the calibration model 28. If the rank is r, the calibration construction means 26 may perform training by weighting with 1 / log2(1+r) or 1 / r.

[0075] For example, in platt scaling, logistic regression is performed, but weighting of the training samples may be performed using LogisticRegression().fit(X, y, sample_weight). This makes the loss function during training of the calibration model 28 larger for higher ranks, allowing training of a calibration model 28 with higher accuracy for predicted values ​​with higher ranks.

[0076] [Outline of information processing device] 10 is a block diagram showing an example of the hardware configuration of an information processing device 100 that realizes the recommendation system 20. The information processing device 100 has a predictor function that predicts a user's rating for a candidate item, a function that acquires verification data for verifying the relationship between the predicted value output from the predictor and the true value of the user's rating for the candidate item, a function that inputs the verification data to the predictor and acquires top items that are candidate items that are relatively high in the ranking of the output predicted values, a function that extracts top data corresponding to the top items from the verification data, and a function that learns a corrector that corrects the input predicted value based on the top data so that the input predicted value becomes closer to the true value.

[0077] The information processing device 100 further includes a function of predicting a user's rating for each of a plurality of candidate items using a predictor, a function of selecting a candidate item having a relatively high ranking in predicted value from the plurality of candidate items as a recommended item to be recommended to the user, a function of correcting the predicted value of the recommended item using a corrector, and a function of outputting the recommended item and the corrected predicted value.

[0078] The information processing device 100 is realized using computer hardware and software. The physical form of the information processing device 100 is not particularly limited, and may be a server computer, a workstation, a personal computer, a tablet terminal, or the like. Here, an example in which the processing functions of the information processing device 100 are realized using one computer will be described, but the processing functions of the information processing device 100 may also be realized by a computer system configured using multiple computers.

[0079] The information processing device 100 includes a processor 102 , a non-transitory tangible computer-readable medium 104 , a communication interface 106 , an input / output interface 108 , and a bus 110 .

[0080] Processor 102 executes instructions stored in computer-readable medium 104. The hardware structure of processor 102 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacturing such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for executing specific processing.

[0081] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functional units may be configured with a single processor. Examples of multiple functional units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by a client or server computer, and this processor operates as multiple functional units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple functional units on a single IC (Integrated Circuit) chip, as typified by an SoC (System on Chip). In this way, the various functional units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0082] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0083] The processor 102 is connected to a computer-readable medium 104, a communication interface 106, and an input / output interface 108 via a bus 110. The processor 102 reads various programs, data, etc. stored in the computer-readable medium 104 and executes various processes. The term "program" includes the concept of a program module and includes instructions equivalent to a program.

[0084] The computer-readable medium 104 is a storage device including a memory 112, which is a primary storage device, and a storage 114, which is an auxiliary storage device. The storage 114 is configured using, for example, a hard disk drive (HDD) device, a solid state drive (SSD) device, an optical disk, a magneto-optical disk, or a semiconductor memory, or an appropriate combination of these. The storage 114 stores various programs, data, and the like.

[0085] The memory 112 is used as a working area for the processor 102, and as a storage unit that temporarily stores programs and various data read from the storage 114. When a program stored in the storage 114 is loaded into the memory 112 and the processor 102 executes the instructions of the program, the processor 102 functions as a means for performing various processes specified by the program. The memory 112 may be composed of a single memory, or may be composed of two or more memories of the same or different types.

[0086] The memory 112 includes a data storage unit 120 and a model storage unit 122. The data storage unit 120 is a storage area for storing prediction target data D11 and data used for model construction. The model storage unit 122 is a storage area for storing a prediction model 22 and a calibration model 28.

[0087] The memory 112 stores a data acquisition program 130, a division program 132, a prediction model learning program 134, a prediction program 136, a top ranking extraction program 138, a calibration model learning program 140, various programs, and various data, all of which are executed by the processor 102.

[0088] The data acquisition program 130 is a program for executing a process for acquiring data stored in the data storage unit 120. The data acquisition program 130 may also execute a process for acquiring data from the input device 152.

[0089] The division program 132 is a program that executes a process of dividing the data acquired by the data acquisition program 130. The division program 132 divides, for example, the model construction data D21 into learning data D22 for the prediction model 22 and calibration data D12.

[0090] The prediction model learning program 134 is a program that executes a process of learning the prediction model 22 that predicts the behavior of users with respect to items.

[0091] The prediction program 136 is a program that executes a process of predicting a user's behavior regarding an item using the prediction model 22.

[0092] The top ranking extraction program 138 is a program that executes a process of extracting items with high rankings in predicted values ​​based on the predicted values ​​acquired by the prediction model 22. The top ranking extraction program 138 is also a program that executes a process of extracting a data set of items with high rankings in predicted values ​​from the calibration data D12 based on the predicted values ​​acquired by the prediction model 22, and setting the extracted data as calibration data D13.

[0093] The calibration model learning program 140 is a program that executes the process of learning the calibration model 28.

[0094] The correction program 142 is a program that executes a process of calibrating and correcting the predicted value obtained by the prediction program 136 using the calibration model 28 .

[0095] The communication interface 106 performs communication processing with an external device via a wired or wireless connection, and exchanges information with the external device. The information processing device 100 is connected to a communication line (not shown) via the communication interface 106. The communication line may be a local area network, a wide area network, or a combination of these. The communication interface 106 can serve as a data acquisition unit that accepts input of various data such as an original data set.

[0096] The information processing device 100 may include an input device 152 and a display device 154. The input device 152 and the display device 154 are connected to the bus 110 via the input / output interface 108. The input device 152 may be, for example, a keyboard, a mouse, a multi-touch panel, or another pointing device, or a voice input device, or an appropriate combination thereof. The display device 154 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. Note that the input device 152 and the display device 154 may be integrally configured, such as a touch panel, or the information processing device 100, the input device 152, and the display device 154 may be integrally configured, such as a touch panel tablet terminal.

[0097] 11 is a functional block diagram showing the functional configuration of the information processing device 100 during model learning. Each function shown in FIG. 11 is realized by the processor 102 executing each program. The information processing device 100 includes a data acquisition unit 160, a division unit 162, a prediction model learning unit 164, a prediction unit 166, a top ranking extraction unit 168, and a calibration model learning unit 170.

[0098] The data acquisition unit 160 acquires data from the data storage unit 120. The data acquisition unit 160 acquires from the data storage unit 120, for example, model construction data D21.

[0099] The dividing unit 162 divides the model construction data D21 (an example of "plural data") acquired by the data acquiring unit 160, and sets a portion as training data D22 for the prediction model 22 and the remaining portion as calibration data D12.

[0100] The prediction model learning unit 164 learns the prediction model 22. The prediction model 22 is stored in the model storage unit 122.

[0101] The prediction unit 166 uses the prediction model 22 to make predictions about the calibration data D12.

[0102] The top ranking extraction unit 168 includes the top ranking extraction means 24. Based on the predicted values ​​acquired by the prediction model 22, the top ranking extraction unit 168 extracts a data set (an example of "top data") of items (an example of "top items") with relatively high rankings in predicted values ​​from the calibration data D12, and sets the extracted data as calibration data D13.

[0103] The calibration model learning unit 170 includes the calibration construction means 26. The calibration model learning unit 170 learns the calibration model 28 using the extracted calibration data D13.

[0104] In this way, the information processing device 100 functions as a “learning device” for the calibration model 28.

[0105] 12 is a functional block diagram showing the functional configuration at the time of inference, i.e., at the time of item recommendation, of the information processing device 100. The information processing device 100 includes a data acquisition unit 160, a prediction unit 166, a top ranking extraction unit 168, a correction unit 172, and an output unit 174.

[0106] The data acquisition unit 160 acquires the prediction target data D11 from the data storage unit 120.

[0107] The prediction unit 166 uses the prediction model 22 to predict the user's behavior regarding the item.

[0108] The top ranking extraction unit 168 extracts items with high rankings in predicted values ​​based on the predicted values ​​acquired by the prediction unit 166.

[0109] The correction unit 172 calibrates and corrects the predicted value obtained by the prediction unit 166 using the calibration model 28.

[0110] The output unit 174 outputs the items with the highest ranked predicted values ​​and the corrected predicted values ​​of those items, and may output these to the display device 154.

[0111] In this way, the information processing device 100 functions as a "recommendation device" for items to the user.

[0112] [Recommendation method] 13 is a flowchart showing the processing of a recommendation method (an example of an "information processing method") implemented by information processing device 100. The recommendation method is realized by processor 102 executing a recommendation program (an example of an "information processing program") stored in computer-readable medium 104. The recommendation program may be provided by a computer-readable non-transitory storage medium. In this case, information processing device 100 may read the recommendation program from the non-transitory storage medium and store it in computer-readable medium 104.

[0113] In step S1, the data acquisition unit 160 acquires model construction data D21 as user behavior history data from the data storage unit 120. The division unit 162 divides the model construction data D21 into learning data D22 for the prediction model 22 and calibration data D12.

[0114] In step S2, the prediction model learning unit 164 learns the prediction model 22 using the learning data D22, which is part of the model construction data D21.

[0115] In step S3, the prediction unit 166 uses the prediction model 22 to perform behavior prediction for each user and each item in each context for the calibration data D12, which is the remaining part of the model construction data D21, and outputs a predicted value.

[0116] In step S4, the top ranking extraction means 24 extracts data on the N items with the top predicted values ​​for each user and each context from the calibration data D12 based on the predicted values ​​acquired by the prediction model 22, and sets the extracted data as calibration data D13. The extracted calibration data D13 is the number of users × number of contexts × N pieces of data.

[0117] In step S5, the calibration model learning unit 170 learns the calibration model 28 for the calibration data D13 extracted in step S4. The calibration model 28 is a learned model that converts a predicted value given as input into a behavior probability and outputs it.

[0118] In step S6, the prediction unit 166 predicts user behavior using the prediction model 22 trained in step S2. The top ranking extraction unit 168 ranks the items based on the predicted values ​​and acquires the top N items with the highest predicted value rankings and their predicted values. The correction unit 172 calibrates and corrects the predicted values ​​using the calibration model 28 trained in step S5. The output unit 174 outputs the top N items and their corrected predicted values.

[0119] Here, the number of top items extracted by the top ranking extraction means 24 in step S4 and the number of top items extracted by the top ranking extraction means 24 in step S6 are both the same number N, but they may also be different numbers.

[0120] In the recommendation method, steps S1 to S5 correspond to the "learning method" of the calibration model .

[0121] [Embodiment 1] Product recommendation based on purchase predictions at convenience stores using the recommendation system 20 will now be described.

[0122] A convenience store wants to push five recommended products to application members (hereafter simply referred to as "members") every day. Push delivery means that information is sent from the system to the member's application. There is also a need to accurately understand the purchase rate of the recommended products that are pushed. By understanding the purchase rate, the convenience store operator can use this information to plan product inventory and adjust coupon discount amounts according to the purchase rate.

[0123] The assumptions for the first embodiment are as follows: There are 1 million members, and each store has approximately 1,000 products. Most members always use the same store. Each member purchases an average of two products per day, and each member has a 60-day purchasing history. The product lineup and the price of each product are known. The discount amount for each product changes daily.

[0124] (1) The data acquisition unit 160 acquires the purchase history stored in the data storage unit 120. The division unit 162 divides the purchase history acquired by the data acquisition unit 160 into one for model learning and one for calibration. Here, the division unit 162 divides the purchase history based on the members, such that 10% (100,000 people) of the members' purchase histories are used for calibration and the remaining 90% (900,000 people) of the members' purchase histories are used for model learning. Note that the division unit 162 may also divide the purchase history based on a period, such as using the purchase history for the most recent 7 days for calibration and the purchase history for the remaining 53 days for learning.

[0125] (2) The prediction model training unit 164 trains a purchase prediction model using the purchase history for model training. Here, the prediction model training unit 164 trains so that products purchased on each day are positive examples (1) and products not purchased are negative examples (0). However, 10 products are selected as negative examples by random sampling. The number of training data is 900,000 people × (2 positive examples + 10 negative examples) × 53 days = approximately 570 million items. Note that since the purchase items for the most recent 7 days are used as features, the total is 60 days - 7 days = 53 days.

[0126] The features of the purchase prediction model may be at least one of the following: member attributes (gender, age, occupation), item attributes of each recommended product (product category, list price, discount rate for each day), item attributes of products purchased by the member in the past week (product category, list price, discount rate for each day), and context information for each day (day of the week, weather, maximum temperature, minimum temperature).

[0127] The prediction model learning unit 164 learns, as a purchase prediction model, factorization machines (FMs) that vectorize each feature and express the interaction between the feature values ​​using their inner product. The purchase prediction model may be a prediction model other than FM, or may be a deep learning version of FM.

[0128] (3) The prediction unit 166 makes predictions for each member, each day, and each product for the calibration data using the purchase prediction model learned by the prediction model learning unit 164. The number of predictions made by the prediction unit 166 is 100,000 people × 53 days × 1,000 products = 5.3 billion.

[0129] (4) The top ranking extraction unit 168 extracts the top five predicted products and their predicted values ​​for each member and each day. The number of data items extracted by the top ranking extraction unit 168 is 100,000 people × 53 days × top 5 = 26.5 million.

[0130] (5) The calibration model learning unit 170 performs logistic regression on the data extracted by the top ranking extraction unit 168, using the predicted value as an explanatory variable and whether or not a purchase is made as a target variable. The calibration model learning unit 170 may use Platt scaling or another method.

[0131] (6) The prediction unit 166 predicts today's purchases of each product by each member using the purchase prediction model learned in (2) for today's push distribution. Furthermore, the correction unit 172 converts the predicted values ​​of the top five products using the logistic regression model learned in (5). The output unit 174 outputs the top five products and the converted predicted values, which are the predicted purchase probabilities. The information processing device 100 transmits the top five products and the predicted purchase probabilities for each member via the communication interface 106 to a terminal device on which the member's application is installed. The terminal device may be, for example, a smartphone, a tablet terminal, or a personal computer.

[0132] The recommendation system 20 is not limited to recommending products with relatively high rankings in predicted values, but may also list multiple products, including products with relatively low rankings in predicted values, and present the list to the user along with the predicted purchase probability. When the number of products is relatively large, the recommendation system 20 may cause the terminal device to scroll through the multiple products.

[0133] Furthermore, the recommendation system 20 does not have to recommend products to the user. For example, the recommendation system 20 may automatically determine the quantity of products to purchase based on the predicted purchase probability without presenting the products and the predicted purchase probability to the user.

[0134] [Additional information on Platt scaling] (1) The information processing device 100 typically applies a prediction model (s = f(x)) to calibration data, which in this disclosure is calibration data from which top items have been extracted, and prepares a data set [(s_i, y_i) | i = 1, 2, ... N] consisting of pairs (s_i, y_i) of predicted values ​​s and true values ​​y. Note that f() is an arbitrary function, and x is a feature used for prediction. Since x is generally multidimensional, it is assumed to be a vector.

[0135] (2) The information processing device 100 learns logistic regression using s as an explanatory variable and y as a response variable, and acquires a calibration model g(). g(s)=σ(bs+c) = 1 / (1+exp(-(bs+c))) Note that σ() represents a sigmoid function.

[0136] (3) The information processing device 100 corrects the predicted value s using the obtained calibration model g(), and obtains the corrected predicted value yd. yd = g(s)

[0137] [Embodiment 2] Video recommendation based on the predicted rating value of a video viewing service using the recommendation system 20 will be described.

[0138] In video viewing services, there is a demand for presenting users with a list of recommended videos. Furthermore, there is a demand for presenting the expected satisfaction level (for example, a rating from 1 to 5) along with the video name in the video list.

[0139] The preconditions for the second embodiment are as follows: The number of users of the video viewing service is 10 million, and the number of videos is 1,000. Each user has a rating history (past rating values) of an average of 50 videos.

[0140] (1) The data acquisition unit 160 acquires the evaluation history stored in the data storage unit 120. The division unit 162 divides the evaluation history acquired by the data acquisition unit 160 into one for model learning and one for calibration. For example, the division unit 162 divides the evaluation history so that 20% (10 items on average per user) is used for calibration and the remaining 80% (40 items on average per user) is used for model learning.

[0141] (2) The prediction model training unit 164 trains a rating prediction model using the rating history for model training. Here, the prediction model training unit 164 uses collaborative filtering based on an inter-item memory-based method, but other prediction models may also be used. In order to use collaborative filtering based on an inter-item memory-based method, the prediction model training unit 164 calculates the similarity between videos. For example, the prediction model training unit 164 may extract the rating values ​​of users who have rated two videos and calculate the cosine similarity.

[0142] (3) The prediction unit 166 uses the rating prediction model learned by the prediction model learning unit 164 to make predictions for each user and each video on the calibration data. For example, the prediction unit 166 makes predictions using a weighted average based on the similarity between the recommendation candidate videos and the user's previously rated videos. The number of predicted values ​​predicted by the prediction unit 166 is 10 million people x 960 videos = 9.6 billion. Note that the number of videos here is 1000 - 40 = 960, excluding the 40 rated videos used for learning.

[0143] (4) The top ranking extraction unit 168 extracts the top five videos with the highest predicted values ​​for each user and their predicted values. The number of video data extracted by the top ranking extraction unit 168 is 10 million people x top 5 = 50 million items.

[0144] (5) The calibration model learning unit 170 divides the data extracted by the top ranking extraction unit 168 into 40 bins of predicted values ​​from 1 to 5 in increments of 0.1 ([1-1.1], [1.1-1.2], etc.), and calculates the average of the ratings of the videos actually rated in the calibration data for each bin. Note that the videos rated by each user are only a portion of the total, and not all of the top five videos necessarily have a rating value. Therefore, it is preferable that the top ranking extraction unit 168 further limits the videos extracted in (4) to those that have a rating value in the calibration data out of the 50 million videos extracted from the top five videos for each user.

[0145] The predicted value in each bin is converted to the average of these estimates. This is a calibration technique called histogram binning, although other techniques can also be used.

[0146] (6) To generate a recommended video list, the prediction unit 166 predicts the ratings of each unwatched video for each user using the rating prediction model learned in (2). The correction unit 172 also converts the predicted values ​​of the top five videos using the conversion conditions learned in (5). The output unit 174 outputs the top five videos and the predicted rating values ​​after conversion. The information processing device 100 transmits the top five videos and the predicted rating values ​​for each user to the terminal device owned by that user via the communication interface 106.

[0147] [Additional information on histogram binning] (1) The information processing device 100 typically applies a prediction model (s = f(x)) to calibration data, which in this disclosure is calibration data from which top items have been extracted, and prepares a data set [(s_i, y_i) | i = 1, 2, ... N] consisting of pairs (s_i, y_i) of predicted values ​​s and true values ​​y. Note that f() is an arbitrary function, and x is a feature used for prediction. Since x is generally multidimensional, it is assumed to be a vector.

[0148] (2) Next, the information processing device 100 obtains the correspondence with the true value for each bin of the data set of (1).

[0149] (a) The information processing device 100 divides the data set of (1) into bins based on the value of s. For example, the data is divided into bins where s is equal to or greater than 1.0 but less than 1.1, bins where s is equal to or greater than 1.1 but less than 1.2, and so on.

[0150] (b) The information processing device 100 calculates the average value of the y value for each bin. For example, the information processing device 100 calculates an average value such that the average of y for bins where s is equal to or greater than 1.0 but less than 1.1 is 1.34, the average of y for bins where s is equal to or greater than 1.1 but less than 1.2 is 1.68, and so on.

[0151] (c) The information processing device 100 obtains a conversion rule g(s) for each bin as the calibration model g(). For example, the information processing device 100 obtains a conversion rule g(s) such that if s is equal to or greater than 1.0 and less than 1.1, it is converted to 1.34, if s is equal to or greater than 1.1 and less than 1.2, it is converted to 1.68, and so on.

[0152] (3) The information processing device 100 corrects the predicted value s using the obtained calibration model g(), and obtains the corrected predicted value yd. yd=g(s)

[0153] For example, if s=1.14, it is converted to yd=1.68 because s is greater than or equal to 1.1 and less than 1.2.

[0154] [About the recommended items] In the present disclosure, an "item" refers to a recommended selection candidate, and may be a person rather than a physical object. An example of an item being a person is a recommendation of a potential partner in a dating app.

[0155] Another example of an item being a person is the prediction (or selection) of members who are likely to cancel their membership in a membership service. In this case, the "users" of the recommendation system are marketers who send coupons to those at risk of canceling, and the "items" are the service members.

[0156] Another example where items are people is triage of patients who need urgent treatment, where the "users" of the recommendation system are doctors and nurses and the "items" are patients.

[0157] [Embodiment 3] The prediction and recommendation of service members at risk of cancelling using the recommendation system 20 will be described.

[0158] In membership services, there is a demand for identifying members who are likely to cancel their membership and implementing measures to prevent cancellation for these identified members. One example of a cancellation prevention measure is to send coupons that offer better deals than other members (for example, discount coupons with relatively larger discount amounts than other members). Since such coupons have the potential to result in a loss, there is a demand for sending them only to members who have a certain level of cancellation risk. Therefore, the predicted value of the cancellation rate is important.

[0159] The prerequisites for the third embodiment are as follows: The membership service has 100,000 members, and the current age, occupation, and service usage index of each member are available. The usage index is, for example, RFM (Recency of last purchase date / Frequency of purchase / Monetary purchase amount). In addition, there is information on those who have canceled their subscriptions over the past year. The cancellation information includes, for example, whether the 100,000 members from a year ago have subsequently canceled their subscriptions, and the age, occupation, and service usage index of each member from a year ago.

[0160] (1) The data acquisition unit 160 acquires the churn customer information stored in the data storage unit 120. The division unit 162 divides the churn customer information acquired by the data acquisition unit 160 into information for model learning and information for calibration. For example, the division unit 162 divides the churn customer information so that 20% (20,000 people) is used for calibration and the remaining 80% (80,000 people) is used for model learning.

[0161] (2) The prediction model learning unit 164 learns a churn prediction model using the churnee information for model learning. For example, the prediction model learning unit 164 learns a Gradient Boosting Decision Tree (GBDT) using member attributes and service usage indicators as feature quantities.

[0162] (3) The prediction unit 166 uses the churn prediction model learned by the prediction model learning unit 164 to predict the churn rate of each member for the calibration data.

[0163] (4) The top ranking extraction unit 168 extracts the top 1% of members with the highest predicted cancellation rates and their cancellation rates.

[0164] (5) The calibration model learning unit 170 performs logistic regression on the extracted data using the churn rate as an explanatory variable and whether or not there is a churn as a target variable. The calibration model learning unit 170 may use Platt scaling or another method.

[0165] (6) The prediction unit 166 predicts the churn rate for each member using the churn prediction model learned in (2) in order to take current churn prevention measures. Furthermore, the correction unit 172 converts the churn rates of the top 1% of members using the conversion conditions learned in (5). The output unit 174 outputs the top 1% of members and the converted churn rates. The information processing device 100 may send coupons via the communication interface 106 to members whose converted churn rates are greater than a threshold.

[0166] [Application to the medical field] The recommendation method of the present disclosure can also be applied to the medical field.

[0167] By applying the first embodiment to the medical field, it is possible to recommend prescriptions for patients. For example, it can be applied to a recommendation system that ranks prescriptions based on the probability of prescription.

[0168] By applying the second embodiment to the medical field, it is possible to recommend hospitals and doctors to patients. For example, it can be applied to a recommendation system that ranks hospitals and doctors based on predicted ratings on review sites.

[0169] By applying the third embodiment to the medical field, it is possible to screen patients in an intensive care unit (ICU). For example, it can be applied to a recommendation system that predicts and recommends patients who should be admitted to the ICU among hospitalized patients.

[0170] 〔others〕 The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technical idea of ​​the present disclosure. [Explanation of symbols]

[0171] 10. Recommendation systems 12…Prediction model 14...Method of extracting top rankings 16...Calibration construction means 20...Recommendation system 22…Prediction model 24...Method of extracting top rankings 26...Calibration construction means 28...Calibration Model 100...Information processing device 102...Processor 104...Computer-readable medium 106...Communication interface 108...Input / output interface 110...Bus 112...Memory 114…Storage 120...Data storage unit 122...Model memory section 130...Data acquisition program 132...Division Program 134…Predictive Model Learning Program 136…Prediction Program 138...Top ranking extraction program 140...Calibration Model Learning Program 142...Correction program 152...Input device 154...Display device 160...Data acquisition section 162...Divided part 164…Prediction model learning section 166…Prediction Department 168...Top ranking extraction section 170...Calibration model learning unit 172...correction unit 174...Output section D1: Data to be predicted D2: Calibration data D11: Data to be predicted D12...Calibration data D13: Extracted calibration data D14…Data D21: Data for model construction D22…Learning data I1, I2, I3...Items S1~S6...Recommendation method steps U1, U2, U3, UM... users

Claims

1. one or more processors; one or more memories storing instructions for execution by the one or more processors; Equipped with The processor: acquiring verification data for verifying the relationship between a predicted value output from a predictor that predicts a user's rating for a candidate item and a true value of the user's rating for the candidate item; inputting the verification data into the predictor, and obtaining top items that are candidate items with relatively high rankings of predicted values ​​output; extracting top data corresponding to the top items from the verification data; learning a corrector that corrects the input predicted value based on the higher-order data so that the input predicted value becomes closer to the true value; Information processing device.

2. The processor: predicting a user's rating for each of a plurality of candidate items using the predictor; selecting a candidate item having a relatively high ranking in predicted value from the plurality of candidate items as a recommended item to be recommended to the user; correcting the predicted value of the recommended item using the corrector; outputting the recommended items and the corrected predicted values; The information processing device according to claim 1 .

3. The predictor predicts the user's rating using collaborative filtering. The information processing device according to claim 1 .

4. the predictor includes a trained model; the processor divides a plurality of data sets and uses a portion of the data sets for training the predictor and a remaining portion of the data sets as the validation data. The information processing device according to claim 1 .

5. the processor limits the top items to candidate items for which the user's ratings are present in the validation data; The information processing device according to claim 1 .

6. The processor extracts a number of top items equal to the number of recommended items. The information processing device according to claim 2 .

7. The corrector includes a parametric model or a non-parametric model in which the predicted value is an explanatory variable and the true value is a response variable. The information processing device according to claim 1 .

8. The corrector corrects the predicted value according to the ranking of the top item. The information processing device according to claim 1 .

9. The corrector includes the rank of the predicted value as an explanatory variable. The information processing device according to claim 7 .

10. the processor performs correction by assigning a relatively larger weight to a predicted value having a relatively higher ranking; The information processing device according to claim 1 .

11. The corrector is trained by assigning a relatively larger weight to a predicted value having a relatively higher ranking. The information processing device according to claim 10.

12. 1. An information processing method executed by one or more processors, comprising: the one or more processors: acquiring verification data for verifying a relationship between a predicted value output from a predictor that predicts a user's rating for a candidate item and a true value of the user's rating for the candidate item; inputting the validation data into the predictor, and acquiring top items that are candidate items with relatively high rankings of output predicted values; extracting top data corresponding to the top items from the verification data; learning a corrector that corrects the input predicted value based on the higher-order data so that the input predicted value becomes closer to the true value; An information processing method including:

13. A program that causes a computer to implement the information processing method according to claim 12.

Images