Information processing method, information processing system, and program

Abstract

To provide an information processing method, an information processing system, and a program capable of generating a suggested item list that is robust against domain shift by applying a plurality of models trained with datasets of domains different from an introduction destination domain.SOLUTION: An information processing system is configured to: acquire one or more candidate items from each of a plurality of models trained with datasets in one or more domains different from an introduction destination domain; select, from among the acquired candidate items, a plurality of candidate items having domains different from each other as suggested items; and generate a suggested item list including the suggested items and having robust performance against domain shift.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing system, and a program. [Background technology]

[0002] It is difficult for users to select the best item from a large number of items, both in terms of time and cognitive ability. For example, for a user of an e-commerce site, the items are the products sold on the site, and for a user of a document information management system, the items are the stored document information.

[0003] Non-Patent Documents 1 and 2 describe research into information recommendation technology, which is a technology that presents selection candidates from among items to assist users in making selections. Note that the "EC" in EC sites is an abbreviation for Electronic Commerce.

[0004] Generally, recommendation systems learn based on data collected at the facility where they are installed. However, when a recommendation system is installed in a facility other than the facility corresponding to the training data, the model's prediction accuracy decreases. The problem of a machine learning model not functioning well in unknown other facilities is called domain shift. As described in Non-Patent Documents 3 and 4, domain generalization, which is research into improving robustness against domain shifts, has been actively studied in recent years, mainly in image recognition. However, there are no research examples of domain generalization in information recommendation technology.

[0005] When learning a learning model applied to an information recommendation system, Even if data from the previous facility cannot be obtained, the information recommendation system will be introduced. If data from the previous facility is available, the learning model can be evaluated using that data and multiple The best learning model can be selected from the candidate learning models.

[0006] However, even when the learning model is introduced, there may be no data available for the facility where it is being introduced. Or, even if data exists for the facility where the system is introduced, the data cannot be accessed. If not, it is difficult to select the best learning model from the multiple candidate learning models mentioned above. is.

[0007] Non-Patent Document 5 describes learning models corresponding to multiple domains, each of which A recommendation technique that aims to make predictions robust to domain shifts by using the average of all prediction results. The technique is described.

[0008] Non-Patent Document 6 describes collaborative filtering, a type of prediction model in information recommendation technology. For a model, combining multiple forecasts, for example by applying the average of multiple forecasts, In addition, methods for attempting to improve prediction accuracy are described. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Dietmar Jannach, Markus Zanker, Alexander Felfernig, Gerhard Friedrich, Katsumi Tanaka, Kazutoshi Kadotani, "Introduction to Information Recommendation Systems - Theory and Practice", Kyoritsu Shuppan, 2012 [Non-patent document 2] Deepak K. Agarwal and Bee-Chung Chen, "Recommender Systems: Theory and Practice of Statistical Machine Learning," Kyoritsu Publishing, 2018 [Non-patent document 3] Jindong Wang1, Cuiling Lan1, Chang Liu1, Yidong Ouyang2, Tao Qin, “Generalizing to Unseen Domains: A Survey on Domain Generalization” Microsoft Research, Beijing, China, 2021. [Non-patent document 4] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, Chen Change Loy, “Domain Generalization in Vision: A Survey”, Central University of Finance and Economics, Beijing, China, 2021. [Non-patent document 5] Zheng Xu, Wen Li, Li Niu, and Dong Xu, "Exploiting Low-rank Structure from Latent Domains for Domain Generalization" School of Computer Engineering, Nanyang Technological University, Singapore, 2014 [Non-patent document 6] "Combining predictions for accurate recommender systems" by Michael Jahrer, Andreas Toscher, and Robert Legenstein, 2010 Summary of the Invention [Problem to be solved by the invention]

[0010] However, even in the evaluation of an information recommendation system to which a learning model is applied before its introduction, if data from the facility where the information recommendation system is to be introduced is unavailable, it is difficult to provide the best information recommendation system for the facility where the system is to be introduced based on the data from the facility where the system is to be introduced.However, even if the facility corresponding to the learning data is different from the facility where the system is to be introduced, there is a demand for a system that is robust against domain shifts and achieves high-performance information recommendation at the facility where the system is to be introduced.

[0011] The recommendation technology described in Non-Patent Document 5 is a method that assumes image processing and is not suitable for information recommendation technology. Specifically, the information recommendation technology described in Non-Patent Document 5 is premised on the output of a single prediction and is not suitable for information recommendation technology that outputs multiple predictions.

[0012] In the method described in Non-Patent Document 6, multiple collaborative filtering models are trained using data from the same domain, and are not robust to domain shifts. Moreover, the method described in Non-Patent Document 6 does not aim for domain generalization.

[0013] The present invention has been made in consideration of the above circumstances, and aims to provide an information processing method, an information processing system, and a program that can generate a recommended item list that is robust against domain shifts by applying multiple models trained using datasets from domains different from the domain to which the method is introduced. [Means for solving the problem]

[0014] An information processing method according to a first aspect of the present disclosure is an information processing method in which an information processing system having one or more processors generates a recommended item list that recommends a plurality of items to a user, in which the information processing system acquires one or more candidate items from each of a plurality of models trained using datasets in one or more domains different from the domain to which the information processing system is introduced, and selects, from the acquired plurality of candidate items, a plurality of candidate items that are in different domains as recommended items, thereby generating a recommended item list that includes the plurality of recommended items and has robust performance against domain shifts.

[0015] According to the information processing method of the first aspect, a list of recommended items that is robust against domain shifts can be generated by applying a plurality of models trained using datasets of domains different from the domain to which the system is introduced.

[0016] An example of a place where the system is introduced is a facility. A facility is a group where multiple users act. Examples of facilities include companies and e-commerce sites.

[0017] An information processing method according to a second aspect may be the information processing method according to the first aspect, wherein the information processing system calculates predicted values ​​that predict user behavior for each candidate item, and selects a recommended item from the multiple candidate items based on the ranking of statistical values ​​calculated from the predicted values ​​of the same candidate item in each of multiple domains different from the domain to which the item is introduced.

[0018] According to this aspect, a recommended item can be selected from a plurality of candidate items based on the predicted value for each candidate item.

[0019] An example of a predictive value for a user's behavior is the probability that the user will perform a positive behavior.

[0020] An information processing method according to a third aspect is the information processing method according to the first or second aspect, wherein the information processing system derives an evaluation value for each of a plurality of candidate lists that are candidates for the recommended item list according to the similarity of attributes between the destination domain and each of the plurality of domains, and the candidate list with the largest minimum evaluation value may be used as the recommended item list.

[0021] According to this aspect, a recommended item list can be selected from a plurality of candidate lists based on the evaluation values ​​of the candidate lists.

[0022] An information processing method according to a fourth aspect is the information processing method according to the third aspect, wherein the information processing system assumes that the user behaves positively toward candidate items of a model trained using data from a domain whose attributes are similar to those of the destination domain, and that the user behaves negatively toward candidate items of a model trained using data from a domain whose attributes are distant from those of the destination domain, and deterministically tests the user's behavior to calculate an evaluation value for each candidate list.

[0023] According to this aspect, a recommended item list can be selected from a plurality of candidate lists based on evaluation values ​​calculated by definitively trying out user actions.

[0024] An information processing method according to a fifth aspect is the information processing method according to the third aspect, wherein the information processing system assumes that a user will take positive action with a first probability toward candidate items of a model trained using as training data a dataset from a domain whose attributes are similar to those of the destination domain, and assumes that a user will take positive action with a second probability toward candidate items of a model trained using as training data a dataset from a domain whose attributes are distant from those of the destination domain, and probabilistically tests the user's behavior to calculate an evaluation value for each candidate list.

[0025] According to this aspect, a recommended item list can be selected from a plurality of candidate lists based on evaluation values ​​calculated by probabilistically testing user behavior.

[0026] An information processing method according to a sixth aspect is the information processing method according to the fifth aspect, wherein the information processing system estimates a first probability from an evaluation result obtained by evaluating each of the plurality of models in a first domain to which the dataset is applied as training data, and Nomo For each of the models, a second probability may be estimated from the evaluation results obtained in a second domain different from the first domain.

[0027] An information processing method according to a seventh aspect may be the information processing method according to the third aspect, wherein the information processing system calculates an evaluation value based on the user's behavior when a candidate list is presented to the user in the destination domain.

[0028] An information processing method according to an eighth aspect is an information processing method according to any one of the third to seventh aspects, wherein the information processing system calculates an evaluation value for each candidate list by applying a weight that is determined for each candidate item according to the ranking of the candidate item included in the candidate list and that is determined according to the evaluation conditions.

[0029] According to this aspect, an evaluation value can be calculated according to the weight of each candidate item.

[0030] An information processing method according to a ninth aspect is the information processing method according to any one of the third to eighth aspects, wherein the information processing system may select one or more candidate items from each of a plurality of candidate lists.

[0031] According to this aspect, candidate items from different sources are selected as recommended items, which can ensure a certain degree of robustness against domain shifts in the recommended item list.

[0032] An information processing method according to a tenth aspect is an information processing method according to any one of the first to eighth aspects, wherein the information processing system may select candidate items to be recommended items from among multiple candidate lists, giving priority to dissimilar candidate lists.

[0033] According to this aspect, selection of recommended items from each of a plurality of candidate lists that are similar to one another is avoided, thereby ensuring a certain level of robustness against domain shifts in the plurality of recommended items.

[0034] In such an aspect, the similarity of the models may be calculated, and the similarity or dissimilarity of the candidate items may be determined based on the similarity of the models.

[0035] An information processing method according to an eleventh aspect is an information processing method according to any one of the first to eighth aspects, wherein when the information processing system presents a recommended item list to the same user multiple times, it may rearrange the order of multiple recommended items included in the recommended item list each time it is presented.

[0036] According to this aspect, a certain level of averaging can be achieved for the recommended item list presented each time.

[0037] In an information processing method according to a twelfth aspect, in the information processing method according to any one of the first to eighth aspects, when the information processing system presents the recommended item list multiple times, it may rearrange the order of the multiple recommended items included in the recommended item list each time it is presented.

[0038] In this aspect, the order of the multiple recommended items may be changed for each user.

[0039] An information processing method according to a thirteenth aspect is an information processing method according to any one of the first to twelfth aspects, wherein the information processing system may apply, as multiple models, trained models trained using datasets in different domains as training data.

[0040] According to this aspect, each of the multiple models depends on a different domain, which can ensure a certain level of robustness against domain shifts in the recommended item list.

[0041] An information processing method according to a fourteenth aspect is the information processing method according to the first aspect, wherein the information processing system generates a plurality of trained models trained using different sets of features in one domain different from the domain to which the information processing system is introduced as training data. Nomo It may also be applied as a driver.

[0042] According to this aspect, even if it is difficult to obtain datasets for different domains, candidate items can be obtained from multiple models with different training data, thereby ensuring a certain level of robustness against domain shifts in the recommended item list.

[0043] An information processing system according to a fifteenth aspect of the present disclosure is an information processing system that generates a recommended item list that recommends one or more items to a user, the information processing system comprising one or more processors and one or more memories that store a program to be executed by the one or more processors, wherein the one or more processors execute instructions of the program to obtain one or more candidate items from each of a plurality of models trained using datasets in one or more domains different from the domain to which the system is introduced, and selects, from the obtained plurality of candidate items, a plurality of candidate items that are in different domains from each other as recommended items, thereby generating a recommended item list that includes the plurality of recommended items and has robust performance against domain shifts.

[0044] The information processing system according to the fifteenth aspect can achieve the same effects as the information processing method according to the first aspect. The components of the information processing methods according to the second to fourteenth aspects can be applied to the components of the information processing device according to the other aspects.

[0045] A program according to a sixteenth aspect of the present disclosure is a program for generating a recommended item list that recommends one or more items to a user, the program enabling a computer to realize a function of acquiring one or more candidate items from each of a plurality of models trained using datasets in one or more domains different from the domain to which the program is introduced, and a function of selecting, from the acquired plurality of candidate items, a plurality of candidate items that are in different domains as recommended items, thereby generating a recommended item list that includes the plurality of recommended items and has robust performance against domain shifts.

[0046] The program according to the 16th aspect can achieve the same effects as the information processing method according to the 1st aspect. The constituent elements of the information processing methods according to the 2nd to 14th aspects can be applied to the constituent elements of the program according to the other aspects. [Effects of the Invention]

[0047] According to the present invention, a list of recommended items that is robust to domain shifts can be generated by applying a plurality of models trained using datasets from domains different from the domain to which the model is introduced. [Brief explanation of the drawings]

[0048] [Figure 1] Figure 1 is a conceptual diagram of a typical recommendation system. [Figure 2] Figure 2 is a conceptual diagram showing an example of supervised machine learning, which is widely used to build recommendation systems. [Figure 3] FIG. 3 is an explanatory diagram showing a typical implementation flow of a recommendation system. [Figure 4] Figure 4 is an explanatory diagram of the introduction process of the recommendation system when data on the facility where the system is to be introduced cannot be obtained. [Figure 5] FIG. 5 is an explanatory diagram of model learning by domain application. [Figure 6]Figure 6 is an explanatory diagram of the recommendation system implementation flow, which includes a step of evaluating the performance of a trained learning model. [Figure 7] FIG. 7 is an explanatory diagram showing examples of learning data and evaluation data used in machine learning. [Figure 8] Figure 8 is a graph that schematically shows the difference in model performance due to differences in datasets. [Figure 9] FIG. 9 is an explanatory diagram showing an example of a recommendation system introduction flow when the learning domain and the introduction domain are different. [Figure 10] Figure 10 is an explanatory diagram showing the problem that occurs when there is no data on the facility where the system is being introduced. [Figure 11] FIG. 11 is a schematic diagram of a typical recommended item list. [Figure 12] FIG. 12 is a schematic diagram showing the evaluation results in the first example of the recommended item list evaluation. [Figure 13] FIG. 13 is a schematic diagram showing the evaluation results in the second example of the recommended item list evaluation. [Figure 14] FIG. 14 is an explanatory diagram illustrating an overview of the information processing method according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram showing a specific example of the recommended item list evaluation. [Figure 16] FIG. 16 is a schematic diagram showing another specific example of the recommended item list evaluation. [Figure 17] FIG. 17 is a block diagram schematically illustrating an example of the hardware configuration of an information processing system according to an embodiment. [Figure 18] FIG. 18 is a functional block diagram showing the functional configuration of the information processing system according to the embodiment. [Figure 19] FIG. 19 is a flowchart showing the procedure of the information processing method according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram showing a recommended item list generating method according to the first embodiment. [Figure 21] FIG. 21 is a schematic diagram showing a recommended item list generating method according to the second embodiment. [Figure 22]FIG. 22 is a schematic diagram showing a recommended item list generating method according to the third embodiment. [Figure 23] FIG. 23 is a schematic diagram showing a recommended item list generating method according to the fourth embodiment. [Figure 24] FIG. 24 is a schematic diagram showing a recommended item list generating method according to the fifth embodiment. [Figure 25] FIG. 25 is a schematic diagram showing a recommended item list generating method according to the sixth embodiment. [Figure 26] FIG. 26 is a schematic diagram showing a recommended item list generating method according to the seventh embodiment. [Figure 27] FIG. 27 is a schematic diagram showing a recommended item list generating method according to the eighth embodiment. [Figure 28] FIG. 28 is a schematic diagram showing a recommended item list generating method according to the ninth embodiment. [Figure 29] FIG. 29 is a schematic diagram showing a recommended item list generating method according to the tenth embodiment. [Figure 30] FIG. 30 is a schematic diagram showing an example of a recommended item list generated by applying the recommended item list generation method according to the eleventh embodiment. [Figure 31] FIG. 31 is a schematic diagram showing another example of a recommended item list generated by applying the recommended item list generating method according to the eleventh embodiment. [Figure 32] FIG. 32 is an explanatory diagram of a first specific example of a plurality of models. [Figure 33] FIG. 33 is an explanatory diagram of a second specific example of a plurality of models. [Figure 34] Figure 34 is a list of variables. DETAILED DESCRIPTION OF THE INVENTION

[0049] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In this specification, the same components are designated by the same reference numerals, and redundant explanations will be omitted where appropriate.

[0050] [Overview of information recommendation technology] In this embodiment, a method for generating data from different domains regarding user behavior history data used for learning and evaluating models used in a recommendation system will be described. First, an overview of information recommendation technology and the need for data from multiple domains will be provided with specific examples. Information recommendation technology is a technology for recommending items to users. Note that recommendations can also be referred to as suggestions.

[0051] FIG. 1 is a conceptual diagram of a typical recommendation system. The recommendation system 10 receives user information and context information as input, and outputs information about items to recommend to the user based on the context. The context refers to various situations, such as the day of the week, the time of day, or the weather. The items can be various objects, such as books, videos, and restaurants.

[0052] A recommendation system 10 generally recommends multiple items simultaneously. FIG. 1 shows an example in which the recommendation system 10 recommends three items IT1, IT2, and IT3. The recommendation is generally considered successful if the user responds positively to the recommended items IT1, IT2, and IT3. Positive responses include, for example, purchasing, watching, and visiting. Such recommendation technology is widely used in, for example, e-commerce sites and gourmet sites that introduce restaurants.

[0053] Figure 2 is a conceptual diagram showing an example of supervised machine learning, which is widely used in building recommendation systems. A recommendation system 10 is built using machine learning technology. Generally, positive and negative examples are prepared based on the user's past behavioral history, and combinations of the user and context are input into a prediction model 12, which is then trained to reduce the prediction error. For example, viewed items viewed by the user are considered positive examples, and unviewed items not viewed by the user are considered negative examples. Machine learning is continued until the prediction error converges, achieving the target prediction performance.

[0054] The trained prediction model 12 is used to recommend items that have a high predicted probability of being viewed for a combination of user and context. The trained prediction model 12 is synonymous with the trained prediction model 12.

[0055] For example, when a combination of user A and context β is input to the trained prediction model 12, the prediction model 12 infers that user A is highly likely to view documents such as item IT3 shown in Figure 1 under the conditions of context β, and recommends items similar to item IT3 to user A. Note that, depending on the configuration of the recommendation system 10, items are often recommended to users without taking the context into consideration.

[0056] [Example of data used in developing a recommendation system] A user's behavioral history is equivalent to the ground truth data in machine learning. Strictly speaking, it is understood as a task setting to infer the next action from the past behavioral history, but it is common to learn latent features based on the past behavioral history.

[0057] The user's behavior history may include, for example, a book purchase history, a video viewing history, or a restaurant visit history.

[0058] Furthermore, the main features include user attributes and item attributes. User attributes can include various elements such as gender, age, occupation, family structure, and residential area. Item attributes can include various elements such as book genre, price, video genre, length, restaurant genre, and location.

[0059] [Model construction and operation] Figure 3 is an explanatory diagram showing a typical implementation flow of a recommendation system. This diagram shows a typical flow when implementing a recommendation system in a facility. In implementing a recommendation system, step 1 is to build a model 14 that performs the desired recommendation task, and step 2 is to implement and operate the built model 14.

[0060] In the case of a machine learning model, constructing the model 14 includes training the model 14 using training data to create a recommendation model that is a predictive model that meets a practical level of recommendation performance. Operating the model 14 means, for example, obtaining an output of a list of recommended items from the trained model 14 in response to an input of a combination of a user and a context.

[0061] Learning data is required to build the model 14. As shown in Figure 3, the model 14 of a recommendation system is generally trained based on data collected at the facility where it is installed. By training using data collected from the facility where it is installed, the model 14 learns the behavior of users at the facility where it is installed, and is able to accurately predict recommended items for users at the facility where it is installed.

[0062] However, due to various reasons, there are cases where data from the facility where the system is to be implemented cannot be obtained. For example, in the case of a document information recommendation system in a company's in-house system or a hospital's in-house system, the company developing the recommendation model may not be able to access data from the facility where the system is to be implemented. When data from the facility where the system is to be implemented cannot be obtained, it is necessary to train the system using data collected at a different facility instead.

[0063] Figure 4 is an explanatory diagram of the introduction process of the recommendation system when data from the facility where the system is being introduced is not available. Guidance When used at the receiving facility, there is a problem that the predictive accuracy of Model 14 decreases due to differences in user behavior between facilities.

[0064] The problem of machine learning models not performing well at unknown facilities different from the facilities they were trained on can be broadly understood as a technical challenge of improving robustness against the problem of domain shift, where the source domain in which the model 14 was trained differs from the target domain to which the model 14 is applied. A problem setting related to domain generalization is domain adaptation, which is a learning method that uses data from both the source domain and the target domain. The purpose of using data from a different domain, even when data from the target domain exists, is to compensate for the small amount of data in the target domain that is insufficient for learning.

[0065] Note that domain generalization may be referred to as "Domain generalization" using English notation, and domain adaptation may be referred to as "Domain adaptation" using English notation.

[0066] Figure 5 is an explanatory diagram of model learning by domain application. Although the amount of data collected at the facility where the system is installed, which is the target domain, is relatively small compared to the amount of data collected at a different facility, by learning using both sets of data, model 14 can predict with a certain degree of accuracy the behavior of users at the facility where the system is installed.

[0067] [Domain Description] The differences in facilities mentioned above are a type of domain difference. In Ivan Cantador et al., Chapter 27: "Cross-domain Recommender System," a paper on domain adaptation in information recommendation, domain differences are classified into the following four types:

[0068] [Item attribute level] For example, comedy movies and horror movies are separate domains. The item attribute level can be referred to as the Item attribute level in English.

[0069] [Item Type Level] For example, movies and TV dramas are separate domains. The item type level can be referred to as the Item type level in English.

[0070] [Item Level] For example, movies and books are separate domains. Note that the item level can be referred to as the Item level in English.

[0071] [System level] For example, movies in cinemas and movies broadcast on television are separate domains. The system level can also be referred to as "System level" in English.

[0072] The differences in facilities shown in Figure 5 etc. fall under the system-level domain of the four classifications mentioned above.

[0073] To formally define a domain, it is defined by the joint probability distribution P(X,Y) of the response variable Y and the explanatory variable X, and if Pd1(X,Y)≠Pd2(X,Y), then d1 and d2 are different domains.

[0074] The joint probability distribution P(X,Y) can be expressed as the product of the distribution of the explanatory variables P(X) and the conditional probability distribution P(Y|X), or the product of the distribution of the objective variable P(Y) and the conditional probability distribution P(Y|X).

[0075] P(X,Y)=P(Y|X)P(X)=P(X|Y)P(Y)

[0076] Therefore, a change in one or more of P(X), P(Y), P(Y|X), and P(X|Y) results in a different domain.

[0077] [Typical pattern of domain shift] [Covariate shift] When the distribution of explanatory variables P(X) differs, it is called a covariate shift. For example, when the distribution of user attributes differs between datasets, or more specifically, when the gender ratio differs, this corresponds to a covariate shift. Note that covariate shift can also be referred to as covariate shift in English.

[0078] [Prior probability shift] When the distribution of the objective variable P(Y) differs, it is called a prior probability shift. For example, when the average view rate or average purchase rate differs between datasets, this corresponds to a prior probability shift. Note that a prior probability shift can also be referred to as a prior probability shift in English.

[0079] [Concept shift] When the conditional probability distributions P(Y|X) and P(X|Y) differ, it is called a concept shift. For example, the probability that a company's research and development department will read data analysis documents is P(Y|X), but this differs between data sets, which is an example of a concept shift. Note that concept shifts can also be referred to as concept shifts in English.

[0080] Research on domain adaptation or domain generalization can be divided into two types: one that assumes one of the above patterns as the main factor, and one that considers how to deal with changes in P(X,Y) without considering which pattern is the main factor. In the former case, many studies assume covariate shifts in particular.

[0081] [Why domain shifts affect you] Because predictive classification models that perform prediction or classification tasks make inferences based on the relationship between explanatory variable X and target variable Y, changes in P(Y|X) naturally result in a decline in either prediction or classification performance. Furthermore, when machine learning predictive classification models, the goal is to minimize either prediction error or classification error within the training data. For example, if the frequency of explanatory variable X = X_1 is higher than the frequency of X = X_2, i.e., if P(X = X_1) > P(X = X_2), there is more data for X = X_1 than for X = X_2, and therefore error reduction for X = X_1 is prioritized over error reduction for X = X_2. Therefore, even if P(X) changes between facilities, either prediction error or classification error will decline.

[0082] Domain shift can be a problem not only for information recommendation but also for models of various tasks. For example, when a model for predicting the risk of employee resignation is trained using data from one company, domain shift can become a problem when it is used by another company.

[0083] Furthermore, for models that predict antibody production volumes of cells, domain shift can become an issue when a model trained using data on one antibody is applied to a different antibody. Similarly, for models that classify customer feedback, such as models that classify VOC into product features, support responses, and others, domain shift can become an issue when a classification model trained using data on one product is applied to a different product. VOC is an abbreviation for Voice of Customer, which is the English term for customer feedback.

[0084] [Evaluation of the model before its implementation] Before introducing a trained model 14 into an actual facility, the performance of the model 14 is often evaluated. Performance evaluation is necessary to determine whether or not to introduce the model, and for research and development of models or learning methods.

[0085] Figure 6 is an explanatory diagram of the recommendation system implementation flow, which includes a step of evaluating the performance of a trained learning model. In Figure 6, a step of evaluating the performance of model 14 is added as step 1.5 between step 1 of training model 14 and step 2 of operating model 14, which are explained in Figure 5. The rest of the configuration is the same as in Figure 5.

[0086] In a typical recommendation system implementation flow, data collected at the facility where the system is implemented is often divided into training data and evaluation data, as shown in Figure 6. The predictive performance of the model 14 is confirmed using the evaluation data before the model 14 begins operation.

[0087] However, when constructing a domain generalization model 14, the training data and the evaluation data must be from different domains. Furthermore, in domain generalization, it is preferable to use data from multiple domains for the training data, and it is even more preferable to have as many domains as possible for training.

[0088] [About generalizability] FIG. 7 is an explanatory diagram showing examples of training data and evaluation data used in machine learning. A data set obtained from a joint probability distribution Pd1(X,Y) of a certain domain d1 is divided into training data and evaluation data. The evaluation data of the same domain as the training data is called the first evaluation data, and is denoted as evaluation data 1 in FIG. 7. In addition, a data set obtained from the joint probability distribution Pd2(X,Y) of a domain d2 different from domain d1 is prepared and used as evaluation data. The evaluation data of a domain different from the training data is called the second evaluation data, and is denoted as evaluation data 2 in FIG. 7.

[0089] The model 14 is trained using the training data of the domain d1, and the performance of the trained model 14 is evaluated using the first evaluation data of the domain d1 and the second evaluation data of the domain d2.

[0090] Fig. 8 is a graph that schematically illustrates differences in model performance resulting from differences in data sets. If the performance of model 14 in the training data is performance A, the performance of model 14 in the first evaluation data is performance B, and the performance of model 14 in the second evaluation data is performance C, then as shown in Fig. 8, the relationship is usually performance A > performance B > performance C.

[0091] The high generalization performance of model 14 generally refers to high performance B or a small difference between performance A and B. In other words, the high generalization performance of model 14 aims to achieve high predictive performance even for untrained data without overfitting to the training data.

[0092] In the context of domain generalization in this specification, this refers to high performance C or a small difference between performance B and performance C. In other words, the goal is to achieve consistently high performance even in domains different from the domain used for learning.

[0093] When training the model 14, it is not possible to use data from the facility where the model is to be introduced, but if data collected at the facility at the time of introduction is available, the performance of the model can be evaluated using the data obtained at the facility where the model is to be introduced. Based on the results of the evaluation, it is possible to select the most appropriate model from multiple candidate models and apply it to the facility where the model is to be introduced. An example is shown in Figure 9.

[0094] FIG. 9 is an explanatory diagram showing an example of a recommendation system implementation flow when the learning domain and the implementation domain are different. As shown in FIG. 9, multiple models may be trained using data collected at a facility other than the implementation facility. Here, an example is shown in which models M1, M2, and M3 are trained using datasets DS1, DS2, and DS3, each collected at a different facility. For example, model M1 is trained using dataset DS1, model M2 is trained using dataset DS2, and model M3 is trained using dataset DS3. Note that the datasets used to train models M1, M2, and M3 may be a combination of multiple datasets collected at different facilities. For example, model M1 may be trained using a dataset that combines datasets DS1 and DS2.

[0095] After training multiple models M1, M2, and M3 in this way, the performance of each of the models M1, M2, and M3 is evaluated using data Dtg collected at the facility where the models are to be introduced. In Figure 9, the symbols A, B, and C shown below each of the models M1, M2, and M3 represent the evaluation results of each of the models M1, M2, and M3. A rating indicates good predictive performance that meets the introduction criteria. A rating indicates poorer performance than A. A rating indicates poorer performance than B, and is not suitable for introduction.

[0096] For example, as shown in Figure 9, assume that the evaluation result of model M1 is rating A, the evaluation result of model M2 is rating B, and the evaluation result of model M3 is rating C. Model M1 is selected as the optimal model for the facility where the model M1 is to be introduced, and the recommendation system 10 applying model M1 is introduced.

[0097] [assignment] As explained using Figure 9, even if data from the facility where the model is to be introduced is not available when training the model, if data collected at the facility where the model is to be introduced is available, the data can be used to evaluate the models and select the best model.

[0098] However, if there is no data for the facility where the model is to be introduced when the model is introduced to the facility, or if there is data for the facility where the model is to be introduced but the data for the facility where the model is to be introduced cannot be accessed when the model is introduced to the facility, the model cannot be selected.

[0099] Figure 10 is an explanatory diagram showing the problem that occurs when there is no data on the facility where the system will be introduced. As shown in Figure 10, when there is no data on the facility where the system will be introduced, it is not possible to evaluate each of Model M1, Model M2, and Model M3, and it is not possible to select the best model.

[0100] In this way, if data on the facility where the model is to be introduced is not available even for evaluation before the model is introduced, it is not possible to select the best model for the facility where the model is to be introduced. Even in such cases, it is desirable to provide high-performance recommendations to the facility where the model is to be introduced. Because the learning domain and the domain where the model is to be introduced are different, it is a challenge to achieve recommendations that are robust to domain shifts. In this embodiment, an information processing method and an information processing system are provided that are capable of presenting recommended items by utilizing multiple models.

[0101] [Recommended item list rating] 11 is a schematic diagram of a typical recommended item list, showing a recommended item list IL100 including five recommended items, item 1 to item 5.

[0102] 11, five recommended items IT are arranged from top to bottom in a predetermined ranking order. The numbers 1 to 5 assigned to items 1 to 5 respectively indicate the relative ranking of the recommended items in the recommended item list IL100.

[0103] The recommended item list IL100 is useful to the user if it contains recommended items that the user desires. An example of a recommended item that the user desires is a recommended item that the user will take a positive action on, such as browsing.

[0104] When recommended items are arranged from top to bottom, users tend to view them in order from the top. Users may not view all the recommended items and may only view part of the list. Therefore, it is desirable that the recommended items desired by users are relatively high up.

[0105] Fig. 12 is a schematic diagram showing the evaluation results in the first example of the recommended item list evaluation. Fig. 12 illustrates the hit rate as an evaluation index for the recommended item list. Fig. 12 illustrates the evaluation values ​​for each of the recommended item list IL110, the recommended item list IL112, and the recommended item list IL114.

[0106] The hit rate is evaluated as 1 if at least one of the recommended items IT included in each of the recommended item lists IL110, IL112, and IL114 is hit. In addition, the hit rate is evaluated as 0 if none of the recommended items IT included in the recommended item list IL110, etc. is hit. Here, a hit of a recommended item IT means that the user takes positive action regarding the recommended item IT.

[0107] For example, item 3 in the recommended item list IL110 is shown with a mark indicating that it is a hit recommended item IT. Similarly, item 2 in the recommended item list IL114 is shown with a mark indicating that it is a hit recommended item IT.

[0108] When the hit rate is applied as an evaluation index for the recommended item list IL110 and the like, the evaluation values ​​of the recommended item list IL110 and the recommended item list IL114 are 1, and the evaluation value of the recommended item list IL110 is 0.

[0109] Fig. 13 is a schematic diagram showing the evaluation results in the second example of the recommended item list evaluation. Fig. 13 shows an example of reciprocal rank as an evaluation index for the recommended item list. Reciprocal rank uses the inverse of the rank as a weight to weight and evaluate each recommended item IT.

[0110] FIG. 13 illustrates a case where item 3 is a hit in recommended item list IL110, none of items 1 to 5 are a hit in recommended item list IL112, and item 3 is a hit in recommended item list IL114.

[0111] In the recommended item list IL110, item 3 with a weight of 1 / 3 has been hit, and items 1, 2, 4, and 5 have not been hit. Therefore, the evaluation value of the recommended item list IL110 is 1 / 3.

[0112] In addition, in the recommended item list IL114, the item 2 with a weight of 1 / 2 is hit, and the items 1 and 3 to 5 are not hit. 114 The evaluation value is 1 / 2.

[0113] Furthermore, the evaluation value of the recommended item list IL112 in which the user has not hit any of items 1 to 5 is 0. Note that the weights and evaluation values ​​to which fractions are applied may be replaced with decimals.

[0114] Another example of an evaluation metric is discounted cumulative gain. Discounted cumulative gain is weighted by 1 / log(1 + rank). In other words, for evaluation metrics that use weights, items with higher rankings are weighted relatively more heavily.

[0115] [Overview of information processing method according to embodiment] Fig. 14 is an explanatory diagram of an overview of the information processing method according to the embodiment. Fig. 14 illustrates a case in which there are a model M101 trained using a data set DS101 of a domain D101, a model M102 trained using a data set DS102 of a domain D102, and a model M103 trained using a data set DS103 of a domain D104, and the domain of the facility where the method will be implemented is unknown.

[0116] The model M101 outputs a plurality of candidate items PIT, including items 45 and 26. The plurality of candidate items PIT are sorted in order of predicted value. The predicted value of the candidate item PIT may be applied to the probability that a user will take a positive action on the candidate item PIT, such as viewing or purchasing. In the model M101, the predicted value of item 45 is 0.6, and the predicted value of item 26 is 0.4.

[0117] Model M102 outputs a number of candidate items PIT, including items 35 and 36. The predicted value of item 35 is 0.7, and the predicted value of item 69 is 0.3.

[0118] Model M103 outputs a plurality of candidate items PIT, including item 49 and item 12. The predicted value of item 49 is 0.5, and the predicted value of item 12 is 0.4.

[0119] 14 are identification numbers of the candidate items PITs that are common to the candidate items PITs output from the model M101, the model M102, and the model M103. The candidate items PIT output from the model M101 can be understood as a recommended item list output from the model M101.

[0120] When multiple models are prepared using datasets from multiple diverse domains, it is assumed that the domain of the facility where the model is being implemented has attributes similar to those of one of the domains of the multiple models. Here, the model domain refers to the domain to which the dataset applied when training the model is provided. "Similar domain attributes" refers to cases where the domain attributes, which are understood as the characteristics of the domain, such as the user's age group, gender, and occupation, are the same or have commonalities in the domain attributes.

[0121] A model trained using a dataset of a domain whose attributes are similar to the domain of the facility where the model is being introduced can perform highly accurate recommendations. However, it is not clear which of the multiple domains has attributes similar to the domain of the facility where the model is being introduced.

[0122] Therefore, we want to generate a high-performance recommendation item list regardless of which of multiple models has a similar domain and attribute to the facility where it is introduced. High performance here means at least robust performance against domain shifts.

[0123] [Evaluation of the recommended item list based on the domain of the facility where the item was introduced] Fig. 15 is a schematic diagram showing a specific example of recommended item list evaluation. This figure shows an example in which a hit rate is applied as an evaluation index. The recommended item list IL120 shown in this figure includes recommended item IT201, recommended item IT202, recommended item IT203, recommended item IT204, recommended item IT205, and recommended item IT206, which are selected from a plurality of candidate items PIT output only from the model M102. Recommended item IT201 to recommended item IT206 are arranged from top to bottom in order of the predicted value of the user taking positive action.

[0124] Regarding the recommended item list IL120, if the domain corresponding to model M101 is close to the attributes of the domain of the facility where the model is to be installed, it is predicted that none of the recommended items IT201 to IT206 will be hit. In this case, the evaluation value of the recommended item list IL120 will be 0. The same applies when the domain corresponding to model M103 is close to the attributes of the domain of the facility where the model is to be installed.

[0125] On the other hand, for the recommended item list IL120, if the domain corresponding to the model M102 is close to the domain attribute of the facility where the model M102 is installed, it is predicted that all of the recommended items IT201 to IT206 will be hits. In this case, the evaluation value of the recommended item list IL120 will be 1.

[0126] The recommended item list IL122 includes a plurality of recommended items IT selected from the candidate items PIT output from each of the models M101, M102, and M103. Specifically, the recommended item list IL122 includes the recommended item IT101 and the recommended item IT102 output from the model M101.

[0127] The recommended item list IL122 includes recommended items IT201 and IT202 output from the model M102 and recommended items IT301 and IT302 output from the model M103.

[0128] In the recommended item list IL122, multiple recommended items IT are arranged from top to bottom in the following order: the top recommended item IT101 for model M101, the top recommended item IT201 for model M102, the top recommended item IT301 for model M103, the second recommended item IT302 for model M103, the second recommended item IT202 for model M102, and the second recommended item IT102 for model M101.

[0129] If the domain corresponding to model M101 is close to the attributes of the domain of the facility where the model is to be introduced, the evaluation value of recommended item list IL122 is 1. If the domain corresponding to model M102 is close to the attributes of the domain of the facility where the model is to be introduced, and if the domain corresponding to model M103 is close to the attributes of the domain of the facility where the model is to be introduced, the evaluation value of recommended item list IL122 is also 1.

[0130] In other words, the recommended item list IL120, which includes recommended items IT selected disproportionately from only the candidate items PIT of model M102, performs well when the attributes of the domain to which model M102 is introduced are close to those of the domain to which model M102 is introduced, but when the attributes of model M101 or model M103 are close to those of the domain to which model M102 is introduced, its performance drops compared to when the attributes of model M102 are close to those of the domain to which model M102 is introduced.

[0131] On the other hand, the recommended item list IL122, which includes recommended items IT selected in a balanced manner from the candidate items PIT of the model M101, the candidate items PIT of the model M102, and the candidate items PIT of the model M103, has high performance in all of the above cases.

[0132] 16 is a schematic diagram showing another specific example of the recommended item list evaluation. This figure shows an example in which reciprocal rank is applied as the evaluation index. The fractions attached to the recommended items IT201 etc. are the weights of the evaluation index for each recommended item IT.

[0133] If the domain corresponding to model M101 is close to the attributes of the domain of the facility where the model is to be introduced, the evaluation value of recommended item list IL120 is 0. Similarly, if the domain corresponding to model M103 is close to the attributes of the domain of the facility where the model is to be introduced, the evaluation value of recommended item list IL120 is 0. Furthermore, if the domain corresponding to model M102 is close to the attributes of the domain of the facility where the model is to be introduced, the evaluation value of recommended item list IL120 is 2.45.

[0134] When the domain corresponding to model M101 is close to the domain attributes of the facility where the model is introduced, the evaluation value of recommended item list IL122 is 1.17. When the domain corresponding to model M102 is close to the domain attributes of the facility where the model is introduced, the evaluation value of recommended item list IL122 is 0.70. When the domain corresponding to model M103 is close to the domain attributes of the facility where the model is introduced, the evaluation value of recommended item list IL122 is 0.58.

[0135] Similarly, when the hit rate is applied as an evaluation index, the recommended item list IL122, which includes recommended items IT selected in a balanced manner from multiple recommendation models, has higher performance than the recommended item list IL120, which includes recommended items IT selected in a biased manner toward recommended items IT from a given recommendation model.

[0136] [Examples of preferred recommended item selection] As a concrete example of the selection of preferred recommended items, consider a recommendation system for retail stores. As training data for the recommendation model, dataset DS11 of purchase history at store S1, dataset DS12 of purchase history at store S2, and dataset DS13 of purchase history at store S3 are available. The purchase history at each store is the behavioral history of users at each store.

[0137] When developing a recommendation system to be introduced in a newly opened store S4, there is no data set of purchase history for the store S4, which is the facility where the system will be introduced. Also, it is unclear whether the domain attributes of store S4 are closer to those of store S1, store S2, or store S3.

[0138] Models M11, M12, and M13 are prepared by training using datasets DS11, DS12, and DS13 for stores S1, S2, and S3, respectively. Models M11, M12, and M13 use a matrix decomposition model, which is a type of model-based collaborative filtering. The number of dimensions of the matrix is ​​100.

[0139] Models M11, M12, and M13 are trained by applying stochastic gradient descent, using the log loss for predicting whether or not a purchase will be made as an index. Note that log loss may also be referred to as logarithmic loss. Also, stochastic gradient descent may also be referred to as stochastic gradient descent.

[0140] Next, using model M11, model M12, and model M13, a recommended item list IL11, a recommended item list IL12, and a recommended item list IL13 are generated for each of the multiple users. The size of the recommended item list IL11, the recommended item list IL12, and the recommended item list IL13 is set to 6. The size of the recommended item list IL11, etc. means the number of recommended items IT included in the recommended item list IL11, etc.

[0141] A reciprocal rank is applied as an evaluation index for the recommended item list IL, and either model M11, model M12, or model M13 is made to have high performance. Specifically, for one-third of all users, the first and sixth recommended items IT of the recommended item list IL are selected from the recommended items IT of model M11.

[0142] In addition, the second and fifth recommended items IT of the recommended item list IL are selected from the recommended items IT of model M12, and the third and fourth recommended items IT are selected from the recommended items IT of model M13.

[0143] Furthermore, for a third of the users different from the above third of the users, the first and sixth recommended items IT in the recommended item list IL are selected from the recommended items IT of model M12. The second and fifth recommended items IT in the recommended item list IL are selected from the recommended items IT of model M13, and the third and fourth recommended items IT are selected from the recommended items IT of model M11.

[0144] For the remaining one-third of the users, the first and sixth recommended items IT in the recommended item list IL are selected from the recommended items IT of model M13, the second and fifth recommended items IT in the recommended item list IL are selected from the recommended items IT of model M11, and the third and fourth recommended items IT are selected from the recommended items IT of model M12.

[0145] That is, the recommended item IT of model M11, the recommended item IT of model M12, and the recommended item IT of model M13 are considered as candidate items that are candidates for the recommended items IT that make up the recommended item list IL. The recommended item list IL includes recommended items IT selected from a plurality of candidate items based on predetermined selection conditions.

[0146] As a result, even if the domain of store S4 has attributes similar to those of any of model M11, model M12, and model M13, the recommended item list IL will obtain similar evaluation values ​​in multiple evaluation indices such as hit rate and reciprocal rank, and will have similar performance. Therefore, the recommendation system that outputs the above-mentioned recommended item list IL can achieve recommendations that are robust against domain shifts.

[0147] [Example of information processing system configuration] Next, a configuration example of an information processing system applied to a recommendation system having robust performance against domain shifts will be described. Fig. 17 is a block diagram showing an example of the hardware configuration of an information processing system according to an embodiment.

[0148] The information processing device 100 is realized using computer hardware and software. The physical form of the information processing device 100 is not particularly limited, and may be a server computer, a workstation, a personal computer, a tablet terminal, or the like. Here, an example in which the processing functions of the information processing device 100 are realized using one computer will be described, but the processing functions of the information processing device 100 may also be realized by a computer system configured using multiple computers.

[0149] The information processing device 100 includes a processor 102 , a non-transitory tangible computer-readable medium 104 , a communication interface 106 , an input / output interface 108 , and a bus 110 .

[0150] The processor 102 includes a central processing unit (CPU) and may include a graphics processing unit (GPU). The processor 102 is connected to a computer-readable medium 104, a communication interface 106, and an input / output interface 108 via a bus 110.

[0151] The processor 102 reads various programs and data stored in the computer-readable medium 104 and executes various processes. The term "program" includes the concept of a program module and includes instructions equivalent to a program.

[0152] The computer-readable medium 104 is, for example, a storage device including a memory 112 as a main storage device and a storage 114 as an auxiliary storage device. The storage 114 is configured using, for example, a hard disk drive, a solid-state drive, an optical disk, a magneto-optical disk, a semiconductor memory, or the like. The storage 114 may also be configured using an appropriate combination of the above-mentioned devices. The storage 114 stores various programs, data, and the like.

[0153] A hard disk drive may be referred to as an HDD, which is an abbreviation of Hard Disk Drive, and a solid state drive may be referred to as an SSD, which is an abbreviation of Solid State Drive.

[0154] The memory 112 includes an area used as a working area for the processor 102 and an area for temporarily storing programs and various data read from the storage 114. A program stored in the storage 114 is loaded into the memory 112, and the processor 102 executes the instructions of the program, and the processor 102 functions as a means for performing various processes specified by the program.

[0155] The memory 112 stores various programs, such as a recommended item list generating program 120, executed by the processor 102, and various data. The recommended item list generating program 120 may include multiple programs.

[0156] That is, the recommended item list generation program 120 acquires multiple candidate items from multiple recommendation models and selects, as recommended items, from the multiple candidate items, a number of candidate items corresponding to a predetermined size of the recommended item list. When selecting recommended items from the multiple candidate items, the recommended item list generation program 120 may apply predetermined selection conditions.

[0157] Furthermore, the recommended item list generating program 120 applies specified sorting conditions to the recommended items, arranges the recommended items, and generates a recommended item list to be presented to the user.

[0158] When obtaining candidate items, candidate items may be obtained from each of a plurality of models stored in memory 112, or candidate items may be obtained from each of a plurality of models stored in a device external to information processing device 100.

[0159] A learning program for learning a plurality of models may be stored in the memory 112. The processor 102 may execute the learning program to learn a plurality of models. The memory 112 may also store learning data used when learning a plurality of models.

[0160] The memory 112 includes a candidate item storage unit 140. The candidate item storage unit 140 stores candidate items used by the recommended item list generating program 120.

[0161] The memory 112 includes a recommended item list storage unit 142. The recommended item list storage unit 142 stores a recommended item list generated when the processor 102 executes the recommended item list generating program 120.

[0162] The communication interface 106 performs communication processing with an external device via a wired or wireless connection, and exchanges information with the external device. The information processing device 100 is connected to a communication line via the communication interface 106.

[0163] The communication line may be a local area network, a wide area network, or a combination of these. The communication line is not shown. The communication interface 106 can function as a data acquisition unit that accepts input of various data, such as an original data set.

[0164] The information processing device 100 includes an input device 152 and a display device 154. The input device 152 and the display device 154 are connected to the bus 110 via the input / output interface 108. The input device 152 may be, for example, a keyboard, a mouse, a multi-touch panel, other pointing devices, a voice input device, or the like. The input device 152 may be an appropriate combination of the above-mentioned keyboards, etc.

[0165] The display device 154 may be, for example, a liquid crystal display, an organic EL display, a projector, or the like. The display device 154 may be an appropriate combination of the above-mentioned liquid crystal displays, etc. The input device 152 and the display device 154 may be integrally configured, like a touch panel, or the information processing device 100, the input device 152, and the display device 154 may be integrally configured, like a touch panel type tablet terminal. An organic EL display may be referred to as OEL, which is an abbreviation for organic electro-luminescence. The EL in organic EL display is an abbreviation for Electro-Luminescence.

[0166] 18 is a functional block diagram showing the functional configuration of the information processing system according to the embodiment. The information processing system 100 includes a candidate item acquisition unit 160, a recommended item selection unit 162, and a recommended item list generation unit 164.

[0167] The candidate item acquiring unit 160 acquires a plurality of candidate items from each of a plurality of recommendation models, and stores the plurality of candidate items in the candidate item storage unit 140.

[0168] The recommended item selection unit 162 selects, as recommended items, candidate items in a number corresponding to the size of the recommended item list from among the plurality of candidate items acquired by the candidate item acquisition unit 160 .

[0169] The recommended item list generating unit 164 generates a recommended item list using the multiple recommended items selected by the recommended item selecting unit 162. The recommended item list generating unit 164 stores the recommended item list in the recommended item list storage unit 142.

[0170] [Information processing procedures] Fig. 19 is a flowchart showing the procedure of the information processing method according to the embodiment. In the candidate item acquisition step S10, the candidate item acquisition unit 160 shown in Fig. 18 acquires a plurality of candidate items. In the candidate item acquisition step S10, the candidate item acquisition unit 160 stores the acquired plurality of candidate items in the candidate item storage unit 140. After the candidate item acquisition step S10, the process proceeds to the recommended item selection step S12.

[0171] In the candidate item acquisition step S10, a predicted value representing the probability that the user will perform positive behavior may be calculated for each candidate item, and a list of multiple candidate items may be acquired for each model, with the predicted values ​​sorted in descending order.

[0172] In the recommended item selection step S12, the recommended item selection unit 162 selects candidate items to be recommended items from the plurality of candidate items acquired in the candidate item acquisition step S10. After the recommended item selection step S12, the process proceeds to the recommended item list generation step S14.

[0173] In the recommended item list generating step S14, the recommended item list generating unit 164 generates a recommended item list using the multiple recommended items selected in the recommended item selecting step S12. In the recommended item list generating step S14, the recommended item list generating unit 164 stores the recommended item list in the recommended item list storage unit 142. After the recommended item list generating step S14, the information processing device 100 ends the procedure of the information processing method.

[0174] In the recommended item list generation step S14, a candidate list evaluation step may be executed to evaluate a plurality of candidate lists including the selected recommended items, and a recommended item list selection step may be executed to select a recommended item list from the plurality of candidate lists depending on the evaluation results of the candidate lists.

[0175] [Recommended item list generation method according to the first embodiment] 20 is a schematic diagram showing a recommended item list generating method according to the first embodiment. A recommended item list generating method for generating a recommended item list by applying the above-described information processing method and information processing system will be described in detail below.

[0176] In the recommended item list generating method according to the first embodiment, first, the processor 102 shown in Fig. 17 executes the recommended item list generating program 120 to acquire candidate items PIT from each of the models M101, M102, and M103. Fig. 20 shows an example in which two candidate items PIT are acquired from each of the models M101, M102, and M103.

[0177] The processor 102 acquires one or more candidate items PIT from each of the models M101, M102, and M103, thereby acquiring a plurality of candidate items PIT. The acquired plurality of candidate items PIT are stored in the candidate item storage unit 140.

[0178] Next, the processor 102 generates a list in which the acquired candidate items PIT are arranged in order of the predicted value of the recommended item for each model for each of the models M101, M102, and M103. The processor 102 may use a predicted value calculated in advance, or may calculate a predicted value.

[0179] Here, the predicted value may be the probability that a user will take positive action for each recommended item, such as viewing or purchasing. In the example shown in Fig. 20, for model M101, item 45 with a predicted value of 0.6 and item 26 with a predicted value of 0.4 are sorted in descending order of predicted value from the top.

[0180] Similarly, for model M102, item 35 with a predicted value of 0.7 and item 69 with a predicted value of 0.3 are sorted in descending order of predicted value from the top. For model M103, item 45 with a predicted value of 0.5 and item 12 with a predicted value of 0.4 are sorted in descending order of predicted value from the top.

[0181] The numbers attached to the candidate items PIT and the recommended items IT are identification numbers. Item 26, shown as the candidate item PIT of model M101, may exist as a candidate item with a low predictive value in each of models M102 and M103. The same is true for item 69 and item 12.

[0182] Next, the processor 102 selects the top candidate items PIT from each of the models M101, M102, and M103 as recommended items IT, and generates a recommended item list IL100 that has robust performance against domain shifts. In the example shown in Fig. 20, the recommended item list IL100 includes items 45, 35, and 49 as recommended items IT.

[0183] The recommended item list generation program 120 applied to the first embodiment includes a predicted value acquisition program that acquires predicted values ​​of candidate items for each model. The recommended item list generation program 120 also includes a list generation program that generates a list in which the candidate items for each model are sorted in order of predicted values. The recommended item list generation program 120 also includes an item selection program that selects the top candidate items from each list. Note that the acquisition of information may include the concept of generating information.

[0184] [Second embodiment Related How to generate a list of recommended items Fig. 21 is a schematic diagram showing a recommended item list generation method according to the second embodiment. In the recommended item list generation method according to the second embodiment, the processor 102 of Fig. 17 calculates statistics of predicted values ​​for each candidate item PIT, and sets the calculated statistics as the predicted value for each candidate item PIT. The processor 102 selects, from the multiple candidate items PITs, candidate items PITs to be set as recommended items IT, the number of which corresponds to the size of the recommended item list, in descending order of the statistical values ​​of the predicted values ​​for each candidate item PIT.

[0185] It is assumed that any one of model M101, model M102, and model M103 shown in FIG. 21 has high accuracy. A model having high accuracy means that candidate items with high predictions are actually viewed by users. An example of a candidate item with high predictions is a candidate item with a relatively large prediction value. The same applies to other embodiments described using FIG. 22, etc.

[0186] For example, consider a case where 100 candidate items with item identification numbers ranging from 1 to 100 are acquired for each of model M101, model M102, and model M103. Processor 102 calculates statistics of the predicted values ​​of each candidate item for each model. The statistics may be values ​​calculated using any statistical indicator, such as the mean, maximum, or median. The mean value may be an arithmetic mean value. Figure 21 shows a case where the mean value of the predicted values ​​of each candidate item for each model is calculated.

[0187] For example, 0.28 is calculated as the average of the predicted value of item 1 of model M101, the predicted value of item 1 of model M102, and the predicted value of item 1 of model M103. Similarly, 0.05 is calculated as the average of the predicted value of item 2 of model M101, the predicted value of item 2 of model M102, and the predicted value of item 2 of model M103. In this way, the average of the predicted values ​​for all candidate items PIT from item 1 to item 100 is calculated, and the calculated average of the predicted values ​​becomes the predicted value for each candidate item.

[0188] Next, the processor 102 sorts items 1 to 100 from top to bottom in descending order of the predicted values ​​to which the average value is applied, and selects, as recommended items IT, the number of candidate items PIT corresponding to the size of the recommended item list IL100, starting from the top candidate item PIT. Fig. 21 illustrates the recommended item list IL100 in which items 45, 35, and 49 are applied to the recommended item IT.

[0189] The recommended item list generating program 120 applied to the second embodiment includes a statistical value obtaining program for obtaining statistical values ​​of predicted values ​​for each item, in addition to a predicted value obtaining program, a list generating program, and an item selecting program.

[0190] [In the third embodiment Related How to generate a list of recommended items FIG. 22 is a schematic diagram showing a recommended item list generation method according to the third embodiment. In the recommended item list generation method according to the third embodiment, the processor 102 of FIG. 17 obtains a list in which candidate items PIT for each model are arranged in descending order of predicted value, and selects the top k candidate items PIT of each model's list as recommended items IT. The size of the recommended item list IL102 is k × the number of models. In other words, the processor 102 selects as recommended items IT the number of candidate items PIT obtained by dividing the number corresponding to the size of the recommended item list IL102 by the number of models. Note that k is a positive integer greater than or equal to 2.

[0191] Figure 22 shows an example in which item 45 with the highest predicted value in model M101, item 35 with the highest predicted value in model M102, and item 49 with the highest predicted value in model M103 are selected as recommended items IT in recommended item list IL102.

[0192] 22, if the model M101 has high accuracy, the item 45 will be hit. Also, if the model M102 has high accuracy, the recommended item list IL102 will hit the item 35, and if the model M103 has high accuracy, the item 49 will be hit.

[0193] The recommended item list generating program 120 applied to the third embodiment includes a predicted value obtaining program, a list generating program, and an item selecting program, similar to the first embodiment.

[0194] [Recommended item list generation method according to the fourth embodiment] Fig. 23 is a schematic diagram showing a recommended item list generation method according to the fourth embodiment. In the recommended item list generation method according to the fourth embodiment, the processor 102 in Fig. 17 obtains a list in which candidate items for each model are arranged in descending order of predicted value.

[0195] When the size of the recommended item list IL104 is large compared to the number of models, the recommended items IT are arranged in a balanced manner, taking into consideration the weights according to the rankings of the recommended items IT in the recommended item list IL104.

[0196] The processor 102 applies the procedure for generating the recommended item list IL104 described above to generate multiple candidate lists that serve as candidates for the recommended item list IL104. For each of the multiple candidate lists, the processor 102 calculates an evaluation value that changes depending on the compatibility between the domain of the facility where the model is to be installed and the domain corresponding to each model. The candidate lists are illustrated in FIG. 24 as candidate list PIL100, etc.

[0197] Here, the compatibility between the domain of the facility where the system is to be introduced and the domain corresponding to each model can be understood as the similarity of the attributes between the domain of the facility where the system is to be introduced and the domain corresponding to each model. For example, if the similarity between the domain of the facility where the system is to be introduced and the domain corresponding to each model is relatively high, it may be determined that the compatibility is good and the attributes are close. The processor 102 compares the minimum evaluation value for each candidate list and determines the candidate list with the largest minimum evaluation value as the recommended item list IL104.

[0198] The recommended item list generation program 120 applied to the fourth embodiment includes a predicted value acquisition program, a list generation program, and an item selection program applied to the fourth embodiment.

[0199] The recommended item list generating program 120 includes a candidate list generating program that generates a list of multiple candidates to be included in the recommended item list IL104, and an evaluation value calculation program that calculates an evaluation value.

[0200] [Recommended item list generation method according to the fifth embodiment] 24 is a schematic diagram showing a recommended item list generation method according to the fifth embodiment. As the recommended item list generation method according to the fifth embodiment, a specific example of candidate list evaluation in the recommended item list generation method according to the fourth embodiment is shown.

[0201] 17 defines the virtual user's behavior regarding the recommended item list depending on which of the multiple models the domain of the facility where the system is installed has the closest attributes to. That is, a user with a similar domain attribute is assumed to perform a positive action such as browsing with a 100% probability, and the user's behavior is deterministically attempted.

[0202] The processor 102 generates a plurality of candidate lists in which the order of the recommended items obtained from the plurality of models is changed, and calculates an evaluation value of a predetermined evaluation index for each of the plurality of candidate lists for each of a plurality of hypothetical user actions. FIG. 24 shows a candidate list PIL100 and a candidate list PIL102 as the plurality of candidate lists. FIG. 24 shows a schematic example of calculating evaluation values ​​for the candidate list PIL100 when each of the models M101, M102, and M103 in FIG. 23 has an attribute close to the domain of the facility where the model is introduced.

[0203] The candidate list PIL100 has six recommended items IT arranged in the following order: the first item of model M101, the first item of model M102, the first item of model M103, the second item of model M101, the second item of model M102, and the second item of model M103.

[0204] In addition, the candidate list PIL102 arranges the six recommended items IT in the following order: the number one item of model M101, the number one item of model M102, the number one item of model M103, the number two item of model M103, the number two item of model M102, and the number two item of model M101.

[0205] 24 illustrates reciprocal rank as an evaluation index for the candidate list PIL100 and the candidate list PIL102. For the candidate list PIL100, when the attribute of the model M101 is close to the domain of the facility where the model is to be introduced, the evaluation value is calculated as 1 + (1 / 4) = 1.25. Similarly, when the attribute of the model M102 is close to the domain of the facility where the model is to be introduced, the evaluation value is calculated as 0.70, and when the attribute of the model M103 is close to the domain of the facility where the model is to be introduced, the evaluation value is calculated as 0.50. The minimum evaluation value of the candidate list PIL100 is 0.50.

[0206] Furthermore, for the candidate list PIL102, when the attribute of model M101 is close to the domain of the facility where the model is to be introduced, the evaluation value is 1.17. When the attribute of model M102 is close to the domain of the facility where the model is to be introduced, the evaluation value is 0.70. When the attribute of model M103 is close to the domain of the facility where the model is to be introduced, the evaluation value is 0.58. The minimum evaluation value of the candidate list PIL100 is 0.58.

[0207] When the minimum evaluation value of candidate list PIL100, 0.50, is compared with the minimum evaluation value of candidate list PIL102, 0.58, the evaluation value of candidate list PIL102, 0.58, is larger, so candidate list PIL102 is the best of the multiple candidate lists and is set as the recommended item list IL.

[0208] The best ordering of recommended items in the candidate list can be determined according to the size of the candidate list, the number of models, and the evaluation index. The evaluation value of the candidate list may be calculated for one or more arbitrary users, and does not have to be calculated for each of multiple users.

[0209] The recommended item list generating program 120 applied to the fifth embodiment includes, as an evaluation value calculation program, a candidate list determination program that determines a candidate list that maximizes the minimum value of the evaluation value for each evaluation condition.

[0210] [Recommended item list generation method according to the sixth embodiment] 25 is a schematic diagram showing a recommended item list generation method according to the sixth embodiment. As the recommended item list generation method according to the sixth embodiment, another specific example of candidate list evaluation in the recommended item list generation method according to the fourth embodiment is shown.

[0211] 25 illustrates reciprocal rank as an evaluation index for the candidate list PIL100 and the candidate list PIL102. In the sixth embodiment, a probabilistic hypothetical case is considered in which, as hypothetical user behavior with respect to a recommended item list, a user with a similar domain attribute will perform positive behavior such as browsing with a 40 percent probability, and a user with a distant domain attribute will perform positive behavior such as browsing with a 20 percent probability.

[0212] For the candidate list PIL100, if the attributes of model M101 are close to the domain of the facility where it is installed, the evaluation value is calculated as 1 x 0.4 + (1 / 2) x 0.2 + (1 / 3) x 0.2 + (1 / 4) x 0.4 + (1 / 5) x 0.2 + (1 / 6) x 0.2 = 0.74.

[0213] Similarly, when the attribute of model M102 is close to the domain of the facility where the model is introduced, the evaluation value is calculated as 0.63, and when the attribute of model M103 is close to the domain of the facility where the model is introduced, the evaluation value is calculated as 0.59. The minimum evaluation value of the candidate list PIL100 is 0.59.

[0214] Furthermore, for the candidate list PIL102, when the attribute of model M101 is close to the domain of the facility where the model is to be introduced, the evaluation value is calculated as 0.72, when the attribute of model M102 is close to the domain of the facility where the model is to be introduced, the evaluation value is calculated as 0.63, and when the attribute of model M103 is close to the domain of the facility where the model is to be introduced, the evaluation value is calculated as 0.61. The minimum evaluation value of the candidate list PIL102 is 0.61. The candidate list PIL102 is used as the recommended item list IL.

[0215] [Recommended item list generation method according to the seventh embodiment] 26 is a schematic diagram showing a recommended item list generation method according to the seventh embodiment. As the recommended item list generation method according to the seventh embodiment, an example of a method for estimating the probability of user behavior in the recommended item list generation method according to the sixth embodiment is shown.

[0216] The probability that a user with similar domain attributes will take positive action is defined as the first probability, and the probability that a user with distant domain attributes will take non-constant action is defined as the second probability. For model M101, a list of items sorted in order of the model M101's predicted value is evaluated using the dataset DS100 of the first domain applied for learning, and the view rate is derived. For example, the view rate is 0.4.

[0217] In addition, a list of items arranged in order of the predicted values ​​of the model M101 is evaluated using a dataset DS102 of a second domain different from the first domain applied to learning, and the view rate is derived. For example, the view rate is 0.2.

[0218] As the first probability of the model M101, the viewing rate of the list in which the items are sorted in order of the predicted value of the model M101 using the dataset DS100 of the domain applied to learning is applied.

[0219] In addition, as the second probability of the model M101, the viewing rate of a list in which items are arranged in order of predicted values ​​of the model M101 using a dataset DS102 of a second domain different from the first domain applied for learning is applied.

[0220] Similarly, as the first probability of the model M102, a browsing rate of 0.4 for a list in which items are sorted in order of the predicted values ​​of the model M102 using the dataset DS110 of the first domain applied for learning may be applied.

[0221] Furthermore, as the second probability of model M102, the browsing rate of a list in which items are arranged in order of the predicted values ​​of model M102 using dataset DS112 of a second domain different from the first domain applied for learning may be applied as 0.2. Similarly, for other models such as model M103 shown in FIG. 17, the first probability and the second probability may be defined using the browsing rate.

[0222] 26 shows an example in which the first probabilities of the models M101 and M102 are the same, but the first probabilities of the models M101 and M102 may be different. The same applies to the second probabilities.

[0223] For example, the second probability may be calculated as described above, and the first probability may be the arithmetic mean value of the viewing rate derived using the dataset of the first domain applied to learning and the viewing rate derived using the dataset of a second domain different from the first domain applied to learning.

[0224] The derivation of the first probability above is based on the idea that the viewing rate of a domain whose attributes are similar to those of the domain of the facility where the program is being introduced will not be the same as the viewing rate of the domain of the facility where the program is being introduced, but will be somewhere between the viewing rate of the domain of the facility where the program is being introduced and the viewing rate of a domain whose attributes are distant from those of the domain of the facility where the program is being introduced.

[0225] [Recommended item list generation method according to the eighth embodiment] FIG. 27 is a schematic diagram showing a recommended item list generation method according to the eighth embodiment. In the operation of a recommendation system, there may be occasions when recommendation information is provided to the same user multiple times. In the recommended item list generation method according to the eighth embodiment, the order of recommended items is changed each time a recommended item list is generated for the same user. This makes it possible to provide recommendation information of a certain level of quality or higher on average to each user. Note that changing the order of recommended items each time a recommended item list is generated is an example of changing the order of recommended items each time a recommendation item list is presented.

[0226] FIG. 27 shows a first recommended item list IL120 and a second recommended item list IL122 for the same user. 、 The evaluation values ​​of the first and second recommended item lists IL122 are shown in Table 1. The evaluation index is the reciprocal rank.

[0227] The evaluation values ​​of the first recommended item list IL120 are arranged in the same order as the candidate items PIT in the candidate list PIL102 shown in Fig. 24, and six recommended items IT are arranged in the same order. The evaluation values ​​of the recommended item list IL120 are the same as the evaluation values ​​of the candidate list PIL102 shown in Fig. 24, and therefore a description thereof will be omitted here.

[0228] The second recommended item list IL122 will be arranged in the following order: the first item of model M103, the first item of model M102, the first item of model M101, the second item of model M101, the second item of model M102, and the second item of model M103.

[0229] For recommended item list IL122, the evaluation value is 0.58 when model M101 has attributes close to the domain of the facility where it is introduced. The evaluation value is 0.70 when model M102 has attributes close to the domain of the facility where it is introduced. The evaluation value is 1.17 when model M103 has attributes close to the domain of the facility where it is introduced.

[0230] The average evaluation value of recommended item list IL120 and recommended item list IL122 when model M101 has attributes close to the domain of the facility where it is introduced is (1.17 + 0.58) / 2 = 0.87. The average evaluation value of model M102 when its attributes close to the domain of the facility where it is introduced is 0.70, and the average evaluation value of model M103 when its attributes close to the domain of the facility where it is introduced is 0.87. The minimum average evaluation value of recommended item list IL120 and recommended item list IL122 is 0.70.

[0231] On the other hand, when the recommended item list IL120 or the recommended item list IL122 is provided to the same user multiple times, the minimum average value of the evaluation values ​​is 0.58. Therefore, the recommended item list generation method according to the eighth embodiment can maximize the minimum average value of the evaluation values ​​for each evaluation condition, which is the evaluation condition as to which model's domain is closest to the attributes of the facility where the model is introduced and its domain.

[0232] [Recommended item list generation method according to the ninth embodiment] FIG. 28 is a schematic diagram showing a recommended item list generation method according to the ninth embodiment. In the recommended item list generation method according to the ninth embodiment, the order of recommended items constituting the recommended item list is changed for each user. Note that user 1 shown in FIG. 28 is the first user, and user 2 is the second user. The change in the order of recommended items for each user is an example of the change in the order of recommended items for each presentation.

[0233] 28 shows an example in which a recommended item list IL120 is presented to a first user, and a recommended item list IL122 is presented to a second user. Note that the recommended item list IL120 and the recommended item list IL122 shown in FIG. 28 are the same as the recommended item list IL120 and the recommended item list IL122 shown in FIG.

[0234] The specific example of the evaluation value shown in Fig. 28 corresponds to the case where the first time shown in Fig. 27 is replaced with the first user and the second time is replaced with the second user. Here, a description of the specific example of the evaluation value will be omitted.

[0235] The recommended item list generation method according to the ninth embodiment changes the order of recommended items in the recommended item list for each user, and can provide recommendation information of a certain level of quality or higher on average to all users. Even when presenting a recommended item list to each user only once, it is preferable to change the order of recommended items in the recommended item list.

[0236] [Recommended item list generation method according to the tenth embodiment] 29 is a schematic diagram showing a recommended item list generation method according to the tenth embodiment. In the recommended item list generation method according to the tenth embodiment, when there are multiple models that are highly similar to one another, recommended items are selected from the candidate items of one of the similar models, and recommended items are not selected from the candidate items of the other model. In other words, recommended items are selected preferentially from the candidate item list obtained from dissimilar models.

[0237] It should be noted that the candidate lists obtained from each of the similar models are an example of a similar candidate list, and the candidate lists obtained from each of the dissimilar models are an example of a dissimilar candidate list.

[0238] 29 illustrates models M102 and M104 as highly similar models. The recommended item list IL136 includes recommended item IT131 from which a candidate item of model M101 is selected and recommended item IT133 from which a candidate item of model M103 is selected.

[0239] The recommended item list IL136 also includes a recommended item IT122 in which a candidate item of the model M102 or a candidate item of the model M104 is selected. Fig. 29 illustrates an example in which an item 35 of the model M102 or an item 35 of the model M104 is selected as the recommended item IT132.

[0240] The similarity of the models can be determined based on the similarity between the domains, the similarity of the generated recommended item lists, etc. For example, the following procedure can be applied to evaluate the similarity of the domains corresponding to the learning data of the models M101 to M104 shown in FIG.

[0241] First, characteristics of each domain are extracted from a dataset of user attributes and item attributes. For example, the average age of users is extracted from the user attributes, and the average price of items is extracted from the item attributes. The domain characteristics extracted from the dataset may be statistics and distributions extracted from metadata such as explanatory variables.

[0242] Next, the characteristics of each domain are extracted from external information separate from the dataset. For example, relevant information outside the dataset, such as the floor area of ​​the facility, which is the domain, and the average annual household income of the city or town where the facility is located, is extracted.

[0243] Next, the characteristics of each domain are expressed as a multidimensional vector using the multiple types of numerical values ​​representing the characteristics obtained in the above process. For example, the characteristics of each domain are expressed as a four-dimensional characteristic vector with variables including the average age of users, the average price of items, the floor area of ​​the facility, and the average annual household income of the facility.

[0244] Next, the similarity of the characteristic vectors in the vector space is calculated. When evaluating the similarity of the characteristic vectors, the values ​​of each dimension of the characteristic vectors are normalized to align the range of values ​​of each dimension.

[0245] Next, the Euclidean distance between the characteristic vectors of each domain is calculated. Domains whose Euclidean distance between the characteristic vectors is less than a predetermined distance can be determined to be similar. Note that the determination of the similarity of models is not limited to the above example. For example, a mode may be applied in which the characteristics of each domain are treated as multidimensional vectors, and external information is not used as parameters, but only explanatory variables are used as parameters.

[0246] 17 may include a selection program that selects recommended items from candidate items based on similarity between domains. The selection program may include a similarity determination program that determines similarity between domains.

[0247] [Recommended item list generation method according to the 11th embodiment] 30 is a schematic diagram showing an example of a recommended item list generated by applying the recommended item list generation method according to the 11th embodiment. In the recommended item list generation method according to the 11th embodiment, weights are defined based on the ranking order of recommended items depending on the user.

[0248] The figure shows an example of a recommended item list IL140 that includes six recommended items IT, recommended item 1 to recommended item 6. Assuming a user who frequently views high-ranking recommended items such as recommended item 1 in the recommended item list IL140, but rarely views low-ranking recommended items such as recommended item 6, the figure shows an example in which the square of the reciprocal of the ranking order is applied as a weight that decays relatively more as the ranking goes lower.

[0249] Fig. 31 is a schematic diagram showing another example of a recommended item list generated by applying the recommended item list generation method according to the 11th embodiment. This figure illustrates a recommended item list IL142 that has different weights based on ranking order from IL140 shown in Fig. 31.

[0250] The recommended item list IL142 assumes a user who browses through all recommended items IT, from the top recommended item IT, such as recommended item 1, to the bottom recommended item IT, such as recommended item 6, and shows an example in which the square root of the reciprocal of the ranking order is applied as a weight that decays relatively less as the ranking order goes lower.

[0251] [First example of multiple models] 32 is an explanatory diagram of a first specific example of a plurality of models. A plurality of different models that output candidate items are trained by applying data sets in different domains as training data.

[0252] The figure illustrates model M101, which is a trained model trained using dataset DS101 of domain D101, model M102, which is a trained model trained using dataset DS102 of domain D102, and model M103, which is a trained model trained using dataset DS103 of domain D103.

[0253] [Second example of multiple models] 33 is an explanatory diagram of a second specific example of a plurality of models. Consider a case where data sets cannot be acquired from a plurality of mutually different domains, and a data set DS100 is acquired from one domain D100.

[0254] For a data set obtained from one domain, multiple different feature sets are extracted, and learning is performed using the multiple different feature sets as training data to generate multiple different models.

[0255] 33 illustrates a model M201 generated using a first feature set extracted from a dataset of one domain, a model M202 generated using a second feature set, and a model M203 generated using a third feature set. Note that feature set 1, feature set 2, and feature set 3 illustrated in FIG. 33 correspond to the first feature set, the second feature set, and the third feature set, respectively.

[0256] Figure 34 is a list of variables. The figure shows examples of user attributes, item attributes, and contexts as variables that can be used as explanatory variables. The figure shows an example of user attribute 1, such as a department of medicine, and an example of user attribute 2, such as a doctor's occupation.

[0257] In addition, the figure shows an example of the type of examination, such as CT, as item attribute 1, an example of the patient's gender as item attribute 2, an example of whether the patient has been hospitalized as context 1, and an example of the time elapsed since the item was created as context 2. CT is an abbreviation for Computed Tomography.

[0258] For example, explanatory variables other than the medical department may be applied as a feature set to the training data of model M201 in Fig. 33. Model M201 has a certain degree of robust performance even when the relationship between the medical department and browsing changes.

[0259] Explanatory variables other than job type may be applied as a feature set to the training data of model M202. Model M202 has a certain degree of robustness even when the relationship between job type and browsing changes.

[0260] Explanatory variables other than hospitalization status can be applied as a feature set to the training data of model M203. Model M202 has a certain degree of robustness even when the relationship between hospitalization status and browsing changes.

[0261] It is assumed that the relationship between explanatory variables and the objective variable will change due to the domain shift. However, it is difficult to grasp which explanatory variables have changed in relation to the objective variable. Therefore, even if the relationship between which explanatory variables has changed in relation to the objective variable, it is necessary to determine whether the relationship between which explanatory variables has changed in relation to the objective variable will be correct in any of the multiple models. Several models are prepared, each of which is appropriate.

[0262] [Effects of the embodiment] The information processing device and the information processing method according to the embodiment can achieve the following advantageous effects.

[0263] [1] In an information processing device that performs information recommendation by recommending a plurality of recommended items to a user, one or more candidate items are acquired from each of a plurality of mutually different models that have been trained using different data sets as training data, and a plurality of candidate items including candidate items from the different models are selected as recommended items from the plurality of candidate items.

[0264] This generates a list of recommended items that is robust to domain shifts.

[0265] [2] Candidate items are sorted in descending order of predicted value for each model, and recommended items are selected starting from the top candidate item for each model. This allows for a balanced selection of candidate items to be recommended from each model.

[0266] [3] If the size of the recommended item list is larger than the number of models, weights based on the ranking order of the candidate items are taken into consideration when selecting recommended items.

[0267] [4] The statistical value of the predicted values ​​for each model for each candidate item is used as the predicted value for the candidate item. Recommended items are selected in descending order of predicted values. This allows for a balanced selection of candidate items that will become recommended items from each model.

[0268] [5] Multiple candidate lists are generated, each consisting of multiple recommended items, each containing one or more recommended items selected from each model. For each of the multiple candidate lists, multiple evaluation values ​​are calculated, each with different attribute similarities between the domain of the facility where the system is installed and the domain of each model. The candidate list with the largest minimum evaluation value for each evaluation condition is extracted and used as the recommended item list. As a result, a recommended item list with robust performance against domain shifts is extracted based on the evaluation values ​​of the candidate lists.

[0269] [6] When providing multiple recommended item lists to the same user, the order of the recommended items in each recommended item list is changed, thereby providing a recommended item list of consistent quality to the target user on average.

[0270] [7] The order of recommended items in the recommended item list can be changed for each user, which results in a consistent quality of recommended item list for all users.

[0271] [8] Among the multiple models, one of the candidate items from each of the multiple models with relatively high similarity is selected as the recommended item, thereby selecting candidate items from each model in a balanced manner to become recommended items.

[0272] [9] The weights according to the ranking order of the candidate list are changed depending on the characteristics of the user, thereby calculating the evaluation value of the candidate list taking the characteristics of the user into consideration.

[0273]

[10] Each of the multiple models is a trained model that has been trained using datasets from different domains, thereby obtaining candidate items that can be used in a wide variety of domains.

[0274]

[11] Each of the multiple models is a trained model that is trained using multiple different feature sets in a dataset for one domain, thereby obtaining candidate items that can be used in a wide variety of domains even when datasets for multiple domains are not available.

[0275] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0276] 10 Recommendation Systems 12 Predictive Models 14 models 100 Information processing device 102 processors 104 Computer-readable medium 106 Communication Interface 108 Input / Output Interface 110 Bus 112 memory 114 Storage 120 Recommended Item List Generation Program 140 Candidate Item Memory Unit 142 Recommended item list storage unit 152 Input Device 154 Display device 160 Candidate Item Acquisition Section 162 Recommended Item Selection Section 164 Recommended Item List Generation Unit D101 Domain D102 Domain D103 Domain DS1 dataset DS2 dataset DS3 dataset DS101 dataset DS102 dataset DS103 dataset DS110 dataset DS112 dataset Dtg Data IT Recommended Items IT1 Item IT2 Items IT3 items IT101 Recommended Items IT102 Recommended Items IT131 Recommended Items IT132 Recommended Items IT133 Recommended Items IT201 Recommended Items IT202 Recommended Items IT203 Recommended Items IT204 Recommended Items IT205 Recommended Items IT206 Recommended Items IT301 Recommended Items IT302 Recommended Items IL Recommended Item List IL100 Recommended Item List IL102 Recommended Item List IL104 Recommended Item List IL110 Recommended Item List IL112 Recommended Item List IL114 Recommended Item List IL120 Recommended Item List IL122 Recommended Item List IL136 Recommended Item List IL140 Recommended Item List IL142 Recommended Item List M1 model M2 model M3 model M101 model M102 model M103 model M201 model M202 model PIL100 Candidate List PIL102 Candidate List PIT candidate items S10 to S14: Each step of the information processing method

Claims

1. An information processing method in which an information processing system including one or more processors generates a recommended item list that recommends a plurality of items to a user, the method comprising: The information processing system, Obtain one or more candidate items from each of a plurality of models trained using datasets in one or more domains different from the domain to which the model is to be introduced; An information processing method for generating a recommended item list that includes a plurality of the recommended items and has robust performance against domain shifts, by selecting a plurality of the candidate items that have different domains from each other as recommended items from among the plurality of acquired candidate items.

2. The information processing system, calculating a predicted value of the user's behavior for each of the candidate items; The information processing method according to claim 1, wherein the recommended item is selected from a plurality of the candidate items based on the ranking of statistical values ​​calculated from the predicted values ​​of the same candidate item in each of a plurality of domains different from the destination domain.

3. The information processing system, deriving an evaluation value for each of a plurality of candidate lists that are candidates for the recommended item list according to the similarity of attributes between the introduction destination domain and each of the plurality of domains; The information processing method according to claim 1 , wherein the candidate list with the largest minimum evaluation value is set as the recommended item list.

4. 4. The information processing method according to claim 3, wherein the information processing system assumes that a user will behave positively toward the candidate items of the model trained using data from a domain whose attributes are similar to those of the destination domain, and that the user will behave negatively toward the candidate items of the model trained using data from a domain whose attributes are distant from those of the destination domain, and deterministically tests the user's behavior to calculate the evaluation value for each candidate list.

5. 4. The information processing method according to claim 3, wherein the information processing system probabilistically tests the user's behavior and calculates the evaluation value for each candidate list, assuming that the user will take positive action with a first probability toward the candidate items of the model trained using as training data a dataset from a domain whose attributes are similar to those of the destination domain, and that the user will take positive action with a second probability toward the candidate items of the model trained using as training data a dataset from a domain whose attributes are distant from those of the destination domain.

6. The information processing system, For each of the plurality of models, estimate the first probability from an evaluation result obtained by evaluating the model in a first domain to which the dataset is applied as training data; The information processing method according to claim 5 , wherein the second probability is estimated from an evaluation result obtained by evaluating each of the plurality of models in a second domain different from the first domain.

7. The information processing method according to claim 3 , wherein the information processing system calculates the evaluation value based on a user's behavior when the candidate list is presented to the user in the destination domain.

8. The information processing method according to claim 3, wherein the information processing system calculates the evaluation value for each candidate list by applying a weight that is defined for each candidate item according to the ranking of the candidate item included in the candidate list and that is defined according to an evaluation condition.

9. The information processing method according to claim 3 , wherein the information processing system selects one or more of the candidate items from each of a plurality of the candidate lists.

10. The information processing method according to claim 3 , wherein the information processing system selects the candidate items to be the recommended items by giving priority to dissimilar candidate lists from among the plurality of candidate lists.

11. The information processing method according to any one of claims 1 to 8, wherein when the information processing system presents the recommended item list to the same user multiple times, the order of the multiple recommended items included in the recommended item list is changed for each presentation.

12. The information processing method according to claim 1 , wherein the information processing system, when presenting the recommended item list multiple times, rearranges the order of the recommended items included in the recommended item list for each presentation.

13. The information processing method according to claim 1 , wherein the information processing system applies, as the plurality of models, trained models trained using data sets in mutually different domains as training data.

14. The information processing method according to claim 1 , wherein the information processing system applies, as the multiple models, trained models that have been trained using different sets of features in a domain different from the domain to which the information processing system is introduced as training data.

15. An information processing system for generating a recommended item list that recommends one or more items to a user, one or more processors; one or more memories in which programs to be executed by the one or more processors are stored; Equipped with The one or more processors execute instructions of the program, Obtain one or more candidate items from each of a plurality of models trained using datasets in one or more domains different from the domain to which the model is to be introduced; An information processing system that selects, from the plurality of acquired candidate items, a plurality of candidate items that have different domains as recommended items, and generates a recommended item list that includes the plurality of recommended items and has robust performance against domain shifts.

16. A program for generating a recommended item list that recommends one or more items to a user, The computer A function of obtaining one or more candidate items from each of a plurality of models trained using datasets in one or more domains different from the domain to which the model is introduced; and A program that realizes a function of selecting, from the plurality of acquired candidate items, a plurality of candidate items having different domains as recommended items, and generating a recommended item list that includes the plurality of recommended items and has robust performance against domain shifts.

Images