Information processing method, information processing device, and program

Abstract

To provide information processing method, and an information processing equipment and program that can prepare a high-performance model for facilities at an unknown introduction destination even when a domain of the facility at the introduction destination is not known in a stage of learning a model.SOLUTION: The present invention relates to an information processing method that one or more processors implement, and the information processing method includes: representing, by the one or more processors, a plurality of second facilities different from first facilities having gathered datasets used to learn a model whose action on an item of a user is predicted; and learning, according to characteristics of the respective second facilities, a plurality of models so as to improve prediction performance at the respective second facilities.SELECTED DRAWING: Figure 25

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing device, and a program, and in particular to an information recommendation technology that makes recommendations robust against domain shifts. [Background technology]

[0002] In systems that provide users with a variety of items, such as electronic commerce (EC) sites or document information management systems, it is difficult in terms of both time and cognitive ability for users to select the best item from the many items available. In the case of EC sites, items are the products sold on the site, while in the case of document information management systems, items are the document information stored in the system.

[0003] Research is being conducted into recommendation technology, which presents selection candidates from a large number of items to assist users in selecting items. Generally, when a recommendation system is introduced into a facility, the model is trained based on data collected at the facility. However, if the same recommendation system is introduced into a facility other than the facility where the data used for training was collected, the model's predictive accuracy can decline. The problem of machine learning models not functioning well in unknown facilities is called domain shift, and in recent years, research into domain generalization, which is research into improving robustness against domain shifts, has become active, particularly in the field of image recognition. However, there are still very few examples of research into domain generalization in recommendation technology.

[0004] Non-Patent Document 1 describes a method for selecting a model to be used for transfer learning, i.e., a pre-trained model for fine-tuning, from models trained in several different languages ​​in interlingual transfer learning applied to cross-language translation.

[0005] Patent Document 1 describes a method of classifying users into a plurality of groups and generating a prediction model for providing a service for each group through federated learning.

[0006] Patent Document 2 describes a method for grouping users using user attributes and Dirichlet processes and generating a prediction model for each group, with the aim of reducing the time required to predict the online behavior of users operating their user terminals.

[0007] Patent Document 3 describes a method for selecting learning data used to generate AI (Artificial Intelligence) for medical facilities such as hospitals, based on attribute information of medical data. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2021-197181 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-062509 [Patent Document 3] Patent Publication No. 2021-086558 [Non-patent literature]

[0009] [Non-Patent Document 1] Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang,Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig “Choosing Transfer Languages ​​for Cross-Lingual Learning”(ACL 2019) [Non-patent document 2] Ivan Cantador, Ignacio Fenandez-Tobias, Shlomo Bwrkovsky, Paolo Cremonesi, Chapter 27:"Cross-domain Recommender System" (2015 Springer) Summary of the Invention [Problem to be solved by the invention]

[0010] Non-Patent Document 1 does not consider whether the candidate models for transfer learning sufficiently cover all possible target languages. Therefore, there may be cases where an appropriate model does not exist depending on the target language. In this regard, Non-Patent Document 1 is not about domain generalization, but is a study on transfer learning, i.e., domain adaptation, and since model learning is also carried out in the target language, the possibility that an appropriate model does not exist among multiple candidate models is not a major issue.

[0011] In Patent Document 1, multiple prediction models are prepared corresponding to multiple groups, but since each model is trained using data from a portion of users, it does not necessarily include models suitable for unknown groups outside the trained group.

[0012] In Patent Document 2, a prediction model suitable for a user is selected from prediction models generated for each group. In Patent Document 2, the purpose of dividing users into groups is to reduce the number of explanatory variables required for the prediction model and shorten the calculation time for predicted values. In the technology described in Patent Document 2, similar to Patent Document 1, multiple prediction models are prepared corresponding to multiple groups, but each model is trained using data from some users, so it does not necessarily include models suitable for unknown groups outside the trained group.

[0013] In Patent Document 3, training data is selected so that there is minimal bias in the attributes of the medical data, and so that the training data has an attribute distribution similar to that of test data from medical facilities that use trained AI. The technology described in Patent Document 3 builds a model assuming a facility different from the facility from which the training data was obtained, but only a single model is prepared, and its effectiveness is limited to cases where the facility where the technology will be introduced is known. In other words, the technology described in Patent Document 3 cannot be applied unless the data from the facility where the technology will be introduced is known at the time of training.

[0014] In the model development stage, there may be cases where the facility where the model will be introduced has not yet been decided, or where the data for learning about that facility is not available even if the facility where the model will be introduced has been identified. Therefore, even in these cases, it is desirable to realize information recommendations that are effective in the facilities where the model will be introduced.

[0015] One way to achieve this kind of information recommendation is, for example, even if data from the facility where the system is to be introduced is not available when learning the model, if there is data collected at the facility where the system is to be introduced at the time of introduction, that data can be used to evaluate candidate models, thereby making it possible to select the most appropriate model from multiple candidates.

[0016] However, if the performance of any of the multiple candidate models prepared in advance is low when evaluated using data from the facility where the system will be introduced, it will be difficult for the facility to make high-performance information recommendations. To avoid this situation, a technology is needed to build multiple models that can handle a wide variety of unknown domains, so that a candidate model set containing multiple candidate models will include at least one appropriate model, regardless of the facility where the system will be introduced.

[0017] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing method, an information processing device, and a program that can prepare a high-performance model for an unknown facility where the model will be introduced, even if the domain of the facility where the model will be introduced is unknown at the model learning stage. [Means for solving the problem]

[0018] An information processing method according to a first aspect of the present disclosure is an information processing method executed by one or more processors, and includes the one or more processors representing characteristics of a plurality of second facilities different from a first facility from which a dataset used to train a model that predicts user behavior toward an item was collected, and training a plurality of models according to the characteristics of each second facility so as to improve predictive performance at each second facility.

[0019] According to this aspect, multiple models are generated that have improved predictive performance for each of multiple second facilities that have characteristics different from the first facility where the dataset was collected. By expressing a variety of characteristics as the characteristics of the second facilities, it is possible to ensure the diversity of the multiple models. By expressing the characteristics of each of the multiple second facilities so as to cover the range of possible characteristics of unknown facilities that are expected to be used as models, and training the model to improve predictive performance at each second facility, it is possible to build a diverse group of models (a set of multiple models) that includes high-performance models for unknown facilities where the model is used.

[0020] The second facility may be a hypothetical facility that can be assumed as an unknown installation facility. The second facility may be a real facility or a non-real facility. The facility includes the concept of a group consisting of multiple users, such as a company, hospital, store, government agency, or e-commerce site. Each facility may be a different domain from the others.

[0021] The information processing method of the present disclosure can be understood as a machine learning method for generating a model to be applied to a system that recommends information, and can also be understood as a method for producing a model (manufacturing method).

[0022] An information processing method according to a second aspect of the present disclosure may be configured such that, in the information processing method according to the first aspect, one or more processors learn a plurality of models corresponding to each of the plurality of second facilities using data included in the dataset based on characteristics of each of the second facilities. It is possible to learn a plurality of models corresponding to each of the plurality of second facilities from a single dataset.

[0023] An information processing method according to a third aspect of the present disclosure may be the information processing method according to the first or second aspect, further comprising: preparing a plurality of datasets collected from a plurality of first facilities; and having one or more processors represent characteristics of different second facilities for each of the first facilities; and learning a plurality of models using data included in each dataset based on the characteristics of each second facility. By learning models corresponding to different second facilities from each of the plurality of datasets in different domains, a plurality of models corresponding to the plurality of second facilities as a whole can be generated.

[0024] An information processing method according to a fourth aspect of the present disclosure may be configured such that, in the information processing method according to any one of the first to third aspects, one or more processors express the difference in probability distribution of explanatory variables between the first facility and the second facility as an expression of the characteristics of the second facility.

[0025] An information processing method according to a fifth aspect of the present disclosure may be configured such that, in the information processing method according to any one of the first to fourth aspects, one or more processors express the difference in conditional probability between the explanatory variables and the objective variables at the first facility and the second facility as an expression of the characteristics of the second facility.

[0026] An information processing method according to a sixth aspect of the present disclosure may be configured such that, in the information processing method according to any one of the first to fifth aspects, one or more processors perform learning by sampling data to be used for learning from a dataset according to characteristics of the second facility. The one or more processors may simulate data for the second facility from an existing dataset by upsampling and / or downsampling the data to reflect differences in characteristics of the second facility relative to the characteristics of the first facility.

[0027] An information processing method according to a seventh aspect of the present disclosure may be configured such that, in the information processing method according to any one of the first to sixth aspects, one or more processors perform learning by weighting data included in the dataset according to characteristics of the second facility. The one or more processors may weight the data used for learning to reflect differences in characteristics of the second facility relative to the characteristics of the first facility, thereby simulating the effect of learning when unknown data of the second facility is used.

[0028] An information processing method according to an eighth aspect of the present disclosure may be configured to include, in the information processing method according to any one of the first to seventh aspects, selecting features to be used in the model according to characteristics of the second facility by one or more processors. A difference in the characteristics between the first facility and the second facility may be expressed as a difference in the features to be used in the model.

[0029] An information processing method according to a ninth aspect of the present disclosure may be configured such that, in the information processing method according to the eighth aspect, one or more processors perform learning by deleting some of the intersection features expressed by combinations of explanatory variables from the features of the model. It is preferable to perform learning by excluding from the features of the model intersection features that show significant differences between different facilities.

[0030] An information processing method according to a tenth aspect of the present disclosure is an information processing method according to any one of the first to ninth aspects, wherein the model is a predictive model used in a recommendation system that recommends items to users, and the characteristics of the second facility represented by the one or more processors may be characteristics of a virtual facility assumed within a range of facility characteristics that may be considered as a facility where the recommendation system is to be installed.

[0031] An information processing method according to an eleventh aspect of the present disclosure may be configured such that, in an information processing method according to any one of the first to tenth aspects, the dataset includes behavioral histories of multiple users with respect to multiple items in a first facility.

[0032] An information processing method according to a twelfth aspect of the present disclosure may be configured to include, in the information processing method according to any one of the first to eleventh aspects, one or more processors storing in a storage device a set of multiple candidate models including multiple models generated by learning.

[0033] An information processing method according to a thirteenth aspect of the present disclosure may be configured such that, in the information processing method according to any one of the first to twelfth aspects, one or more processors store in a storage device a set of multiple candidate models including a first model trained to improve predictive performance at a first facility using data included in a dataset, and multiple second models that are multiple models trained based on the respective characteristics of multiple second facilities.

[0034] The processor that performs the training of the first model may be a processor separate from one or more processors that perform the training of the second model. The first model may be prepared as an existing model.

[0035] An information processing method according to a fourteenth aspect of the present disclosure may be configured as the information processing method according to the thirteenth aspect, further including one or more processors learning a first model using data included in the dataset.

[0036] An information processing method according to a 15th aspect of the present disclosure may be configured such that, in the information processing method according to any one of the first to 14th aspects, one or more processors evaluate the performance of each of a plurality of candidate models, including a plurality of models, using data collected at a third facility different from the first facility, and extract a model suitable for the third facility from among the plurality of candidate models based on the results of the evaluation.

[0037] An information processing device according to a sixteenth aspect of the present disclosure is an information processing device having one or more processors and one or more memories for storing instructions to be executed by the one or more processors, wherein the one or more processors represent the characteristics of a plurality of second facilities different from a first facility from which a dataset used to train a model that predicts user behavior toward an item was collected, and trains a plurality of models according to the characteristics of each second facility so as to improve predictive performance at each second facility.

[0038] The information processing device according to the sixteenth aspect may have a configuration including the same specific aspect as the information processing method according to any one of the second to fifteenth aspects described above.

[0039] A program according to a seventeenth aspect of the present disclosure enables a computer to perform the following functions: expressing the characteristics of multiple second hypothetical facilities that are different from the first facility from which a dataset used to train a model that predicts user behavior toward an item was collected; and training multiple models based on the characteristics of each second facility and in accordance with the characteristics of each second facility so as to improve predictive performance at each second facility.

[0040] The program according to the seventeenth aspect may have a configuration including the same specific aspect as the information processing method according to any one of the second to fifteenth aspects described above. [Effects of the Invention]

[0041] According to the present disclosure, even if the domain of the facility where the model will be introduced is unknown at the model learning stage, it is possible to generate multiple models that can be used for a variety of facilities, and to prepare high-performance models for unknown facilities where the model will be introduced. [Brief explanation of the drawings]

[0042] [Figure 1] Figure 1 is a conceptual diagram of a typical recommendation system. [Figure 2] Figure 2 is a conceptual diagram showing an example of supervised machine learning, which is widely used to build recommendation systems. [Figure 3] FIG. 3 is an explanatory diagram showing a typical implementation flow of a recommendation system. [Figure 4] FIG. 4 is an explanatory diagram of the introduction flow of the recommendation system when data on the facility where the system is to be introduced cannot be obtained. [Figure 5] FIG. 5 is an explanatory diagram of model learning using domain adaptation. [Figure 6] FIG. 6 is an explanatory diagram of the recommendation system implementation flow, which includes a step of evaluating the performance of the trained model. [Figure 7] FIG. 7 is an explanatory diagram showing examples of learning data and evaluation data used in machine learning. [Figure 8] FIG. 8 is a graph that schematically shows the difference in model performance depending on the data set. [Figure 9] FIG. 9 is an explanatory diagram showing an example of a recommendation system introduction flow when the learning domain and the introduction domain are different. [Figure 10] FIG. 10 is an explanatory diagram showing the problem when the learning domain and the introduction domain are different. [Figure 11] FIG. 11 is an explanatory diagram showing an example of the diversity of candidate models. [Figure 12] FIG. 12 is an explanatory diagram showing the variety of facilities that are expected to be introduced into the recommendation system. [Figure 13] FIG. 13 is a conceptual diagram in the case where a model suitable for prediction at each facility shown in FIG. 12 is prepared. [Figure 14] FIG. 14 is an explanatory diagram showing an example of a learning facility where a data set used for learning is collected. [Figure 15] FIG. 15 is an explanatory diagram showing the relationship between facility characteristics and a model generated by performing learning using data collected from the learning facility shown in FIG. [Figure 16] FIG. 16 is an explanatory diagram showing how data collected from a learning facility is used to learn a model suited to a facility with characteristics different from that of the learning facility. [Figure 17] FIG. 17 is a block diagram schematically illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Figure 18] FIG. 18 is a functional block diagram showing the functional configuration of the information processing device. [Figure 19] FIG. 19 is a chart showing an example of behavior history data. [Figure 20] FIG. 20 is a graph showing an example of the age distribution of users at a learning facility. [Figure 21] FIG. 21 is an example of a directed acyclic graph (DAG) that represents the dependency between variables in a joint probability distribution P(X, Y). [Figure 22] FIG. 22 is a diagram showing a specific example of the probability expression of the conditional probability distribution P(Y|X). [Figure 23] Figure 23 is an explanatory diagram showing the relationship between an equation expressing the conditional probability of a user's behavior toward an item (Y=1) for a combination of the user's behavioral characteristics and the item's characteristics, and a DAG expressing the dependency between variables in the joint probability distribution P(X, Y). [Figure 24] Figure 24 is an explanatory diagram showing the relationship between a user's behavioral characteristics defined by a combination of user attribute 1 and user attribute 2, an item's behavioral characteristics defined by a combination of item attribute 1 and item attribute 2, and a DAG that represents the dependency between variables. [Figure 25]FIG. 25 is an explanatory diagram showing an example of sampling learning data from a data set of a learning facility according to the characteristics of a virtual facility. [Figure 26] FIG. 26 is an explanatory diagram of a case where the weighting of the learning data is changed depending on the characteristics of the virtual facility. [Figure 27] FIG. 27 is a graph showing an example of what types of documents are being viewed by what departments of users in a certain company. [Figure 28] FIG. 28 is an explanatory diagram showing an example in which the inner product of a user characteristic vector and an item characteristic vector is expressed as the sum of inner products of attribute vectors. [Figure 29] FIG. 29 is an explanatory diagram of a case where different vector expressions are used depending on the intersection object. DETAILED DESCRIPTION OF THE INVENTION

[0043] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

[0044] Overview of information recommendation technology First, we will provide an overview of information recommendation technology and provide specific examples of its challenges. Information recommendation technology is a technology for recommending (suggesting) items to users.

[0045] FIG. 1 is a conceptual diagram of a typical recommendation system 10. The recommendation system 10 receives user information and context information as input, and outputs information about items to recommend to the user based on the context. Context refers to various "situations," such as the day of the week, the time of day, or the weather. Items can be various objects, such as books, videos, or restaurants.

[0046] A recommendation system 10 typically recommends multiple items simultaneously. Figure 1 shows an example in which the recommendation system 10 recommends three items IT1, IT2, and IT3. The recommendation is generally considered successful if the user responds positively to the recommended items IT1, IT2, and IT3. A positive response could be, for example, a purchase, a viewing, or a visit. This type of recommendation technology is widely used, for example, on e-commerce sites and gourmet sites that introduce restaurants.

[0047] The recommendation system 10 is constructed using machine learning technology. Figure 2 is a conceptual diagram showing an example of supervised machine learning, which is widely used in constructing the recommendation system 10. In general, positive and negative examples are prepared based on the user's past behavioral history, and combinations of the user and context are input into the prediction model 12, which is then trained to reduce the prediction error. For example, viewed items viewed by the user are considered positive examples, and unviewed items not viewed by the user are considered negative examples. Machine learning is continued until the prediction error converges, and the target prediction performance is achieved.

[0048] The trained prediction model 12 is used to recommend items with a high predicted viewing probability for a combination of a user and a context. For example, when a combination of a user A and a context β is input to the trained prediction model 12, the prediction model 12 infers that there is a high probability that user A will view a document such as item IT3 under the conditions of context β, and recommends an item similar to item IT3 to user A. Note that, depending on the configuration of the recommendation system 10, items are often recommended to users without taking the context into consideration.

[0049] [Examples of data used in developing recommendation systems] A user's behavioral history is roughly equivalent to the "ground truth" in machine learning. Strictly speaking, it can be understood as a task setting to infer the next (unknown) behavior from the past behavioral history, but it is common to learn latent features based on the past behavioral history.

[0050] The user's behavior history may include, for example, a book purchase history, a video viewing history, or a restaurant visit history.

[0051] Furthermore, the main features include user attributes and item attributes. User attributes can include various elements such as gender, age, occupation, family structure, and residential area. Item attributes can include various elements such as book genre, price, video genre, length, restaurant genre, and location.

[0052] [Model construction and operation] Figure 3 is an explanatory diagram showing a typical implementation flow of a recommendation system. This diagram shows a typical flow when implementing a recommendation system in a facility. Implementing a recommendation system involves first constructing a model 14 that performs the desired recommendation task (Step 1), and then introducing and operating the constructed model 14 (Step 2). In the case of a machine learning model, "constructing" a model 14 includes training the model 14 using learning (training) data to create a predictive model (recommendation model) that meets a practical level of recommendation performance. "Operating" a model 14 means, for example, obtaining an output list of recommended items from the trained model 14 in response to an input combination of a user and a context.

[0053] Learning data is required to build the model 14. As shown in Figure 3, the model 14 of a recommendation system is generally trained based on data collected at the facility where it is installed. By training using data collected from the facility where it is installed, the model 14 learns the behavior of users at the facility where it is installed, and is able to accurately predict recommended items for users at the facility where it is installed.

[0054] However, for various reasons, data from the facility where the system is to be implemented may not be available. For example, in the case of a document information recommendation system for a company's in-house system or a hospital's in-house system, the company developing the recommendation model often does not have access to data from the facility where the system is to be implemented. When data from the facility where the system is to be implemented is not available, it is necessary to train the system using data collected at a different facility instead.

[0055] Figure 4 is an explanatory diagram of the introduction flow of the recommendation system when data from the facility where the system is being introduced cannot be obtained. Guidance When used at the receiving facility, there is a problem that the predictive accuracy of Model 14 decreases due to differences in user behavior between facilities.

[0056] The problem of machine learning models not performing well at unknown facilities different from the facilities they were trained on can be broadly understood as a technical challenge of improving robustness against the problem of domain shift, where the source domain in which the model 14 was trained differs from the target domain to which the model 14 is applied. A problem setting related to domain generalization is domain adaptation, which is a learning method that uses data from both the source and target domains. The purpose of using data from a different domain even when data from the target domain exists is to compensate for the small amount of data in the target domain that is insufficient for learning.

[0057] Fig. 5 is an explanatory diagram of the case where model 14 is trained by domain adaptation. Although the amount of data collected at the facility where the model is introduced, which is the target domain, is relatively small compared to the amount of data collected at a different facility, by training using both sets of data, model 14 becomes able to predict with a certain degree of accuracy the behavior of users at the facility where the model is introduced.

[0058] [Domain Description] The difference in "facility" mentioned above is a type of difference in domain. In Non-Patent Document 2 (Ivan Cantador et al., Chapter 27: "Cross-domain Recommender System"), which is a document on research into domain adaptation in information recommendation, differences in domain are classified into the following four types:

[0059] [1] Item attribute level: For example, comedy movies and horror movies are separate domains.

[0060] [2] Item type level: For example, movies and TV series are separate domains.

[0061] [3] Item level: For example, movies and books are separate domains.

[0062] [4] System level: For example, movies in cinemas and movies broadcast on television are separate domains.

[0063] The differences in the "facilities" shown in Figure 5 etc. correspond to the system-level domain [4] among the four classifications mentioned above.

[0064] To formally define a domain, it is defined by the joint probability distribution P(X,Y) of the response variable Y and the explanatory variable X, and when Pd1(X,Y) ≠ Pd2(X,Y), d1 and d2 are different domains.

[0065] The joint probability distribution P(X,Y) can be expressed as the product of the distribution of the explanatory variables P(X) and the conditional probability distribution P(Y|X), or the product of the distribution of the objective variable P(Y) and the conditional probability distribution P(Y|X).

[0066] P(X,Y)=P(Y|X)P(X)=P(X|Y)P(Y) Therefore, a change in one or more of P(X), P(Y), P(Y|X), and P(X|Y) results in a different domain.

[0067] [Typical pattern of domain shift] [Covariate shift] When the distribution of explanatory variables P(X) differs, it is called a covariate shift. For example, when the distribution of user attributes differs between datasets, or more specifically, when the ratio of men to women differs, this corresponds to a covariate shift.

[0068] [Prior probability shift] When the distribution of the objective variable P(Y) differs, it is called a prior probability shift. For example, when the average viewing rate or average purchase rate differs between datasets, this corresponds to a prior probability shift.

[0069] [Concept shift] When the conditional probability distributions P(Y|X) and P(X|Y) differ, it is called a concept shift. For example, the probability that a company's R&D department will read a data analysis document is P(Y|X), but this differs between data sets, which is an example of a concept shift.

[0070] Research on domain adaptation or domain generalization can be divided into two types: one that assumes one of the above patterns as the main factor, and one that considers how to deal with changes in P(X,Y) without considering which pattern is the main factor. In the former case, many studies assume covariate shifts in particular.

[0071] [Why domain shifts affect your business] Prediction / classification models that perform prediction or classification tasks make inferences based on the relationship between explanatory variable X and target variable Y, so naturally, prediction / classification performance declines when P(Y|X) changes. Furthermore, when machine learning a prediction / classification model, the prediction / classification error within the training data is minimized. However, for example, when the frequency with which the explanatory variable X = X_1 is greater than the frequency with which X = X_2 is achieved, i.e., when P(X=X_1)>P(X=X_2), there is more data with X=X_1 than with X=X_2, and so reducing the error for X=X_1 is prioritized over reducing the error for X=X_2. Therefore, prediction / classification performance declines when P(X) changes between facilities.

[0072] Domain shift can be a problem not only for information recommendation but also for models of various tasks. For example, when a model for predicting the risk of employee resignation is trained using data from one company, domain shift can become a problem when it is used by another company.

[0073] In addition, for models that predict antibody production by cells, domain shift can become a problem when a model trained using data on one antibody is used for another antibody.Furthermore, for models that classify Voice of Customer (VOC), such as a model that classifies VOC into "product features," "support responses," and "other," domain shift can become a problem when a classification model trained using data on one product is used for another product.

[0074] [Evaluation of the model before its implementation] Before introducing a trained model 14 into an actual facility, the performance of the model 14 is often evaluated. Performance evaluation is necessary to determine whether or not to introduce the model, and for research and development of models or learning methods.

[0075] FIG. 6 is an explanatory diagram of a recommendation system implementation flow that includes a step of evaluating the performance of the trained model 14. In FIG. 6, a step of evaluating the performance of the model 14 is added as "Step 1.5" between Step 1 (the step of training the model 14) and Step 2 (the step of operating the model 14) described in FIG. 5. The rest of the configuration is the same as in FIG. 5. As shown in FIG. 6, in a typical recommendation system implementation flow, data collected at the facility where the system is to be implemented is often divided into training data and evaluation data. After the predictive performance of the model 14 is confirmed using the evaluation data, operation of the model 14 begins.

[0076] However, when constructing a domain generalization model 14, the training data and the evaluation data must be from different domains. Furthermore, in domain generalization, it is preferable to use data from multiple domains for the training data, and it is even more preferable to have as many domains as possible for training.

[0077] [About generalization] FIG. 7 is an explanatory diagram showing examples of training data and evaluation data used in machine learning. A data set obtained from a joint probability distribution Pd1(X,Y) of a certain domain d1 is divided into training data and evaluation data. The evaluation data of the same domain as the training data is called "first evaluation data" and is denoted as "evaluation data 1" in FIG. 7. In addition, a data set obtained from a joint probability distribution Pd2(X,Y) of a domain d2 different from domain d1 is prepared and used as evaluation data. The evaluation data of a domain different from the training data is called "second evaluation data" and is denoted as "evaluation data 2" in FIG. 7.

[0078] The model 14 is trained using the training data of the domain d1, and the performance of the trained model 14 is evaluated using the first evaluation data of the domain d1 and the second evaluation data of the domain d2.

[0079] Fig. 8 is a graph showing a schematic diagram of differences in model performance due to differences in data sets. If the performance of model 14 in the training data is performance A, the performance of model 14 in the first evaluation data is performance B, and the performance of model 14 in the second evaluation data is performance C, then as shown in Fig. 8, the relationship is usually performance A > performance B > performance C.

[0080] High generalization performance of Model 14 generally refers to high performance B or a small difference between performance A and B. In other words, the aim is to achieve high predictive performance even for untrained data, without overfitting to the training data.

[0081] In the context of domain generalization in this specification, this refers to high performance C, or a small difference between performance B and performance C. In other words, the goal is to achieve consistently high performance even in domains different from the domain used for learning.

[0082] In this embodiment, we assume a situation in which, when learning model 14, it is not possible to use data from the facility where the model is introduced, but it is possible to prepare data (correct answer data) including behavioral history collected at the facility where the model is introduced during model evaluation before introduction (pre-implementation evaluation).

[0083] In such cases, it is possible to generate multiple candidate models by training the model using data collected at a facility other than the facility where the system will be introduced, and then evaluate the performance of each candidate model using data collected at the facility where the system will be introduced before introducing it, and then select the most suitable model from the multiple candidate models based on the evaluation results and apply it to the facility where the system will be introduced. Figure 9 shows an example of this.

[0084] FIG. 9 is an explanatory diagram illustrating an example of a recommendation system implementation flow when the learning domain and the implementation domain are different. As shown in FIG. 9, multiple models may be trained using data collected at a facility other than the implementation facility. Here, an example is shown in which models M1, M2, and M3 are trained using datasets DS1, DS2, and DS3 collected at different facilities. For example, model M1 is trained using dataset DS1, model M2 is trained using dataset DS2, and model M3 is trained using dataset DS3. Note that the datasets used to train each of models M1, M2, and M3 may be a combination of multiple datasets collected at different facilities. For example, model M1 may be trained using a dataset that combines dataset DS1 and dataset DS2.

[0085] After training multiple models M1, M2, and M3 in this way, the performance of each model M1, M2, and M3 is evaluated using data Dtg collected at the facility where the models will be introduced. In Figure 9, the symbols "A," "B," and "C" shown below each model M1, M2, and M3 represent the evaluation results of each model. A rating of A indicates good predictive performance that meets the introduction criteria. A rating of B indicates performance that is inferior to A. A rating of C indicates performance that is even inferior to B and is not suitable for introduction.

[0086] For example, as shown in Figure 9, if the evaluation result of model M1 is "A," the evaluation result of model M2 is "B," and the evaluation result of model M3 is "C," model M1 will be selected as the optimal model for the facility where it will be introduced, and the recommendation system 10 that applies model M1 will be introduced.

[0087] 〔assignment〕 As explained in FIG. 9, if there is even one model with good performance among the multiple models M1 to M3 that have been trained and prepared in advance, it will be possible for the facility where the system is introduced to recommend good information.

[0088] However, there may be cases where none of the pre-trained models M1, M2, and M3 receive an A rating in the pre-implementation evaluation. Figure 10 shows an example of this.

[0089] As shown in Figure 10, if the evaluation results of each model M1, M2, and M3 are all rated B, or if they are all rated B or C, it will be difficult to provide high-performance information recommendations in the facility where the model is introduced, regardless of which model is selected.

[0090] To avoid such a situation, it is important to have sufficient diversity among the multiple candidate models prepared in advance. For example, if the performance of models M1, M2, and M3 differ significantly from each other, it will be more likely that one of the models can be applied to accommodate the differences in the characteristics of the possible (anticipated) facilities where the models will be introduced.

[0091] Figure 11 is an explanatory diagram showing the diversity of candidate models. For example, as shown in Figure 11, assume that there are three patterns of characteristics for facilities that can be introduced. The numbers "Introduction Facility 1," "Introduction Facility 2," and "Introduction Facility 3" in the figure indicate facilities with different patterns of facility characteristics.

[0092] FIG. 11 shows an example of the evaluation results when the performance of three models M1, M2, and M3, which are multiple candidate models prepared in advance, is evaluated using data from each facility for these three patterns.

[0093] When the performance of candidate models M1, M2, and M3 was evaluated using data collected at installation facility 1 shown in the upper part of Figure 11, the evaluation results showed that model M1 received an A rating, model M2 received a B rating, and model M3 received a C rating. When the performance of candidate models M1, M2, and M3 was evaluated using data collected at installation facility 2 shown in the middle part of Figure 11, the evaluation results showed that model M1 received a C rating, model M2 received an A rating, and model M3 received a B rating. When the performance of candidate models M1, M2, and M3 was evaluated using data from installation facility 3 shown in the lower part of Figure 11, the evaluation results showed that model M1 received a B rating, model M2 received a C rating, and model M3 received an A rating.

[0094] In this case, model M1 can be applied to the facility of the first pattern (target facility 1), model M2 to the facility of the second pattern (target facility 2), and model M3 to the facility of the third pattern (target facility 3).

[0095] In this way, it is desirable to prepare a variety of candidate models that will include at least one good model for any deployment. A collection of candidate models is called a candidate model set.

[0096] In this embodiment, as shown in FIG. 11, we aim to construct multiple candidate models so that the candidate model set will include at least one good model (a model with an A rating) for any unknown installation facility.

[0097] FIG. 12 is an explanatory diagram showing the diversity of facilities that are expected to be introduced into the recommendation system 10. In order to ensure the diversity of candidate models, first consider what facility characteristics may be present in unknown facilities that are to be introduced into the system. Each of the horizontal and vertical axes in FIG. 12 represents some kind of facility characteristic. While FIG. 12 shows a vector space with two axes, facility characteristic A and facility characteristic B, actual facility characteristics can be multidimensional. For example, in the case of a hospital, facility characteristics may be the distribution of patient ages (generations) or the proportion of medical histories, such as what diseases are most prevalent among patients.

[0098] The facility characteristics expected for unknown installation facilities are distributed, for example, within the range enclosed by the elliptical closed curve in Figure 12. No matter which facility within this range of possible facility characteristics is selected as the installation facility, it is desirable to have a model suitable for that facility in the candidate model set. In other words, as shown in Figure 13, it is desirable that the multiple models M1, M2, M3, Mk...Mn included in the candidate model set are distributed roughly evenly within the range of possible facility characteristics.

[0099] If there are many facilities from which data to be used for learning can be collected, and if the facility data to be used for learning is sufficiently large and diverse, it is relatively easy to prepare models suitable for possible facilities in the candidate model set.

[0100] However, in reality, as shown in Figure 14, there are only a few facilities (hereinafter referred to as learning facilities) that can collect data used for model training, and the data used for training is often not very diverse. The notations in the figure, such as "Learning Facility 1," "Learning Facility 2," and "Learning Facility 3," each with a number, indicate that they are different facilities. For example, it may be the case that the only data available for training is data obtained from Learning Facilities 1 to 3.

[0101] Learning facilities 1 to 3 shown in FIG. 14 are unevenly distributed within a limited range within the range of possible facility characteristics. In this case, as shown in FIG. 15, candidate models can only be prepared within the limited range of learning facility characteristics. Model M1 in FIG. 15 is a model trained using data from learning facility 1. Similarly, model M2 in FIG. 15 is a model trained using data from learning facility 2, and model M3 is a model trained using data from learning facility 3. Model M4 in FIG. 15 is a model trained using a mixture of data from learning facilities 1 to 3. In other words, with only the data from learning facilities 1 to 3, it is not possible to construct a model that can cover the right half and upper region of the range of possible facility characteristics, and it is not possible to prepare a model that can provide good information recommendations to facilities with facility characteristics in this range.

[0102] As explained in Figures 14 and 15, when there are few training facilities, the diversity of the multiple candidate models that can be prepared is insufficient, and the range of possible facility characteristics cannot be fully covered, making it difficult to prepare models in the candidate model set that are suitable for the unknown facilities where the system will be introduced.

[0103] In response to this problem, in this embodiment, as shown in Figure 16, an information processing method and information processing device are provided that can learn multiple models Ma, Mb, and Mc that correspond to facility characteristics that may be considered as possible facility characteristics for the facility where the system is to be introduced, and prepare them as candidate models.

[0104] Overview of information processing device according to embodiment 17 is a block diagram illustrating an example of a hardware configuration of an information processing device 100 according to an embodiment. The information processing device 100 has a function of representing the characteristics of a virtual destination facility different from the learning facility, and a function of learning a model according to the characteristics of the virtual destination facility so as to improve prediction performance at the virtual destination facility, and represents the characteristics of a plurality of virtual destination facilities and learns a plurality of models corresponding to each of the plurality of virtual destination facilities.

[0105] Based on the data set collected at the learning facility, the information processing device 100 generates a model suitable for the virtual destination facility by upsampling, downsampling, or weighting the learning data, or an appropriate combination of these, to reflect the characteristics of the virtual destination facility.

[0106] Imagining a virtual implementation facility is equivalent to imagining a joint probability distribution Pdh(X,Y) that is different from the joint probability distribution P(X,Y) of the learning facility (learning domain). "Characteristics of a virtual installation facility" are characteristics of a facility that is assumed to be an unknown installation facility. This assumed facility may be an actual facility or a non-existent facility. A "virtual installation facility" is called a "virtual facility." A virtual facility may also be called an "assumed facility."

[0107] The information processing device 100 can be realized using computer hardware and software. The physical form of the information processing device 100 is not particularly limited, and it may be a server computer, a workstation, a personal computer, a tablet terminal, or the like. Here, an example in which the processing functions of the information processing device 100 are realized using one computer will be described, but the processing functions of the information processing device 100 may also be realized by a computer system configured using multiple computers.

[0108] The information processing device 100 includes a processor 102 , a non-transitory tangible computer-readable medium 104 , a communication interface 106 , an input / output interface 108 , and a bus 110 .

[0109] The processor 102 includes a CPU (Central Processing Unit). The processor 102 may also include a GPU (Graphics Processing Unit). The processor 102 is connected to a computer-readable medium 104, a communication interface 106, and an input / output interface 108 via a bus 110. The processor 102 reads various programs, data, etc. stored in the computer-readable medium 104, and executes various processes. The term "program" includes the concept of a program module and includes instructions equivalent to a program.

[0110] The computer-readable medium 104 is, for example, a storage device including a memory 112 as a main storage device and a storage 114 as an auxiliary storage device. The storage 114 is, for example, a hard disk. drive The storage 114 may be configured using a hard disk drive (HDD), a solid state drive (SSD), an optical disk, a magneto-optical disk, or a semiconductor memory, or an appropriate combination of these. The storage 114 stores various programs, data, and the like.

[0111] The memory 112 is used as a working area for the processor 102, and as a storage unit that temporarily stores programs and various data read from the storage 114. When a program stored in the storage 114 is loaded into the memory 112 and the processor 102 executes the instructions of the program, the processor 102 functions as a means for performing various processes defined by the program.

[0112] The memory 112 stores various programs and data executed by the processor 102, such as a facility property acquisition program 130, a virtual property expression program 132, a virtual facility learning program 134, and a learning model 136. The learning model 136 may be included in the virtual facility learning program 134.

[0113] The facility characteristic acquisition program 130 is a program that executes a process for acquiring information indicating the characteristics of a learning facility and / or an unknown facility where the facility is to be introduced. The facility characteristic acquisition program 130 may acquire information indicating the characteristics of the learning facility, for example, by statistically processing data included in a dataset collected at the learning facility. The facility characteristic acquisition program 130 may also accept input of information indicating the characteristics of the facility via a user interface, or may automatically collect public information indicating the characteristics of the facility from the Internet.

[0114] The virtual characteristic expression program 132 is a program that executes processing to express the characteristics of a virtual facility that is different from the learning facility. The virtual characteristic expression program 132 expresses, for example, the difference in probability distribution of explanatory variables between the learning facility and the virtual facility. The virtual characteristic expression program 132 may also express, for example, the difference in conditional probability between explanatory variables and objective variables between the learning facility and the virtual facility.

[0115] The virtual facility learning program 134 is a program that executes a process of learning the learning model 136 so as to improve the prediction performance in the virtual facility according to the characteristics of the virtual facility expressed by the virtual characteristic expression program 132.

[0116] The memory 112 includes a dataset storage unit 140 and a candidate model storage unit 142. The dataset storage unit 140 is a storage area for storing a dataset collected at a learning facility (hereinafter referred to as an original dataset). The candidate model storage unit 142 is a storage area for storing a candidate model, which is a learned (trained) model learned by the virtual facility learning program 134.

[0117] The communication interface 106 performs communication processing with an external device via a wired or wireless connection, and exchanges information with the external device. The information processing device 100 is connected to a communication line (not shown) via the communication interface 106. The communication line may be a local area network, a wide area network, or a combination of these. The communication interface 106 can serve as a data acquisition unit that accepts input of various data such as an original data set.

[0118] The information processing device 100 may include an input device 152 and a display device 154. The input device 152 and the display device 154 are connected to the bus 110 via the input / output interface 108. The input device 152 may be, for example, a keyboard, a mouse, a multi-touch panel, or another pointing device, or a voice input device, or an appropriate combination thereof. The display device 154 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. Note that the input device 152 and the display device 154 may be integrally configured as a touch panel, or the information processing device 100, the input device 152, and the display device 154 may be integrally configured as a touch panel tablet terminal.

[0119] 18 is a functional block diagram showing the functional configuration of the information processing device 100. The information processing device 100 includes a data acquisition unit 220, a data storage unit 222, a facility characteristic acquisition unit 230, a virtual characteristic representation unit 232, and a virtual facility learning unit 234. The data acquisition unit 220 acquires a dataset DS collected at a learning facility. The dataset DS includes behavioral histories of multiple users with respect to multiple items at the learning facility.

[0120] The dataset DS acquired via the data acquisition unit 220 is stored in the data storage unit 222. The dataset storage unit 140 (see FIG. 17) is included in the data storage unit 222. The data storage unit 222 may store a plurality of datasets collected from a plurality of learning facilities.

[0121] The facility characteristic acquisition unit 230 acquires facility characteristic information indicating the characteristics (facility characteristics) of facilities such as learning facilities. The facility characteristic acquisition unit 230 may acquire the facility characteristic information of learning facilities by performing statistical processing or the like using data of a data set stored in the data storage unit 222. The facility characteristic acquisition unit 230 may also acquire facility characteristic information of various facilities from public information available on the Internet or the like.

[0122] The virtual characteristic representation unit 232 represents the characteristics of a virtual facility that is assumed to be the facility where the system will be introduced. The virtual characteristic representation unit 232 may represent the characteristics of multiple virtual facilities that are different from the learning facility. The virtual facility learning unit 234 trains the learning model 136 based on the represented characteristics of the virtual facility so as to improve the prediction performance of the virtual facility. The virtual facility learning unit 234 may generate multiple learning models 136 corresponding to each virtual facility based on the characteristics of each of the multiple virtual facilities.

[0123] The virtual facility learning unit 234 includes a sampling unit 242 that samples learning data from the dataset DS, a learning model 136, a loss calculation unit 244, and an optimizer 246. The sampling unit 242 performs upsampling and / or downsampling according to the characteristics of the virtual facility to match the data distribution expected in the virtual facility.

[0124] The learning data sampled by the sampling unit 242 is input to the learning model 136, and a prediction result according to the input data is output from the learning model 136. The learning model 136 is constructed as a mathematical model that predicts a user's behavior regarding an item.

[0125] The loss calculation unit 244 calculates the loss value between the prediction (inference) result output from the learning model 136 and the correct answer data (teacher data) associated with the input data, based on the two.

[0126] The optimizer 246 determines the amount of update for the parameters of the learning model 136 based on the loss calculation result so that the prediction result output by the learning model 136 approaches the correct data, and performs parameter update processing for the learning model 136. The optimizer 246 updates the parameters based on an algorithm such as gradient descent. The virtual facility learning unit 234 may acquire learning data sample by sample and update the parameters, or may acquire learning data and update the parameters in units of mini-batches that combine multiple learning data.

[0127] In this way, machine learning is performed using the learning data sampled from the dataset, thereby optimizing the parameters of the learning model 136 and generating a learning model 136 with the desired predictive performance. The learned (trained) learning model 136 is stored as a candidate model in the candidate model storage unit 142 (see FIG. 17).

[0128] The virtual facility learning unit 234 may also include a weighting control unit 248 that controls the weight of the learning data during learning. The weighting control unit 248 weights the learning data according to the virtual characteristics so as to match the data distribution expected in the virtual facility.

[0129] The virtual property representation unit 232 generates a plurality of different properties of the virtual facility, thereby obtaining a plurality of candidate models suited to the respective properties.

[0130] For example, the virtual characteristic representation unit 232 represents probability distributions Ph1(X), Ph2(X), and Ph3(X) of explanatory variables, which are multiple distributions different from each other, as a hypothetical distribution indicating the characteristics of the virtual facility, and the virtual facility learning unit 234 learns models Mc1, Mc2, and Mc3 corresponding to each characteristic.

[0131] Furthermore, for example, the virtual characteristic representation unit 232 may represent conditional probabilities Ph1(Y|X), Ph2(Y|X), and Ph3(Y|X), which are multiple different distributions, as virtual distributions indicating the characteristics of the virtual facility, and the virtual facility learning unit 234 may learn models Mc1, Mc2, and Mc3 corresponding to the respective characteristics.

[0132] [Examples of behavioral history] Figure 19 is a chart showing an example of behavioral history data. Here, we consider the case of behavioral history in a company's document viewing system. Figure 17 shows an example of a table of user behavioral history regarding document viewing obtained from a company's document viewing system. Here, "item" refers to a document. The table shown in Figure 17 includes columns for "time," "user ID," "item ID," "user attribute 1," "user attribute 2," "item attribute 1," "item attribute 2," and "viewed or not."

[0133] "Time" is the date and time the item was viewed. "User ID" is an identification code that identifies the user, and a unique ID (identification) is defined for each user. "Item ID" is an identification code that identifies the item, and a unique ID is defined for each item. "User attribute 1" is, for example, the department to which the user belongs. "User attribute 2" is, for example, the age of the user. "Item attribute 1" is, for example, the document type as a classification category for the item. "Item attribute 2" is, for example, the file type of the item. "Viewed" has a value of "1" if the item has been viewed (viewed). Note that because there are a huge number of items that have not been viewed, it is common to only record viewed items (viewed = 1) in the record.

[0134] 19, "viewed or not" is an example of a target variable Y, and "user attribute 1," "user attribute 2," "item attribute 1," and "item attribute 2" are each an example of an explanatory variable X. The number of types of explanatory variables X and their combinations are not limited to the example in FIG. 17. The explanatory variables X may further include a context 1, a context 2, a user attribute 3, and an item attribute 3, which are not shown.

[0135] [Examples of facility characteristics] Here, we will explain the distribution of user ages as a specific example of facility characteristics. Figure 20 shows an example of the age distribution of users at a learning facility. The horizontal axis represents age, and the vertical axis represents frequency. The graph shown in Figure 20 can be understood as equivalent to a histogram or density distribution P(X) of user ages at the target facility.

[0136] In Figure 20, graph Gr0, shown by a solid line, represents the age distribution of users at a learning facility. Assume that this learning facility has a relatively even age distribution, with an average age of, say, 40 years old. In contrast, another facility may have a higher proportion of older users, as shown in graph Gr1 (pattern 1), or a higher proportion of younger users, as shown in graph Gr2 (pattern 2). The average age of users at a virtual facility with an age distribution of pattern 1 could be, for example, 60 years old, while the average age of users at a virtual facility with an age distribution of pattern 2 could be, for example, 25 years old.

[0137] Such differences in the age distribution of users between facilities correspond to a type of domain shift known as a "covariate shift." Possible age distribution patterns for unknown facilities that are potential installation sites can be estimated from publicly available information, such as publicly available corporate information and various statistical data. For example, age distributions by prefecture are publicly available. Companies also disclose the average age of their employees. Hospitals also sometimes disclose the results of surveys such as patient satisfaction surveys, which often include the demographic distribution of respondents. Based on such publicly available or easily accessible information, it is possible to predict the age distribution of potential facilities in advance.

[0138] [Explanation of learning methods] Next, a learning method executed by the information processing device 100 will be described. Here, the case of matrix factorization, which is often used in information recommendation, will be described as an example. For example, if there is data such as the table shown in FIG. 19 as the behavior history of a user at a learning facility, the processor 102 first learns the dependencies between variables based on this data. More specifically, the processor 102 uses a model in which the user and items are each represented by a vector and the sum of their dot products becomes the behavior probability, and updates the parameters of the model to minimize the error in behavior prediction.

[0139] A vector representation of a user can be expressed, for example, by adding up the vector representations of each of the user's attributes. The same is true for the vector representation of an item. A model that has learned the dependencies between variables is equivalent to expressing the joint probability distribution P(X,Y) between the objective variable Y and each explanatory variable X in a given behavioral history dataset.

[0140] Fig. 21 is an example of a directed acyclic graph (DAG) that represents the dependency relationships between variables in a joint probability distribution P(X, Y). Fig. 21 shows an example in which four variables, user attribute 1, user attribute 2, item attribute 1, and item attribute 2, are used as explanatory variables X. The relationship between each of these explanatory variables X and the objective variable Y, which is the user's behavior toward an item, can be represented, for example, by a graph like the one shown in Fig. 21.

[0141] During learning, a vector representation of the joint probability distribution P(X,Y) is obtained based on the dependency relationships between variables, such as the DAG shown in Fig. 21. The graph in Fig. 21 shows that the target variable, a user's behavior with respect to an item, depends on the user's behavioral characteristics and the item's characteristics, and that the user's behavioral characteristics depend on user attribute 1 and user attribute 2, and that the item's characteristics depend on item attribute 1 and item attribute 2.

[0142] As shown in Figure 21, the combination of user attribute 1 and user attribute 2 defines the user's behavioral characteristics. Also, the combination of item attribute 1 and item attribute 2 defines the item's characteristics. The user's behavior toward an item is defined by the combination of the user's behavioral characteristics and the item's characteristics.

[0143] Generally, the relationship P(X, Y) = P(X) × P(Y|X) holds, and when the graph in Figure 21 is applied to this equation, it is expressed as follows.

[0144] P(X) = P(user attribute 1, user attribute 2, item attribute 1, item attribute 2) P(Y|X) = P(user's behavior toward an item | user attribute 1, user attribute 2, item attribute 1, item attribute 2) P(X,Y) = P(user attribute 1, user attribute 2, item attribute 1, item attribute 2) × P(user's behavior toward the item | user attribute 1, user attribute 2, item attribute 1, item attribute 2) Furthermore, the graph shown in FIG. 21 shows that it can be decomposed into elements as follows: P(Y|X) = P(user's behavior toward an item | user's behavioral characteristics, item's characteristics) × P(user's behavioral characteristics | user attribute 1, user attribute 2) × P(item's behavioral characteristics | item attribute 1, item attribute 2)

[0145] [Example of probability expression of conditional probability distribution P(Y|X)] For example, the probability that a user will view an item (Y=1) is expressed as a sigmoid function of the inner product of the user characteristic vector and the item characteristic vector. This type of expression is called matrix factorization. The reason for using the sigmoid function is that the value of the sigmoid function ranges from 0 to 1, and the value of the function directly corresponds to the probability. The model expression is not limited to the sigmoid function, and other functions may also be used.

[0146] A specific example of the probability expression of P(Y|X) is shown in Figure 22. Formula F22A shown in the upper part of Figure 22 is an example of a formula that expresses the user characteristic vector θu and the item characteristic vector φi as five-dimensional vectors using matrix decomposition, and expresses the sigmoid function σ(θu·φi) of their inner product (θu·φi) as the conditional probability P(Y=1|user, item).

[0147] u is an index value that distinguishes users. i is an index value that distinguishes items. Note that the dimension of the vector is not limited to five dimensions, and can be set to an appropriate number of dimensions as a hyperparameter of the model.

[0148] The user characteristic vector θu is expressed as the sum of the user's attribute vectors. For example, as shown in formula F22B in the middle of FIG. 22, the user characteristic vector θu is expressed as the sum of the user attribute 1 vector and the user attribute 2 vector. Furthermore, the item characteristic vector φi is expressed as the sum of the item's attribute vectors. For example, as shown in formula F22C in the bottom of FIG. 22, the item characteristic vector φi is expressed as the sum of the item attribute 1 vector and the item attribute 2 vector.

[0149] Figure 23 is an explanatory diagram showing the relationship between formula F22A, which expresses the conditional probability of a user's behavior toward an item (Y=1) for a combination of the user's behavioral characteristics and the item's characteristics, and a DAG, which expresses the dependency relationship between variables in the joint probability distribution P(X, Y). As shown in Figure 23, formula F22A expresses the conditional probability of the part enclosed by the dashed-line frame FR1 in the DAG shown in Figure 23.

[0150] Figure 24 is an explanatory diagram showing the relationship between a user's behavioral characteristics defined by a combination of user attribute 1 and user attribute 2, an item's characteristics defined by a combination of item attribute 1 and item attribute 2, and a DAG that expresses the dependency relationships between variables. As shown in Figure 24, formula F22B represents the relationship of the part surrounded by a frame FR2 indicated by a dashed line in the DAG shown in Figure 24. Furthermore, formula F22C represents the relationship of the part surrounded by a frame FR3 indicated by a dashed line in the DAG shown in Figure 24.

[0151] The value of each vector shown in FIG. 23 is determined by learning from data (learning data) included in the dataset of user behavior history for a given domain.

[0152] For example, the vector values ​​are updated using, for example, stochastic gradient descent (SGD) so that P(Y=1|user, item) is large for pairs of users and items who viewed the item, and P(Y=1|user, item) is small for pairs of users and items who did not view the item.

[0153] In the case of the joint probability distribution P(X, Y) shown in FIGS. 23 and 24, the parameters to be learned from the data are as follows: User characteristic vector: θu Item characteristic vector: φi User attribute 1 vector: Vk_u^1 User attribute 2 vector: Vk_u^2 Item attribute 1 vector: Vk_i^1 Item attribute 2 vector: Vk_i^2 However, these parameters satisfy the following relationship: θu=Vk_u^1+Vk_u^2 φi=Vk_i^1+Vk_i^2 k is an index value that distinguishes attributes. For example, if user attribute 1 has 10 types of department, user attribute 2 has 6 levels of age, item attribute 1 has 20 types of document type, and item attribute 2 has 5 types of file type, then the number of attribute types is 10 + 6 + 20 + 5 = 41, so k can take on values ​​from 1 to 41. For example, k=1 corresponds to the sales department of user attribute 1, and the index value of user attribute 1 for user u is expressed as k_u^1.

[0154] The values ​​of each vector, user attribute 1 vector Vk_u^1, user attribute 2 vector Vk_u^2, item attribute 1 vector Vk_i^1, and item attribute 2 vector Vk_i^2, are determined by learning from the learning data.

[0155] As a loss function for learning, for example, logloss shown in the following equation (1) is used.

[0156] L=-{Y ui log σ(θu·φi)+(1-Y ui ) log (1-σ(θu·φi))}(1) Y if user u views item i ui = 1, and the larger the predicted probability σ(θu φi), the smaller the loss L. Conversely, if user u does not view item i, Y ui =0, and the smaller σ(θu·φi) is, the smaller the loss L is.

[0157] The parameters of the vector representation are learned so that the above loss L is reduced. For example, when optimizing using stochastic gradient descent, one record is randomly selected from all training data (if it does not depend on the context, one ui pair is selected from all ui pairs), the partial derivative (gradient) of each parameter of the loss function is calculated for the selected record, and the parameters are changed in the direction that reduces the loss L in proportion to the magnitude of the gradient.

[0158] For example, the parameters of the user attribute 1 vector (Vk_u^1) are updated according to the following equation (2).

[0159]

number

[0160] In equation (2), α is the learning rate.

[0161] Generally, among a large number of items, there are far more items with Y=0 than items with Y=1, so when saving behavioral history data as shown in Figure 19 as a table, only Y=1 is retained, and pairs of user u and item i that are not included in the behavioral history data are learned as Y=0. In other words, if only positive example data is saved, negative examples can be easily generated as those not included in the positive example data.

[0162] [Model representation] The means of expressing the joint probability distribution of explanatory variable X and target variable Y are not limited to matrix factorization. For example, logistic regression or Naive Bayes may be applied instead of matrix factorization. Any predictive model can also be used to express the joint probability distribution by calibrating it so that the output score is close to the probability P(Y|X). For example, SVM (Support Vector Machine), GDBT (Gradient Boosting Decision Tree), and neural network models of any architecture can also be used.

[0163] [About sampling of training data] In this embodiment, the data of the learning facility is upsampled or downsampled to match the data distribution of the virtual facility. Fig. 25 is an explanatory diagram showing an example of sampling learning data from the data set of the learning facility according to the characteristics of the virtual facility.

[0164] FIG. 25 shows an example of sampling learning data used to train a model corresponding to a virtual facility having an age distribution of Pattern 2 shown in graph Gr2 of FIG. 20. Processor 102 upsamples or downsamples the data of the learning facility to match the age distribution of Pattern 2 (FIG. 25). In the case of the learning facility data shown in FIG. 25, since there is less data on younger users, the data of younger users is upsampled as shown in graph Gr2, and since there is more data on older users, the data is downsampled. By performing learning using the sampled data in this way, a model suitable for a virtual facility having an age distribution of Pattern 2 can be prepared.

[0165] The distribution shown in graph Gr2 in Fig. 25 is an example of a probability distribution that expresses the difference from the probability distribution of the explanatory variables in the learning facility. Although not shown in the figure, when training a model corresponding to a virtual facility with the age distribution shown in graph Gr1 in Fig. 20, sampling that reflects the distribution of graph Gr1 can be performed.

[0166] [Sampling explanation] As mentioned above, SGD selects one record from the training dataset at each learning step. This operation is repeated until the prediction error of the training model 136 converges.

[0167] Without upsampling or downsampling, all records are selected (almost) the same number of times and used for training. For example, records 1 to 4 in the dataset table are each used four times for training. Note that since sampling is done stochastically, there may be variation in the number of times used for training within the range of stochastic fluctuations.

[0168] In contrast, for example, if record 1 has a low age and is upsampled, record 4 has a high age and is downsampled, and records 2 and 3 are neither upsampled nor downsampled, then the number of times each record is used for training is 8 times for record 1, 4 times for records 2 and 3, and 2 times for record 4.

[0169] [Example of weighting of training data] In Figure 25, an example is explained in which data sampling is performed according to the characteristics of the virtual facility. However, instead of or in combination with data sampling, it is also possible to generate a model suitable for the virtual facility by changing the weight of the learning data during learning.

[0170] 26 is an explanatory diagram of a case where the weighting of the learning data is changed according to the characteristics of the virtual facility. For example, the processor 102 may weight the learning data using the density ratio between the density of the explanatory variable in the learning facility, P_learning facility(X), and the density of the explanatory variable in the virtual facility, P_virtual facility(X), for learning.

[0171] The density ratio w of explanatory variables of learning facilities and virtual facilities is expressed by the following equation: 。

[0172] w = P_Virtual Facility(X) / P_Learning Facility(X) This density ratio w is used as a weight, and if w>1, the weight during learning is increased, and if w<1, the weight during learning is decreased.

[0173] This is a technique called importance sampling, and is a representative example of a method for dealing with covariate shift. Other methods for dealing with covariate shift may also be applied.

[0174] When weighting the training data, the weight w for each training record is used for the loss function explained in the formula. ui That is, instead of formula (1), the loss function shown in the following formula (3) can be applied.

[0175] L=-w ui {Y ui log σ(θu·φi)+(1-Y ui ) log (1-σ(θu·φi))} (3) If the training data is not weighted, the weight w in Equation (3) ui is always "1".

[0176] In importance sampling, the weights w ui is defined as follows:

[0177] w ui =P_Virtual Facility(X) / P_Learning Facility(X) X is a vector consisting of a combination of explanatory variables, for example, X = (user attribute 1, user attribute 2, item attribute 1, item attribute 2).

[0178] For example, if the learning facility has P(X) = 0.1 for a young user and P(X) = 0.2 for a virtual facility, the learning weight for the data (records) of the young user u is w ui = 0.2 / 0.1 = 2. For example, if an elderly user has P(X) = 0.15 in the learning facility and P(X) = 0.05 in the virtual facility, the weight of the elderly user u's data during learning is w ui =0.05 / 0.15=0.33.

[0179] [When the conditional probabilities P(Y|X) are different] So far, we have used the age distribution of users as an example to explain how to respond when the probability distribution P(X) of the explanatory variable as a facility characteristic differs, but similarly, multiple models can be prepared when the conditional probability P(Y|X) differs between the learning facility and the virtual facility.

[0180] Figure 27 is a graph that shows an example of what types of documents are viewed by users in what departments at a certain company. Figure 27 shows the probability that users in the R&D department and the sales department will view data analysis documents and product catalogs, respectively.

[0181] The conditional probability P(Y|X) of which department's users are viewing which documents may differ depending on the facility. This corresponds to a concept shift.

[0182] Figure 27 expresses the assumption that the types of documents that users in the research and development department frequently view will vary greatly depending on the facility (company), but the types of documents that users in the sales department frequently view will not vary much depending on the facility (company).

[0183] In this case, "R&D department x document type," which is part of the cross feature "department x document type," is removed from the features of the prediction model for training. In this way, removing features that are highly domain-dependent from the features of the prediction model can result in a model that is robust to domain shifts.

[0184] [Decomposition into attribute vectors] The inner product of the user characteristic vector θu and the item characteristic vector φi described in Fig. 22 is decomposed into the sum of inner products of attribute vectors, as shown in Fig. 28. That is, it can be expressed as in the following equation (4). θu·φi=(Vk_u^1·Vk_i^1)+(Vk_u^2·Vk_i^1)+(Vk_u^1·Vk_i^2)+(Vk_u^2·Vk_i^2) (4)

[0185] [On intersection features and vector representation] In a recommendation model that represents attributes as vectors, the operation of deleting an intersection feature is equivalent to constraining the inner product of the corresponding attribute vectors to be 0. For example, to delete the intersection feature between R&D department, which is one of user attribute 1 (department), and data analysis material, which is one of item attribute 1 (document type), it is sufficient to constrain the inner product of user attribute 1 vector (R&D department) for item attribute 1 and item attribute 1 vector (data analysis material) for user attribute 1 to be 0.

[0186] This can be achieved, for example, by adding to the loss function during training a loss proportional to the absolute value of the dot product between vectors whose dot product you want to make 0. For example, to constrain the dot product between user attribute 1 and item attribute 1 vector (data analysis material) to be 0, use the following loss function:

[0187] L= -[Y ui log σ(θu·φi)+(1-Y ui ) log (1-σ(θu·φi))] + λ Σ |Vk_u^1·Vk_i^1| (5) The final term in equation (5) is a sum that includes only the combination of cross features to be removed. The coefficient λ is a hyperparameter that controls the magnitude of the loss of this added final term.

[0188] This makes it possible to improve the prediction accuracy for the user's items while correcting so as not to use the cross feature defined in the final term.

[0189] When we want to remove cross features, that is, when introducing a loss function such as equation (5) above to remove cross features, it is more desirable to use a different vector representation depending on the cross target. For example, methods such as field-aware factorization machines (FFM) have such vector representations.

[0190] [Example of using different vector representations depending on the intersection object] FIG. 29 is an explanatory diagram of a case where different vector expressions are used depending on the intersection object. When attribute vectors having different vector expressions depending on the intersection object are used, the dot product of the user characteristic vector θu and the item characteristic vector φi is decomposed into the sum of dot products of attribute vectors with vector expressions according to the intersection object, as shown in FIG. 29. That is, in the vector expression of user attribute 1, the user attribute 1 vector for item attribute 1 is a vector that is different from the user attribute 1 vector for item attribute 2. Similarly, in the vector expression of item attribute 1, the item attribute 1 vector for user attribute 1 is a vector that is different from the item attribute 1 vector for user attribute 2. The same applies to the vector expression of user attribute 2 and the vector expression of item attribute 2.

[0191] By adopting such a vector representation, it is possible to effectively learn only the intersection feature to be deleted, while separating it from the others.

[0192] [Specific application examples] Here we will explain an example of an in-hospital information recommendation system for hospitals. The datasets that can be used for learning include behavioral history (browsing history) data from Hospital 1, the learning facility, as well as user attributes (affiliated medical department) and item attributes (test type, patient age). We would like to introduce the in-hospital information recommendation system to many hospitals in the future, but the facilities (hospitals) where we will introduce it have not yet been decided, and naturally, data for these undecided facilities has not yet been prepared.

[0193] Assume that the age distribution of patients at Hospital 1, a learning facility, is as follows: 10% in their 20s, 10% in their 30s, 20% in their 40s, 20% in their 50s, 20% in their 60s, 10% in their 70s, and 10% in their 80s. The age distribution of patients is an example of a facility's characteristics.

[0194] The age distribution of patients at such a learning facility can be determined, for example, by statistically processing the data contained in the dataset.

[0195] Since it is expected that the age distribution of patients will differ depending on the hospital, processor 102 represents the age distribution of patients in a hospital other than Hospital 1 based on publicly available information. For example, processor 102 generates the age distribution of patients in a hospital (virtual facility A) that has a higher proportion of elderly patients than Hospital 1 as follows: 5% in their 20s, 5% in their 30s, 10% in their 40s, 10% in their 50s, 25% in their 60s, 30% in their 70s, and 15% in their 80s.

[0196] Similarly, processor 102 generates the age distribution of patients in a hospital (virtual facility B) with a large number of young people, for example, 20% in their 20s, 30% in their 30s, 20% in their 40s, 10% in their 50s, 10% in their 60s, 5% in their 70s, and 5% in their 80s.

[0197] Taking the ratio of the distributions of the learning facility (Hospital 1) and virtual facility A, the ratios are 0.5 for those in their 20s, 0.5 for those in their 30s, 0.5 for those in their 40s, 0.5 for those in their 50s, 1.25 for those in their 60s, 3.0 for those in their 70s, and 1.5 for those in their 80s.

[0198] When processor 102 trains a model using data from the training facility, it weights the data on the age of each patient with the weight of the distribution ratio value described above and trains the model. This allows training of a model (model A) that aims to improve performance in virtual facility A.

[0199] Similarly, processor 102 calculates the distribution ratio for virtual facility B relative to the learning facility, and weights the data with the weight of this distribution ratio value to learn a model. This makes it possible to learn a model (model B) that aims to improve performance in virtual facility B.

[0200] Also, a model (model O) trained without weighting using the data from the original learning facility is prepared. This model O may be generated by the processor 102 performing training using the data set from the learning facility, or may be generated by training using an information processing device (not shown) other than the information processing device 100. In this way, model O, model A, and model B are prepared as candidate models.

[0201] Even in unknown facilities where the system will be introduced in the future, it is assumed that the patient age distribution will be close to either the learning facility, virtual facility A, or virtual facility B. Therefore, it is expected that Model O, Model A, or Model B will have high performance at the facility where the system is introduced.

[0202] Thereafter, when a facility where the system is to be introduced is specifically identified and data on the specific facility is available prior to the introduction of the system to the actual specific facility, processor 102 uses the data on the specific facility to evaluate the performance of each of Model O, Model A, and Model B, and extracts a model suitable for the specific facility from among the multiple candidate models based on the evaluation results. For example, if Model B has the best evaluation result among the three candidate models, Model B is selected as the optimal model. The specific facility in this case is an example of the "third facility" in this disclosure. Processor 102 may extract one optimal model from among the multiple candidate models, or may extract two or more models with acceptable equivalent performance.

[0203] Hospital 1, which is a learning facility, is an example of a "first facility" in the present disclosure. Model O is an example of a "first model" in the present disclosure. Virtual facility A and virtual facility B are each an example of a "second facility" in the present disclosure. The age distribution in each of virtual facility A and virtual facility B is an example of a "characteristic of the second facility" in the present disclosure. Model A and Model B are each an example of a "second model" in the present disclosure.

[0204] In the above example, multiple candidate models were constructed by learning multiple models A and B corresponding to multiple virtual facilities A and virtual facilities B, respectively, from the dataset of Hospital 1. However, if multiple datasets collected at multiple learning facilities are given, as shown in Figure 16, multiple candidate models can be constructed overall by learning models corresponding to one or more virtual facilities from the dataset of each learning facility.

[0205] [Regarding programs that operate computers] A program that causes a computer to realize some or all of the processing functions of the information processing device 100 can be recorded on a computer-readable medium such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible non-transitory information storage medium, and the program can be provided through this information storage medium.

[0206] In addition, instead of providing the program by storing it on such a tangible, non-transitory computer-readable medium, it is also possible to provide the program signal as a download service using a telecommunications line such as the Internet.

[0207] Furthermore, some or all of the processing functions of the information processing device 100 may be realized by cloud computing, and may also be provided as SaaS (Software as a Service).

[0208] [Hardware configuration of each processing unit] The hardware structure of the processing units in the information processing device 100 that execute various processes, such as the data acquisition unit 220, facility characteristic acquisition unit 230, virtual characteristic representation unit 232, virtual facility learning unit 234, sampling unit 242, weighting control unit 248, loss calculation unit 244, and optimizer 246, is, for example, various processors as shown below.

[0209] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units, GPUs, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically to execute specific processes.

[0210] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Alternatively, multiple processing units may be configured with a single processor. A first example of multiple processing units configured with a single processor is a configuration in which one or more CPUs and software are combined to form a single processor, as typified by client or server computers, and this processor functions as multiple processing units. A second example is a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0211] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0212] Advantages of the embodiment According to this embodiment, even if there are restrictions on the domain of the dataset that can be used for learning in the model learning stage, it is possible to build multiple models with improved predictive performance in each of multiple virtual facilities that are different from the learning facility. sex By training multiple models corresponding to the above, a candidate model set including multiple models that can be used in a variety of domains can be constructed.

[0213] According to this embodiment, when the domain (learning domain) of a facility or the like where data used for model learning is collected is different from the domain (destination domain) of a facility or the like where the model is introduced, it is possible to provide a recommended item list that is robust to domain shifts.

[0214] [Other application examples] In the above-described embodiment, user behavior regarding document viewing was used as an example, but the scope of application of the present disclosure is not limited to document viewing, and the technology of the present disclosure can be applied to predicting user behavior regarding various items, regardless of the purpose, such as viewing medical images, purchasing products, or watching content such as videos.

[0215] 〔others〕 The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technical idea of ​​the present disclosure. [Explanation of symbols]

[0216] 10 Recommendation Systems 12 Predictive Models 14 models 100 Information processing device 102 processors 104 Computer-readable medium 106 Communication Interface 108 Input / Output Interface 110 Bus 112 memory 114 Storage 130 Facility Characteristics Acquisition Program 132 Virtual Property Expression Program 134 Virtual Facility Learning Program 136 Learning Model 140 Dataset Storage Unit 142 Candidate model storage unit 152 Input Device 154 Display device 220 Data Acquisition Unit 222 Data Storage Unit 230 Facility Characteristics Acquisition Department 232 Virtual characteristic representation part 234 Virtual Facility Learning Department 242 Sampling section 244 Loss calculation unit 246 Optimizer 248 Weighting Control Unit DS dataset DS1 dataset DS2 dataset DS3 dataset Dtg Data F22A type F22B formula F22C formula FR1 frame FR2 frame FR3 frame Gr0 graph Gr1 graph Gr2 graph IT1 Item IT2 Items IT3 items M1 model M2 model M3 model M4 model Mk model Mn model Ma model Mb model Mc Model

Claims

1. 1. An information processing method executed by one or more processors, comprising: the one or more processors: expressing characteristics of a plurality of second facilities different from the first facility that collected the dataset used to train a model that predicts user behavior with respect to items; training a plurality of models according to characteristics of each of the second facilities so that prediction performance is improved at each of the second facilities; An information processing method including:

2. the one or more processors: training the plurality of models corresponding to each of the plurality of second facilities using data included in the dataset based on characteristics of each of the second facilities; The information processing method according to claim 1 .

3. a plurality of said data sets collected from a plurality of said first facilities respectively are provided; the one or more processors: expressing different characteristics of the second facility for each of the first facilities; training the plurality of models using data included in each of the datasets based on characteristics of each of the second facilities; The information processing method according to claim 1 .

4. the one or more processors express a difference in probability distribution of explanatory variables between the first facility and the second facility as a representation of the characteristic of the second facility. The information processing method according to claim 1 .

5. the one or more processors express a difference in conditional probability between explanatory variables and objective variables at the first facility and the second facility as a representation of the characteristic of the second facility; The information processing method according to claim 1 .

6. the one or more processors: performing the learning by sampling data to be used for learning from the dataset according to characteristics of the second facility; The information processing method according to claim 1 .

7. the one or more processors: performing the learning by weighting the data included in the dataset according to characteristics of the second facility; The information processing method according to claim 1 .

8. the one or more processors: selecting a feature to be used in the model according to a characteristic of the second facility; The information processing method according to claim 1 .

9. the one or more processors: and performing the learning by deleting a part of the intersection features expressed by a combination of explanatory variables from the features of the model. The information processing method according to claim 8.

10. the model is a prediction model used in a recommendation system that recommends items to a user, the characteristics of the second facility represented by the one or more processors are characteristics of a virtual facility assumed within a range of facility characteristics that may be assumed as a facility where the recommendation system is to be installed; The information processing method according to claim 1 .

11. The dataset includes behavioral histories of a plurality of users with respect to a plurality of items at the first facility. The information processing method according to claim 1 .

12. the one or more processors: storing a set of multiple candidate models including the multiple models generated by performing the learning in a storage device; The information processing method according to claim 1 .

13. the one or more processors: storing in a storage device a set of multiple candidate models including a first model trained to improve predictive performance at the first facility using data included in the dataset, and multiple second models, the multiple models trained based on characteristics of each of the multiple second facilities; The information processing method according to claim 1 .

14. the one or more processors: training the first model using data included in the dataset; The information processing method according to claim 13.

15. the one or more processors: evaluating the performance of each of a plurality of candidate models including the plurality of models using data collected at a third facility different from the first facility, and extracting a model suitable for the third facility from among the plurality of candidate models based on a result of the evaluation. The information processing method according to claim 1 .

16. one or more processors; and one or more memories in which instructions to be executed by the one or more processors are stored, The one or more processors: expressing characteristics of a plurality of second facilities different from the first facility that collected the dataset used to train a model that predicts user behavior with respect to an item; training a plurality of models according to characteristics of each of the second facilities so that prediction performance is improved at each of the second facilities; Information processing device.

17. On the computer, A function of expressing the characteristics of a plurality of second facilities different from the facility that collected the dataset used to train a model that predicts user behavior regarding items; a function of learning a plurality of models so as to improve prediction performance in each of the second facilities according to characteristics of each of the second facilities; A program to make this happen.

Images