Information processing device, information processing method, information processing system, and storage medium
The information processing device employs multiple predictors to learn relationships between related information and observed values, improving the selection process and enhancing the accumulation of gains by addressing discrepancies in existing bandit algorithms.
Patent Information
- Application Number
- JP2022556750
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-10-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-10-14
AI Technical Summary
Existing bandit algorithms, such as those described in Non-Patent Document 1, face challenges in accurately selecting options with unknown probability distributions due to discrepancies between the original and learned relationships, leading to suboptimal accumulation of observed values.
An information processing device that uses multiple predictors to independently learn the relationship between related information and observed values, allowing for more accurate selection of options by utilizing learning data to determine the optimal choice.
This approach enhances the accumulation of observed values by leveraging multiple predictors to adaptively learn and refine the selection process, reducing the likelihood of selecting non-optimal options and increasing overall gain.
Smart Images

Figure 0007586188000010 
Figure 0007586188000011 
Figure 0007586188000012
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for selecting one of a plurality of options whose probability distribution of the payoff is unknown. [Background technology]
[0002] Bandit algorithms are known as a technique for selecting one of multiple options with unknown probability distribution of payoffs. Bandit algorithms repeatedly perform trials to select one of multiple options and observe the payoff, and in each trial, determine the option to be selected by referring to the observed payoffs obtained in the previous trial. Bandit algorithms maximize the cumulative observed values obtained up to a certain number of trials and identify the option that maximizes the expected payoff.
[0003] For example, Non-Patent Document 1 describes a contextual bandit algorithm that can be applied when there is a relationship between the observed value of the payoff and the information (context) associated with the option. In each trial, the algorithm described in Non-Patent Document 1 learns the relationship between the observed value and the context, using the observed value and the context obtained in the previous trial as learning data. In addition, the algorithm determines the option to be selected by predicting the payoff of each option using the learned relationship. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Y. Abbasi-Yadkori et. al. "Improved Algorithms for Linear Stochastic Bandits", In Advances in Neural Information Processing Systems 24, pages 2312-2320, 2011. Summary of the Invention [Problem to be solved by the invention]
[0005] The algorithm described in Non-Patent Document 1 has room for improvement in terms of increasing the accumulation of observed values. The reason is that in each trial, if there is a deviation between the original relationship between the observed values and the context and the learned relationship, an option different from the optimal option may be selected.
[0006] One aspect of the present invention has been made in consideration of the above problems. That is, an object of one aspect of the present invention is to provide a technique for further increasing the accumulation of observed values when selecting one of multiple options whose probability distributions are unknown. [Means for solving the problem]
[0007] An information processing device according to one aspect of the present invention includes an acquisition means for acquiring related information associated with each of a plurality of options, a decision means for deciding which of the plurality of options to select, and a storage means for storing in a storage device, as learning data, an observed value of a gain obtained by selecting the option decided by the decision means and related information of the option. The decision means decides which of the plurality of options to select using one of a plurality of predictors that learn a relationship between the related information and the observed value by referring to the learning data, each of which learns the relationship independently of each other.
[0008] An information processing method according to one aspect of the present invention includes an information processing device acquiring related information associated with each of a plurality of options, determining which of the plurality of options to select, and storing an observed value of a gain obtained by selecting the determined option and related information of the option as learning data in a storage device. The information processing method also includes the information processing device using any of a plurality of predictors that learn a relationship between the related information and the observed value by referring to the learning data, each predictor learning the relationship independently of each other, in order to determine which of the plurality of options to select.
[0009] A storage medium according to one aspect of the present invention is a storage medium storing a program that causes a computer to function as an information processing device. The program causes the computer to function as an acquisition means for acquiring associated information associated with each of a plurality of options, a decision means for deciding which of the plurality of options to select, and a storage means for storing, in a storage device, as learning data, an observed value of a gain obtained by selecting the option decided by the decision means and associated information of the option. The decision means decides which of the plurality of options to select by using one of a plurality of predictors that learn a relationship between the associated information and the observed value by referring to the learning data, each of which learns the relationship independently of each other.
[0010] An information processing system according to an aspect of the present invention is an information processing system including an information processing device and a server. The information processing device includes an acquisition means for receiving related information associated with each of a plurality of options from the server, a decision means for deciding which of the plurality of options to select and transmitting information indicating the decided option to the server, and a storage means for receiving from the server an observed value of a gain obtained by selecting the option decided by the decision means, and storing the received observed value and related information of the option in a storage device as learning data. The decision means determines which of the plurality of options to select using one of a plurality of predictors that learn a relationship between the related information and the observed value by referring to the learning data, each of which learns the relationship independently of each other. The server includes an acquisition means for acquiring the related information and transmitting it to the information processing device, a selection means for selecting an option indicated by the information received from the information processing device, and an observation means for observing a gain obtained by the selection by the selection means and transmitting the observed observed value to the information processing device.
[0011] An information processing method according to an aspect of the present invention includes an information processing device receiving related information associated with each of a plurality of options from a server, determining which of the plurality of options to select and transmitting information indicating the determined option to the server, receiving from the server an observed value of a gain obtained by selecting the determined option, and storing the received observed value and related information of the option in a storage device as learning data. The information processing method also includes the information processing device determining which of the plurality of options to select by using one of a plurality of predictors that learn a relationship between the related information and the observed value by referring to the learning data, each predictor learning the relationship independently of each other. The information processing method also includes the server acquiring the related information and transmitting it to the information processing device, selecting an option indicated by the information received from the information processing device, observing a gain obtained by selecting the option, and transmitting the observed observed value to the information processing device. Effect of the Invention
[0012] According to one aspect of the present invention, the accumulation of observed values can be further increased. [Brief description of the drawings]
[0013] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Diagram 2] 1 is a flowchart showing a flow of an information processing method executed by an information processing device according to a first exemplary embodiment of the present invention. [Diagram 3] FIG. 11 is a block diagram showing a configuration of an information processing system according to a second exemplary embodiment of the present invention. [Figure 4] 11 is a flow chart showing the flow of an information processing method executed by an information processing system according to an exemplary embodiment 2 of the present invention. [Diagram 5] FIG. 11 is a block diagram showing a configuration of an information processing device according to an exemplary embodiment 3 of the present invention. [Figure 6] FIG. 11 is a diagram for explaining an information processing method executed by an information processing device according to an exemplary embodiment 3 of the present invention. [Figure 7] 11 is a flowchart showing a schematic flow of an information processing method executed by an information processing device according to an exemplary embodiment 3 of the present invention. FIG. [Figure 8] FIG. 11 is a flow chart showing a detailed flow of a part of an information processing method executed by an information processing device according to an exemplary embodiment 3 of the present invention. [Figure 9] FIG. 11 is a block diagram showing a configuration of an information processing system according to an exemplary embodiment 4 of the present invention. [Figure 10] FIG. 11 is a flow chart showing a schematic flow of an information processing method executed by an information processing system according to an exemplary embodiment 4 of the present invention. [Figure 11] FIG. 11 is a flow chart showing a detailed flow of a part of an information processing method executed by an information processing system according to an exemplary embodiment 4 of the present invention. [Figure 12] 11 is a graph showing an example using an information processing device according to an exemplary embodiment 4 of the present invention. [Figure 13] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device in each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Example 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.
[0015] <Outline of information processing device> The information processing device 1 is a device that repeats the following trial: acquire related information linked to each of a plurality of options, determine which of the plurality of options to select, and accumulate the observation values and related information obtained by selecting the determined option as learning data.
[0016] <Configuration of information processing device> The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1.
[0017] As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, a determination unit 12, and a storage unit 13. The acquisition unit 11 is a component that realizes an acquisition means in this exemplary embodiment. The determination unit 12 is a component that realizes a determination means in this exemplary embodiment. The storage unit 13 is a component that realizes a storage means in this exemplary embodiment.
[0018] The acquisition unit 11 acquires related information linked to each of the multiple options. Hereinafter, the related information linked to an option is also simply referred to as "related information of the option." For example, the acquisition unit 11 may acquire the related information of each option via an input device (not shown). Also, for example, the acquisition unit 11 may acquire the related information of each option from another device (not shown) communicatively connected to the information processing device 1.
[0019] The decision unit 12 decides which of the multiple options to select. Specifically, the decision unit 12 uses one of the multiple predictors to decide which of the multiple options to select. Here, each predictor refers to the learning data stored in the storage device to learn the relationship between the related information and the observed value of the gain. The gain is a quantitative expression of the efficacy obtained by selecting an option. The observed value is the gain observed by actually selecting an option. In addition, each predictor learns the relationship independently of each other. Here, learning independently of each other may mean, for example, that the sets of learning data used by each predictor for learning are different from each other.
[0020] The accumulation unit 13 acquires an observation value of the gain obtained by selecting the option determined by the determination unit 12. The accumulation unit 13 also accumulates the acquired observation value and related information of the option as learning data in a storage device (not shown). The storage device may be included in the information processing device 1, or may be communicably connected to the information processing device 1.
[0021] <Information processing method flow> The flow of the information processing method S1 executed by the information processing device 1 configured as above will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. The information processing device 1 executes the information processing method S1 in each trial of determining which one to select from a plurality of options.
[0022] As shown in FIG. 2, the information processing method S1 includes steps S11 to S13.
[0023] (Step S11) In step S11, the acquisition unit 11 acquires related information for each of a plurality of options.
[0024] (Step S12) In step S12, the determination unit 12 uses one of a plurality of predictors to determine which of a plurality of options to select.
[0025] (Step S13) In step S13, the accumulation unit 13 acquires an observation value of the gain obtained by selecting the option determined by the determination unit 12. The accumulation unit 13 also accumulates the acquired observation value and related information of the option in the storage device as learning data.
[0026] <Advantages of this exemplary embodiment> As described above, in this exemplary embodiment, a configuration is adopted in which one of a plurality of predictors is used to determine which of a plurality of options is to be selected. Each predictor refers to the learning data and learns the relationship between the related information and the observed value independently of each other. Therefore, in order to determine which of a plurality of options is to be selected, it is possible to use a predictor that has more appropriately learned the relationship between the observed value and the related information. As a result, this exemplary embodiment can determine an option that further increases the cumulative gain.
[0027] Exemplary embodiment 2 A second exemplary embodiment of the present invention will now be described in detail with reference to the drawings.
[0028] <Outline of information processing system> An information processing system 10A according to this exemplary embodiment is a system in which an information processing device 1A, which is a modification of the exemplary embodiment 1, functions in cooperation with a server 3A.
[0029] <Configuration of information processing system> The configuration of the information processing system 10A will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing system 10A. As shown in Fig. 3, the information processing system 10A includes an information processing device 1A and a server 3A. The information processing device 1A and the server 3A are communicatively connected via a network N1. The network N1 is a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks.
[0030] (Configuration of information processing device 1A) As shown in FIG. 3, the information processing device 1A includes an acquisition unit 11A, a determination unit 12A, and a storage unit 13A.
[0031] The acquiring unit 11A has almost the same configuration as the acquiring unit 11 in the exemplary embodiment 1, but differs in that it receives related information from the server 3A.
[0032] The determination unit 12A has a configuration similar to that of the determination unit 12 in the first exemplary embodiment, but transmits information indicating the determined option to the server 3A. send In other respects, the determination unit 12 is configured similarly to the determination unit 12.
[0033] The storage unit 13A has a configuration similar to that of the storage unit 13 in the first exemplary embodiment, but differs in that it receives observed values from the server 3A.
[0034] (Server 3A configuration) As shown in FIG. 3, the server 3A includes an acquisition unit 31A, a selection unit 32A, and an observation unit 33A.
[0035] The acquisition unit 31A acquires related information for each of the multiple options and transmits the acquired related information to the information processing device 1A. For example, the acquisition unit 31A may acquire related information from each of multiple terminals (not shown) communicatively connected to the server 3A.
[0036] The selection unit 32A selects an option indicated by the information received from the information processing device 1 A. For example, the selection unit 32A may transmit information indicating that the option has been selected to the terminal from which the related information of the option has been acquired.
[0037] The observation unit 33A observes the gain obtained by the selection by the selection unit 32A, and transmits the observed observation value to the information processing device 1A. For example, the observation unit 33A receives information from a terminal to which the selection unit 32A transmitted information, and obtains the result of observing the received information as the observation value.
[0038] <Information processing method flow> The flow of the information processing method S1A executed by the information processing system 10A configured as above will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S1A. The information processing system 10A executes the information processing method S1A in each trial of determining which one to select from a plurality of options.
[0039] As shown in FIG. 4, an information processing method S1A includes steps S11A to S16A.
[0040] (Step S11A) In step S11A, the acquisition unit 31A of the server 3A acquires related information for each of the multiple options, and transmits each piece of acquired related information to the information processing device 1A.
[0041] (Step S12A) In step S12A, the acquisition unit 11A of the information processing device 1A receives the related information for each option from the server 3A.
[0042] (Step S13A) In step S13A, the determination unit 12A determines which of the multiple options to select using one of the multiple predictors. Details of the determination process are as described in step S12 in the exemplary embodiment 1. In addition, the determination unit 12A transmits information indicating the determined option to the server 3A.
[0043] (Step S14A) In step S14A, the selection unit 32A of the server 3A selects an option indicated by the information received from the information processing device 1A.
[0044] (Step S15A) In step S15A, the observation unit 33A observes the gain obtained by the selection by the selection unit 32A, and transmits the observed value to the information processing device 1A.
[0045] (Step S16A) In step S16A, the accumulation unit 13A of the information processing device 1A accumulates the received observation value and the related information of the option determined in step S13A in the storage device as learning data.
[0046] <Advantages of this exemplary embodiment> With the above configuration, this exemplary embodiment achieves the same effects as the first exemplary embodiment by having the information processing device 1A cooperate with the server 3A.
[0047] Exemplary embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Here, this exemplary embodiment solves the problems described in [Problems to be Solved by the Invention], and further solves the following problems in the related art described in Non-Patent Document 1.
[0048] <Problems with related technologies> The algorithm described in the above-mentioned Non-Patent Document 1 assumes a linear model as the relationship between the observed value and the context. Therefore, when the relationship between the observed value and the context cannot be expressed by a linear model, the algorithm may not be able to select an option that increases the accumulation of the observed value. The reason for this is explained below. When the algorithm is used, an option other than the optimal option may be selected in each trial due to a deviation between the original relationship between the observed value and the linear model. As a result, a loss may occur in each trial by comparing the gain that should be obtained from the optimal option with the observed value. This is because the accumulation of losses increases as the number of trials increases.
[0049] Another contextual bandit algorithm is described in Reference 1 below, but has the following problems. That is, the algorithm described in Reference 1 assumes that it is possible to identify differences from a linear model in the relationship between observed values and context. The algorithm also assumes that the set of options in each trial does not change. Therefore, the algorithm has a problem in that it cannot be applied when it is not possible to identify differences from a linear model or when the set of options changes in each trial.
[0050] [Reference 1] Tor Lattimore et. al. "Learning with good feature representations in bandits and in RL with a generative model", arXiv preprint arXiv:1911.07676, 2019. To solve the problems of the related art described above, an information processing device 1B according to this exemplary embodiment has the configuration described below.
[0051] <Outline of information processing device> The information processing device 1B is a device that repeats trials of acquiring related information associated with each of a plurality of options, determining which of the plurality of options to select, and storing the observed value and related information obtained by selecting the determined option as learning data. In each trial, the information processing device 1B executes processing using each predictor, starting from the first predictor among the plurality of predictors, until it arrives at a predictor suitable for determining the option. The information processing device 1B also stores learning data including the observed value of the gain obtained by selecting the determined option in association with the predictor used to determine the option.
[0052] <Configuration of information processing device 1B> The configuration of an information processing device 1B according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 1B.
[0053] 5, the information processing device 1B includes a control unit 110B and a storage unit 150B. The control unit 110B includes an acquisition unit 11B, a determination unit 12B, and an accumulation unit 13B. The storage unit 150B stores a learning data group Ψ including one or more pieces of learning data.
[0054] The acquiring unit 11B is a component that realizes an acquiring means in this exemplary embodiment. The deciding unit 12B is a component that realizes a deciding means in this exemplary embodiment. The storing unit 13B is a component that realizes a storing means in this exemplary embodiment. The memory unit 150B is a component that realizes a storage device in this exemplary embodiment.
[0055] The acquisition unit 11B acquires related information for each of the multiple options. The set of multiple options from which related information is acquired may be variable in each trial. That is, the number of options that are the target in each trial may be different from the number of options that are the target in other trials. Furthermore, at least one option that is the target in each trial may be different from the options in other trials.
[0056] The determination unit 12B uses one of the multiple predictors to determine which of the multiple options to select. The determination unit 12B will be described in detail later.
[0057] The accumulation unit 13B acquires an observation value of the gain obtained by selecting the option determined by the determination unit 12B. The accumulation unit 13B also accumulates learning data including the acquired observation value and the option determined by the determination unit 12B in the memory unit 150B in association with the predictor used to determine the option.
[0058] (Detailed configuration of the decision unit) A detailed configuration of the determination unit 12B will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a block diagram showing the configuration of the information processing device 1B as described above. Fig. 6 is a diagram showing a schematic explanation of an information processing method executed by the information processing device 1B. As shown in Fig. 5, the determination unit 12B includes a management unit 121B, a prediction unit 122B, a first determination unit 123B, and a progress unit 124B.
[0059] The management unit 121B manages S predictors. Here, S is an integer equal to or greater than 2. The management unit 121B generates S predictors and initializes each of them. Initialization means generating a predefined initial function as a prediction function and a prediction error function, which will be described later. The management unit 121B executes a prediction process using each predictor in turn in each trial, executes a progress process if the judgment condition is satisfied, and executes a first determination process if the judgment condition is not satisfied. When the management unit 121B executes the first determination process using any predictor, it does not execute any process using other predictors in the trial thereafter. Details of the prediction process, the first determination process, and the progress process will be described later.
[0060] Here, as shown in FIG. 6, the order of use of the S predictors is determined by the management unit 121B. In the example of FIG. 6, the order is determined as predictor 1, predictor 2, . . . predictor S. In other words, in this example, the reference symbol "s" (s=1, 2, . . . , S) attached to the predictors represents the order determined for the predictors. In addition, the predictor currently used by the management unit 121B is also described as the current predictor. In addition, the management unit 121B ending a process using the current predictor and starting a process using the next predictor is also described as proceeding to the next predictor, or the process proceeding to the next predictor. In other words, the management unit 121B executes each of the above-mentioned processes using the current predictor in the order of predictor 1, predictor 2, . . ., and proceeds to the next predictor until the first determination process is performed.
[0061] (Configuration of prediction section) The prediction unit 122B executes the prediction process described above. The prediction process is a process of predicting a prediction error associated with a predicted value for each of the option groups targeted by the current predictor, using the current predictor. Specifically, the prediction process includes a learning process that causes the current predictor to learn the relationship between the observed value and related information, and a calculation process that calculates a predicted value and a prediction error using the relationship.
[0062] (Target options) Here, the target set of options refers to one or more options among multiple options. As shown in Figure 6, the predictor s predicts the set of options I t,s The target is the set of options I targeted by predictor 1. t,1 The option group I is made up of options to which the related information acquired by the acquisition unit 11B is linked. t,2 is option group I t,1 In this way, a group of options targeted by the next predictor is extracted from a group of options targeted by the current predictor. The extraction of the group of options is performed by a progression unit 124B, which will be described later.
[0063] (Learning process) The learning process is a process in which the current predictor learns the relationship between the observed value and related information. Here, the relationship may be a linear relationship, for example. For learning by the current predictor, one or more pieces of learning data associated with the predictor are used from among the learning data set Ψ stored in the storage unit 150B. One piece of learning data includes related information of an option and an observed value of a gain obtained by selecting the option. For example, as shown in FIG. 6, a predictor s is provided with a learning data set Ψ t,s are associated with the training data set Ψ t,s is included in the learning data set Ψ stored in the storage unit 150B. t,s is made up of the learning data associated with the predictor s up to trial t-1. The process of associating the learning data with the predictor s is executed by the above-mentioned accumulation unit 13B.
[0064] Here, learning by the predictor s is done by using a training data set Ψ t,s The prediction unit 122B constructs a prediction function and a prediction error function through machine learning using the predictor s. The prediction function is a function that predicts an observed value of a gain based on related information. The prediction error function is a function that calculates a prediction error associated with a predicted value by the prediction function. Hereinafter, the prediction function and the prediction error function constructed through machine learning using the predictor s are also simply referred to as the prediction function and the prediction error function constructed by the predictor s.
[0065] (Calculation process) The calculation process is a process of calculating a predicted value and a prediction error by applying a prediction function and a prediction error function constructed by the current predictor to the associated information of each of the option groups targeted by the current predictor.
[0066] (Configuration of the first decision unit) The first determination unit 123B performs the above-mentioned first determination process. The first determination process is a process for determining which of the option groups targeted by the current predictor is to be selected when the judgment condition is not satisfied.
[0067] (Determination condition) Here, the judgment condition is a condition related to each prediction error predicted by the prediction process. In this exemplary embodiment, the judgment condition is that each prediction error is equal to or less than a first threshold. In this exemplary embodiment, the first threshold is an example of the "threshold" described in the claims. In other words, when the judgment condition is not satisfied, it means that at least one of the prediction errors is larger than the first threshold.
[0068] (Structure of the Progression Section) The progression unit 124B performs the progression process described above. The progression process is a process for extracting a group of options targeted by the next predictor from a group of options targeted by the current predictor when a judgment condition is satisfied, and is a process for proceeding to the next predictor. Specifically, the progression unit 124B extracts, from the group of options targeted by the current predictor, an option that satisfies a condition that it is highly likely to be the optimal option, as a group of options targeted by the next predictor. As an example, the progression unit 124B extracts, from the group of options targeted by the current predictor, one or more options whose sum of a prediction value and a prediction error is equal to or greater than a predetermined value, as a group of options targeted by the next predictor. After that, the management unit 121B proceeds to the process to the next predictor. That is, the management unit 121B executes the prediction process and the first determination process or the progression process using the next predictor.
[0069] (Example of processing that proceeds using a predictor) A specific example of the process proceeding in this way starting from predictor 1 will be described with reference to FIG. 6. In the example of FIG. 6, the process proceeds in this way starting from predictor 1, and a first determination process is executed using predictor s. This also results in the option to be selected being determined using predictor s. Specifically, prediction process is executed using predictor 1, and the judgment condition is satisfied, so the process proceeds to predictor 2. Next, prediction process is executed using predictor 2, and the judgment condition is satisfied, so the process proceeds to the next predictor. Then, the process proceeds to the next predictor, prediction process is executed using predictor s, and the judgment condition is not satisfied. Therefore, the option group I targeted by predictor s is determined as follows: t,s One of the options is selected. For example, option group I t,sThe option with the prediction error larger than the first threshold is determined. In addition, the training data including the observed values obtained by selecting the option is added to the training data set Ψ t,s will be added.
[0070] <Information processing method flow> The flow of the information processing method S1B executed by the information processing device 1B configured as above will be described with reference to Fig. 7. Fig. 7 is a flow diagram showing the flow of the information processing method S1B. The information processing device 1B executes the information processing method S1B in each trial of determining which one to select from a plurality of options.
[0071] As shown in FIG. 7, the information processing method S1B includes steps S11B to S15B.
[0072] (Step S11B) In step S11B, the acquisition unit 11B acquires related information for each of the multiple options.
[0073] (Step S12B) In step S12B, the determination unit 12B uses one of the multiple predictors to determine which option to select. Details of this step will be described later.
[0074] (Step S13B) In step S13B, the determination unit 12B outputs information indicating the determined option. For example, the determination unit 12B may display the information indicating the determined option on a display device or the like.
[0075] (Step S14B) In step S14B, the storage unit 13B acquires an observation value of the gain obtained by selecting the determined option. For example, the storage unit 13B may acquire the observation value via an input device.
[0076] (Step S15B) In step S15B, the accumulation unit 13B accumulates the learning data including the observation value and the related information of the determined option in the storage unit 150B in association with the predictor used to determine the option in step S12B.
[0077] With this, the information processing device 1B ends the information processing method S10.
[0078] <Detailed flow of decision process> Next, a detailed flow of the process in step S12B will be described with reference to Fig. 8. Fig. 8 is a flow chart showing a detailed flow of the process for determining which option to select. As shown in Fig. 8, step S12B includes steps S21B to S27B.
[0079] (Step S21B) In step S21B, the management unit 121B starts processing using the first predictor.
[0080] (Step S22B) In step S22B, the prediction unit 122B causes the current predictor to learn the relationship between the observed value and the related information. Specifically, the prediction unit 122B uses the current predictor to construct a prediction function and a prediction error function.
[0081] (Step S23B) In step S23B, the prediction unit 122B uses the current predictor to predict a prediction error associated with a predicted value of a gain for each of the options targeted by the current predictor. Specifically, the prediction unit 122B calculates a predicted value and a prediction error by applying the prediction function and the prediction error function constructed in step S22B to the related information of each option.
[0082] (Step S24B) In step S24B, the management unit 121B judges whether or not a judgment condition for each prediction error predicted in step S23B is satisfied. Here, the judgment condition is that each prediction error is equal to or smaller than a first threshold value.
[0083] (Yes in step S24B: step S25B) If the answer is Yes in step S24B, in step S25B, the progression unit 124B extracts one or more options from the option group targeted by the current predictor as an option group targeted by the next predictor. For example, the progression unit 124B extracts the predicted value predicted in step S23B and the sum of the prediction error is greater than a predetermined value.
[0084] (Step S26B) In step S26B, management section 121B starts processing using the next predictor, and repeats the processing from step S22B.
[0085] (No in step S24B: step S27B) On the other hand, if the answer is No in step S24B, in step S27B, the first determination unit 123B determines which of the options targeted by the current predictor to select. Here, the first determination unit 123B determines to select an option whose prediction error is equal to or greater than the first threshold. Then, the information processing device 1B ends the determination process in step S12B.
[0086] <Advantages of this exemplary embodiment> As described above, the information processing device 1B and the information processing method S1B according to this exemplary embodiment employ the following configurations (i) to (iii). Configuration (i) is a configuration for predicting a prediction error associated with a predicted value of a gain for each of a group of options targeted by the predictor using a current predictor. Configuration (ii) is a configuration for determining which of the group of options targeted by the predictor is to be selected when a judgment condition for each prediction error is not satisfied. Configuration (iii) is a configuration for extracting one or more options from the target group of options as a group of options targeted by another predictor when the judgment condition is satisfied, and predicting the prediction error of each option using the other predictor. In addition, in this exemplary embodiment, a condition that each prediction error is equal to or less than a first threshold is employed as a judgment condition.
[0087] This exemplary embodiment further employs the following configurations (iv) and (v). The configuration (iv) is a configuration in which learning data including an observation value obtained by selecting a determined option and related information of the option is stored in a storage unit in association with a predictor used to determine the option. The configuration (v) is a configuration in which each of the multiple predictors learns the relationship between the observation value and related information by referring to the learning data associated with itself among the learning data stored in the storage unit.
[0088] With these configurations, the present exemplary embodiment narrows down options using a predictor with a smaller prediction error, and then determines which option to select using at least one other predictor that should be further trained because it has a large prediction error. Therefore, the present exemplary embodiment can select an option with a smaller prediction error.
[0089] In addition, in the present exemplary embodiment, in further training of the predictor used to determine the option, the observed values obtained by selecting the option with a large prediction error are used as training data, so that the predictor with insufficient prediction accuracy can be trained more effectively.
[0090] Exemplary embodiment 4 The fourth exemplary embodiment of the present invention will now be described in detail with reference to the drawings.
[0091] <Outline of information processing system> An information processing system 10C according to this exemplary embodiment is a system in which an information processing device 1C functions in cooperation with a server 3C. The information processing device 1C is a modified version of the information processing device 1B according to exemplary embodiment 3. The server 3C is a modified version of the server 3A according to exemplary embodiment 2.
[0092] In this exemplary embodiment, a user is applied as an example of an option in exemplary embodiment 2 or 3. Furthermore, feature information indicating the features of a user is applied as an example of related information in exemplary embodiment 2 or 3. Furthermore, "sending a product promotion to the user" is applied as an example of "selecting an option" in exemplary embodiment 2 or 3. Furthermore, the result of sending the promotion is applied as an example of an observed value in exemplary embodiment 2 or 3. Hereinafter, a description such as "similar to exemplary embodiment 2 or 3" means that the same explanation can be given by replacing the option with the user, the related information with feature information, the selection of an option with the sending of the promotion, and the observed value with the sending result in the description of exemplary embodiment 2 or 3.
[0093] Specifically, the information processing system 10C according to the present exemplary embodiment is a system that selects one of a plurality of users and sends a promotion to the selected user. The promotion is, for example, a promotion for selling a product. Although the type of promotion is not limited, the following description will be given assuming that there is one type of promotion.
[0094] The information processing system 10C repeats the process of acquiring characteristic information indicating the characteristics of each of the multiple users, determining which of the multiple users to select, and storing the promotion sending results and the characteristic information of the determined users as learning data. The sending results indicate the purchasing behavior of the users to whom the promotion was sent, for example, whether or not they purchased the product.
[0095] <Configuration of information processing system> The configuration of the information processing system 10C will be described with reference to FIG. 9. FIG. 9 is a block diagram showing the configuration of the information processing system 10C. As shown in FIG. 9, the information processing system 10C includes an information processing device 1C and a server 3C. The information processing device 1C and the server 3C are communicably connected via a network N1. The server 3C is communicably connected to a plurality of terminals 9C-i (i=1, 2, ..., I: I is an integer of 2 or more) via a network N2. Although three terminals 9C-i are shown in FIG. 9, the number of terminals 9C-i to which the server 3C is connected may be two or four or more. The networks N1 and N2 are each a wireless LAN, a wired LAN, a WAN, a public line network, a mobile data communication network, or a combination of these networks.
[0096] (Server 3C configuration) The server 3C is a device that transmits information indicating a promotion to any one of a plurality of terminals 9C-i in each trial t (t=1, 2, ..., T: T is an integer equal to or greater than 2). That is, the information transmitted by the server 3C is presented to a user i of any one of the terminals 9C-i. Hereinafter, transmitting information indicating a promotion to the terminal 9C-i will also be described as sending a promotion to the user i. The user i performs a purchasing behavior in response to the promotion sent to him / her. Herein, the purchasing behavior is assumed to be either purchasing or not purchasing a product. However, the purchasing behavior is not limited to these two types of behavior. Furthermore, the purchasing behavior may include three or more types of behavior.
[0097] As shown in FIG. 9, the server 3C includes an acquisition unit 31C, a selection unit 32C, an observation unit 33C, and a communication unit 34C.
[0098] The acquisition unit 31C receives characteristic information of each of the target users i from the terminal 9C-i in each trial t. The acquisition unit 31C also transmits each received characteristic information to the information processing device 1C. Note that the set of "target users i in each trial t (i.e., target terminals 9C-i)" is variable. That is, the number of target users i in each trial t may be different from the number of target users in other trials. Also, at least one target user i in each trial t may be different from the users in other trials. Also, the number of users i determined by the information processing device 1C in each trial is not limited, but is assumed to be 1 here.
[0099] The selection unit 32C sends a promotion to the user i indicated by the information received from the information processing device 1C.
[0100] The observation unit 33C observes the promotion sending result by the selection unit 32C and transmits it to the information processing device 1C. Observing the promotion sending result means, for example, observing the purchasing behavior of the user i who is the promotion sending destination. In this case, the sending result is information indicating whether the user i purchased the product.
[0101] The communication unit 34C transmits and receives information to and from the information processing device 1C via the network N1.
[0102] (Configuration of information processing device 1C) As shown in FIG. 9, an information processing device 1C includes a control unit 110C, a storage unit 150C, and a communication unit 160C.
[0103] The storage unit 150C stores a learning data group Ψ. The learning data constituting the learning data group Ψ includes user characteristic information and the results of sending promotions to the user.
[0104] The communication unit 160C transmits and receives information to and from the server 3C via the network N1.
[0105] The control unit 110C includes an acquisition unit 11C, a determination unit 12C, and a storage unit 13C.
[0106] Here, the acquisition unit 11C is a configuration that realizes an acquisition means in this exemplary embodiment. The determination unit 12C is a configuration that realizes a determination means in this exemplary embodiment. The accumulation unit 13C is a configuration that realizes an accumulation means in this exemplary embodiment. The storage unit 150C is a configuration that realizes a storage device in this exemplary embodiment.
[0107] The acquisition unit 11C receives the feature information of each of the multiple users i from the server 3C in each trial t. As described above, the set of multiple users i from which feature information is to be acquired is variable in each trial. The feature information is information indicating the feature of the user i, and has a number of dimensions d. Here, the number of dimensions of the feature information refers to, for example, the number of dimensions of a vector space required to express the feature information as a vector. As an example, when the feature information is expressed by multiple numerical values, the number of the numerical values is the dimension of the feature information. As a specific example, the feature information may include four items, namely, the identification information of the user i, the history of promotions sent to the user i, the history of products purchased by the user i, and the age of the user i. In this case, the number of dimensions of the feature information is four. However, the specific example does not limit the configuration of the feature information and the number of dimensions d.
[0108] The determination unit 12C uses one of the S predictors to determine which of the multiple users i to select. The predictor used to determine the user i is referred to as a predictor s1. The determination unit 12C also transmits information indicating the determined user i to the server 3C. Details of the determination unit 12C will be described later.
[0109] The accumulation unit 13C receives the result of sending the promotion to the user i from the server 3C. In addition, the accumulation unit 13C accumulates the learning data including the received sending result and the feature information of the user i in the memory unit 150B in association with the predictor s1 used to determine the user i.
[0110] (Detailed configuration of the decision unit) The following describes a detailed configuration of the determination unit 12C. The determination unit 12C includes a management unit 121C, a prediction unit 122C, a first determination unit 123C, a second determination unit 125C, and a progression unit 124C.
[0111] The management unit 121C manages S predictors in the same manner as the management unit 121B in exemplary embodiment 3. Details of the S predictors are as described in exemplary embodiment 3. However, details of the process executed by the management unit 121C using each predictor are different from those in exemplary embodiment 3.
[0112] Specifically, the management unit 121C executes a prediction process using each predictor in turn in each trial, executes a progress process if the judgment condition is satisfied, and executes a first determination process or a second determination process if the judgment condition is not satisfied. C When the first determination process or the second determination process has been performed using any one of the predictors, thereafter, in the same trial, processes using other predictors are not performed.
[0113] (Determination condition) The judgment condition used by the management unit 121C is different from the judgment condition used by the management unit 121B in the exemplary embodiment 3. In this exemplary embodiment, the judgment condition is that "each prediction error is greater than a second threshold and is equal to or smaller than a first threshold." Specifically, the first threshold is expressed by the following formula (1). Moreover, the second threshold is expressed by the following formula (2).
[0114]
number
[0115]
number
[0116] (Number of dimensions of feature information) Here, the judgment condition is a condition based on the number of dimensions of the feature information. Specifically, the larger the number of dimensions d of the feature information, the larger the value of α in the above-mentioned formulas (1) and (2) is set. For example, α is expressed by the following formulas (3) and (4).
[0117]
number
[0118]
number
[0119] (Configuration of prediction section) The prediction unit 122C executes the prediction process described above. The prediction process is a process of predicting a prediction error associated with a predicted value of a sending result for each of a user group targeted by the current predictor s, using the current predictor s. Specifically, the prediction process is a process of predicting a relationship θ t,s The prediction process includes a learning process for learning the relationship θ t,s Using this, the predicted value of the sending result γ t,s (i) and the prediction error w t,s The prediction function and the prediction error function used by the prediction unit 122C in the calculation process are expressed by the following equations (5) and (6).
[0120]
number
[0121]
number
[0122] (Configuration of the first decision unit) The first determination unit 123C performs the first determination process described above. The first determination process in this exemplary embodiment is different from the first determination process in exemplary embodiment 3. The first determination process in this exemplary embodiment is performed when the judgment condition is not satisfied and the following formula (7) is satisfied, and the first determination process determines whether the current predictor s is a user group I t,s This is a process of determining which of the above should be selected.
[0123]
number
[0124] (Configuration of the second decision unit) The second determination unit 125C performs the above-mentioned second determination process. The second determination process is a process in which, when the judgment condition is not satisfied and the following formula (8) is satisfied, the user group I targeted by the current predictor is determined as follows: t,s This is a process of determining which of the above should be selected.
[0125]
number
[0126] (Structure of the Progression Section) The progression unit 124C performs the above-mentioned progression process. The progression process is a process in which, when the determination condition is satisfied, the option set I t,s The next predictor s+1 targets the set of options I t,s+1 and proceeds to the next predictor s+1.
[0127] Here, the option group extracted by progression unit 124C is expressed by the following formula (9).
[0128]
number
[0129] In this way, when each prediction error predicted by the current predictor s is greater than the second threshold and is equal to or less than the first threshold, the progression unit 124C refers to the prediction result by the current predictor s and selects the target user group I t,s After narrowing down the predictors, the process proceeds to the next predictor s+1.
[0130] <Information processing method flow> The flow of the information processing method S1C executed by the information processing system 10C configured as above will be described with reference to Fig. 10. Fig. 10 is a flow diagram showing the flow of the information processing method S1C. The information processing system 10C executes the information processing method S1C in each trial of determining which of a plurality of users i to select.
[0131] As shown in FIG. 10, the information processing method S1C includes steps S11C to S18C.
[0132] (Step S11C) In step S11C, the acquisition unit 31C of the server 3C acquires characteristic information for each of the multiple users, and transmits each piece of acquired characteristic information to the information processing device 1C.
[0133] (Step S12C) In step S12C, the acquisition unit 11C of the information processing device 1C receives the characteristic information of each user from the server 3C.
[0134] (Step S13C) In step S13C, the determination unit 12C determines which of the multiple users to select. The details of the determination process will be described later.
[0135] (Step S14C) In step S14C, the determination unit 12C transmits information indicating the determined user to the server 3C.
[0136] (Step S15C) In step S15C, selection unit 32C of server 3C sends a promotion to the user indicated by the information received from information processing device 1C.
[0137] (Step S16C) In step S16C, the observation unit 33C observes the purchasing behavior of the user to whom the promotion has been sent by the selection unit 32C, and transmits the observed sending result to the information processing device 1C.
[0138] (Step S17C) In step S17C, accumulation unit 13C of information processing device 1C accumulates the received sending result and the user feature information determined in step S13C in storage unit 150C as learning data.
[0139] <Detailed flow of decision process> Next, a detailed flow of the process in step S13C will be described with reference to Fig. 11. Fig. 11 is a flow chart showing a detailed flow of the process of determining which user to select. As shown in Fig. 11, step S13C includes steps S21C to S29C.
[0140] (Step S21C) In step S21C, the management unit 121C starts processing using the first predictor.
[0141] (Step S22C) In step S22C, the prediction unit 122C causes the current predictor to learn the relationship between the sending result and the feature information. Specifically, the prediction unit 122C uses the current predictor to construct a prediction function and a prediction error function.
[0142] (Step S23C) In step S23C, the prediction unit 122C uses the current predictor to predict a prediction error associated with the predicted value of the sending result for each of the users targeted by the current predictor. Specifically, the prediction unit 122C calculates a predicted value and a prediction error by applying the feature information of each user to the prediction function and the prediction error function constructed in step S22C.
[0143] (Step S24C) In step S24C, management section 121C determines whether or not each prediction error predicted in step S23C is equal to or smaller than a second threshold value.
[0144] (Yes in step S24C: step S25C) If the answer is Yes in step S24C, in step S25C, the second determination unit 125C determines to select a user who has the largest sum of the predicted value and the prediction error predicted in step S23C. Then, the second determination unit 125C transmits information indicating the determined user to the server 3C and ends the determination process.
[0145] (No in step S24C: step S26C) On the other hand, if it is determined as No in step S24C, in step S26C, the management section 121C determines whether or not each prediction error predicted in step S23C is equal to or smaller than the first threshold value.
[0146] (No in step S26C: step S27C) If the result of step S26C is No, then in step S27C, the first determination unit 123C determines to select a user whose prediction error predicted in step S23C exceeds a first threshold value. Then, the first determination unit 123C transmits information indicating the determined user to the server 3C, and ends the determination process.
[0147] (Yes in step S26C: step S28C) If it is determined that the answer is Yes in step S26C, in step S28C, progression unit 124C extracts one or more options from the group of options targeted by the current predictor, as a group of options targeted by the next predictor.
[0148] (Step S29C) In step S29C, management section 121C starts processing using the next predictor, and repeats the processing from step S22C.
[0149] <Example> Next, an example of a simulation performed using the information processing device 1C in this exemplary embodiment will be described with reference to Fig. 12. Fig. 12 is a graph showing the simulation results.
[0150] In Fig. 12, the horizontal axis indicates the total number of trials T, and the vertical axis indicates the accumulated loss. Example G1 shows the results of a simulation using information processing device 1C. Comparative examples G2 to G4 are simulation results performed using related techniques A to C for comparison. Here, as related techniques A, B, and C, Greedy, LinUCB, and LinTS, which are known as well-known bandit algorithms, were used.
[0151] The simulations in Example G1 and Comparative Examples G2 to G4 were carried out under the following conditions. That is, the number of users was set to 2. That is, i was set to 1 or 2. In addition, the characteristic information x of the users to be acquired in the first half and the second half of the total number of trials T was set to 1. t (i) and the observed value γ t (i) is changed as follows. In other words, in this embodiment, the user's characteristic information x t (i) and the observed value γ t The relationship in (i) differs significantly between the first and second halves and cannot be expressed by a single linear model.
[0152] (first half:t <T / 2) x t (1)=(ε,0), xt(2)=(ε,0) Gamma t (1)=-ε / 2, γt(2)=-ε / 2 where 0<ε≪1 (Second half: t≧T / 2) x t (1)=(2,1), xt(2)=(0,0) Gamma t (1)=1 / 2, γt(2)=0 12, in Example G1 using the information processing device 1C, the cumulative loss was 0 regardless of the total number of trials T. In contrast, in Comparative Examples G2 to G4 using Related Technologies A to C, the cumulative loss increased as the total number of trials T increased.
[0153] In this manner, in this Example G1, the result was obtained that by using the information processing device 1C according to this exemplary embodiment, the cumulative loss that occurs in the Comparative Examples G2 to G4 does not occur.
[0154] <Advantages of this exemplary embodiment> In this exemplary embodiment, in addition to the same effects as those of the exemplary embodiments 2 and 3, it is possible to further increase the accumulation of observed values (sent results). This is because a condition according to the number of dimensions of the user's feature information is used as a judgment condition for judging whether or not to advance the process from each predictor to the next predictor. This makes it possible to more fully absorb the difference between the original relationship between the user's feature information and observed values and the linear model.
[0155] [Modifications] In each of the above exemplary embodiments, the number of predictors generated initially may be one. For example, each information processing device generates a new predictor when it is determined that none of the predictors already generated in each trial is appropriate for determining an option. For example, in exemplary embodiments 3 to 4, when one or more predictors already generated satisfy the judgment condition, the determination unit generates a new predictor and allows the new predictor to proceed with the process.
[0156] [Software implementation example] A part or all of the functions of the information processing devices 1, 1A, 1B, and 1C may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.
[0157] In the latter case, the information processing devices 1, 1A, 1B, and 1C are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 13. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1, 1A, 1B, and 1C. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1, 1A, 1B, and 1C.
[0158] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.
[0159] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.
[0160] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.
[0161] [Additional Note 1] The present invention is not limited to the exemplary embodiments described above, and various modifications are possible within the scope of the claims. For example, exemplary embodiments obtained by appropriately combining the technical means disclosed in the exemplary embodiments described above are also included in the technical scope of the present invention.
[0162] [Additional Note 2] Some or all of the exemplary embodiments described above may be described as follows. However, the present invention is not limited to the following described aspects.
[0163] (Appendix 1) An acquisition means for acquiring related information associated with each of the plurality of options; A determination means for determining which of the plurality of options to select; a storage means for storing the observed value of the gain obtained by selecting the option determined by the determination means and related information of the option in a storage device as learning data; The determining means is determining which of the plurality of options to select by using any of a plurality of predictors that refer to the learning data and learn a relationship between the associated information and the observed value, each predictor learning the relationship independently of each other; 23. An information processing apparatus comprising:
[0164] According to the configuration of Supplementary Note 1, one of the multiple predictors is used to determine which of the multiple options to select. In addition, each predictor refers to the learning data and learns the relationship between the related information and the observed value independently of each other. Therefore, in order to determine which of the multiple options to select, it is possible to use a predictor that has more appropriately learned the relationship between the observed value and the related information. As a result, the present exemplary embodiment can determine an option that further increases the cumulative gain.
[0165] (Appendix 2) 2. The information processing device according to claim 1, The determining means is (i) using any one of the predictors, predicting a prediction error associated with a predicted value of the payoff for each of a group of options, which is one or more options targeted by the predictor; (ii) determining which of the alternatives targeted by the predictor is to be selected when the criterion for each prediction error is not satisfied; (iii) if the determination condition is satisfied, extracting one or more options from the target group of options, setting the extracted one or more options as a group of options to be targeted by another predictor, and predicting the prediction error of each option using the other predictor; 23. An information processing apparatus comprising:
[0166] According to the configuration of Supplementary Note 2, it is possible to use a condition for determining whether or not the relationship between the observed value and the related information is predicted with a higher accuracy than a certain standard as the judgment condition. This makes it possible to narrow down the options using a trained predictor with higher accuracy, and then to determine which option to select using another predictor to be trained. As a result, even if a single predictor cannot accurately predict the relationship between the observed value and the related information, it is possible to select an option that increases the accumulation of the observed value.
[0167] (Appendix 3) The information processing device according to claim 2, The determining means is The judgment condition includes that each prediction error is equal to or less than a threshold value. 23. An information processing apparatus comprising:
[0168] According to the configuration of Supplementary Note 3, the options are narrowed down using a predictor with a smaller prediction error, and then the prediction error It is possible to determine which option to select using a predictor that requires further learning because is greater than the threshold. As a result, even if a single predictor cannot accurately learn the relationship between the observed value and related information, it is possible to select the option that is optimal for learning from among options with smaller prediction errors.
[0169] (Appendix 4) The information processing device according to claim 2 or 3, The determining means is 2. An information processing apparatus comprising: a determining condition based on a number of dimensions of the related information.
[0170] According to the configuration of Supplementary Note 4, it is possible to more accurately absorb the deviation between the original relationship between the observed value and the related information and the relationship learned by each predictor.
[0171] (Appendix 5) An information processing device according to any one of claims 1 to 4, The storage means includes: storing the learning data in the storage device in association with a predictor used by the determining means to determine the option; each of the plurality of predictors learns the relationship by referring to learning data associated with itself among the learning data stored in the storage device; 23. An information processing apparatus comprising:
[0172] According to the configuration of Supplementary Note 5, the predictor used to determine the option can be further trained to predict the observed value with higher accuracy.
[0173] (Appendix 6) 6. An information processing device according to any one of claims 1 to 5, Each predictor learns a linear relationship as the relationship. 23. An information processing apparatus comprising:
[0174] According to the configuration of Supplementary Note 6, it is possible to use a predictor that has learned a more appropriate linear relationship between the observed values and related information in order to determine which of a plurality of options to select.
[0175] (Appendix 7) An information processing device, Obtaining related information associated with each of the multiple options; determining which of the plurality of options to select; and Storing the observed value of the gain obtained by selecting the determined option and related information of the option as learning data in a storage device; In order to determine which of the plurality of options to select, a plurality of predictors are used, the plurality of predictors being configured to learn a relationship between the associated information and the observed value by referring to the learning data, and each predictor is configured to learn the relationship independently of each other. 23. An information processing method comprising:
[0176] According to the configuration of Supplementary Note 7, the same effects as those of the information processing device described in Supplementary Note 1 are achieved.
[0177] (Appendix 8) A storage medium storing a program for causing a computer to function as an information processing device, The program causes the computer to An acquisition means for acquiring related information associated with each of the plurality of options; A determination means for determining which of the plurality of options to select; functioning as a storage means for storing the observed value of the gain obtained by selecting the option determined by the determination means and related information of the option as learning data in a storage device; The determining means is determining which of the plurality of options to select by using any of a plurality of predictors that refer to the learning data and learn a relationship between the associated information and the observed value, each predictor learning the relationship independently of each other; A storage medium storing a program characterized by:
[0178] According to the configuration of Supplementary Note 8, the same effects as those of the information processing device described in Supplementary Note 1 are achieved.
[0179] (Appendix 9) An information processing system including an information processing device and a server, The information processing device includes: An acquisition means for receiving related information associated with each of a plurality of options from the server; a determination means for determining which of the plurality of options to select and transmitting information indicating the determined option to the server; a storage means for receiving from the server an observed value of a gain obtained by selecting the option determined by the determination means, and storing the received observed value and related information of the option in a storage device as learning data; The determining means is determining which of the plurality of options to select by using any of a plurality of predictors that refer to the learning data and learn a relationship between the associated information and the observed value, each predictor learning the relationship independently of each other; The server, an acquisition means for acquiring the related information and transmitting the related information to the information processing device; A selection means for selecting an option indicated by the information received from the information processing device; and an observation means for observing a gain obtained by the selection by the selection means and transmitting the observed observation value to the information processing device. An information processing system comprising:
[0180] According to the configuration of Supplementary Note 9, the same effects as those of the information processing device described in Supplementary Note 1 are achieved.
[0181] (Appendix 10) An information processing device, receiving related information associated with each of the plurality of options from a server; determining which of the plurality of options to select and transmitting information indicating the determined option to the server; and receiving from the server an observed value of a gain obtained by selecting the determined option, and storing the received observed value and related information of the option in a storage device as learning data; To determine which of the plurality of options to select, determining which of the plurality of options to select by using any of a plurality of predictors that refer to the learning data and learn a relationship between the associated information and the observed value, each predictor learning the relationship independently of each other; The server, acquiring the related information and transmitting the related information to the information processing device; Selecting an option indicated by the information received from the information processing device; and Observing a gain obtained by selecting the option and transmitting the observed value to the information processing device. 23. An information processing method comprising:
[0182] According to the configuration of Supplementary Note 10, the same effects as those of the information processing device described in Supplementary Note 1 are achieved.
[0183] [Additional Note 3] Some or all of the above-described exemplary embodiments can be further expressed as follows.
[0184] An information processing device comprising at least one processor, the processor executing an acquisition process for acquiring related information linked to each of a plurality of options, a decision process for deciding which of the plurality of options to select, and a storage process for storing in a storage device as learning data an observed value of a gain obtained by selecting the option decided by the decision process and related information of the option, the decision process using one of a plurality of predictors that learn a relationship between the related information and the observed value by referring to the learning data, each predictor learning the relationship independently of each other, to decide which of the plurality of options to select.
[0185] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process, the determination process, and the accumulation process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]
[0186] 1, 1A, 1B, 1C Information processing device 10A, 10C Information Processing Systems 11, 11A, 11B, 11C Acquisition Department 12, 12A, 12B, 12C decision section 13, 13A, 13B, 13C storage section 110B, 110C Control section 121B, 121C Management Department 122B, 122C Prediction Section 123B, 123C 1st decision section 124B, 124C progress section 125C 2nd decision section 150B, 150C storage section 160C Communication Department 3, 3A, 3C Server 31A, 31C Acquisition Department 32A, 32C selection section 33A, 33C Observation Section 34C Communication Department
Claims
1. An acquisition means for acquiring related information associated with each of the plurality of options; A determination means for determining which of the plurality of options to select; a storage means for storing the observed value of the gain obtained by selecting the option determined by the determination means and related information of the option in a storage device as learning data; The determining means is a plurality of predictors that learn a relationship between the associated information and the observed value by referring to the learning data are caused to learn the relationship independently from each other, and one of the plurality of learned predictors is used to determine which of the plurality of options is to be selected; The determining means is (i) using any one of the predictors, predicting a prediction error associated with a predicted value of the payoff for each of a group of options, which is one or more options targeted by the predictor; (ii) determining which of the alternatives targeted by the predictor is to be selected when the decision condition for each prediction error is not satisfied; (iii) when the judgment condition is satisfied, extracting one or more options from the target group of options, and predicting the prediction error of each option using another predictor, with the extracted one or more options being a group of options targeted by another predictor; The related information is feature information indicating features of each of the plurality of options.
23. An information processing apparatus comprising:
2. The determining means is The judgment condition includes that each prediction error is equal to or less than a threshold value.
2. The information processing apparatus according to claim 1,
3. The determining means is 3. The information processing apparatus according to claim 1, wherein the determination condition is based on a number of dimensions of the related information.
4. The storage means includes: storing the learning data in the storage device in association with a predictor used by the determining means to determine the option; each of the plurality of predictors learns the relationship by referring to learning data associated with itself among the learning data stored in the storage device; 4. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises: a first input section;
5. Each predictor learns a linear relationship as the relationship.
5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises: a first input section;
6. An information processing device, Obtaining related information associated with each of the multiple options; determining which of the plurality of options to select; and Storing the observed value of the gain obtained by selecting the determined option and related information of the option as learning data in a storage device; In determining which of the plurality of options to select, a plurality of predictors that refer to the learning data and learn a relationship between the associated information and the observed value are caused to learn the relationship independently of each other, and one of the plurality of learned predictors is used to determine which of the plurality of options to select; In determining which of the plurality of options to select, (i) using any one of the predictors, predicting a prediction error associated with a predicted value of the payoff for each of a group of options, which is one or more options targeted by the predictor; (ii) determining which of the alternatives targeted by the predictor is to be selected when the decision condition for each prediction error is not satisfied; (iii) when the judgment condition is satisfied, extracting one or more options from the target group of options, and predicting the prediction error of each option using another predictor, with the extracted one or more options being a group of options targeted by another predictor; The related information is feature information indicating features of each of the plurality of options.
23. An information processing method comprising:
7. A storage medium storing a program for causing a computer to function as an information processing device, The program causes the computer to An acquisition means for acquiring related information associated with each of the plurality of options; A determination means for determining which of the plurality of options to select; functioning as a storage means for storing the observed value of the gain obtained by selecting the option determined by the determination means and related information of the option as learning data in a storage device; The determining means is a plurality of predictors that learn a relationship between the associated information and the observed value by referring to the learning data are caused to learn the relationship independently from each other, and one of the plurality of learned predictors is used to determine which of the plurality of options is to be selected; The determining means is (i) using any one of the predictors, predicting a prediction error associated with a predicted value of the payoff for each of a group of options, which is one or more options targeted by the predictor; (ii) determining which of the alternatives targeted by the predictor is to be selected when the decision condition for each prediction error is not satisfied; (iii) when the judgment condition is satisfied, extracting one or more options from the target group of options, and predicting the prediction error of each option using another predictor, with the extracted one or more options being a group of options targeted by the other predictor; The related information is feature information indicating features of each of the plurality of options. A storage medium storing a program characterized by:
8. An information processing system including an information processing device and a server, The information processing device includes: An acquisition means for receiving related information associated with each of a plurality of options from the server; a determination means for determining which of the plurality of options to select and transmitting information indicating the determined option to the server; a storage means for receiving from the server an observed value of a gain obtained by selecting the option determined by the determination means, and storing the received observed value and related information of the option in a storage device as learning data; The determining means is a plurality of predictors that learn a relationship between the associated information and the observed value by referring to the learning data are caused to learn the relationship independently from each other, and one of the plurality of learned predictors is used to determine which of the plurality of options is to be selected; The server, an acquisition means for acquiring the related information and transmitting the related information to the information processing device; A selection means for selecting an option indicated by the information received from the information processing device; an observation means for observing a gain obtained by the selection by the selection means and transmitting the observed observation value to the information processing device; The related information is feature information indicating features of each of the plurality of options. An information processing system comprising:
9. An information processing device, receiving related information associated with each of the plurality of options from a server; determining which of the plurality of options to select and transmitting information indicating the determined option to the server; and receiving from the server an observed value of a gain obtained by selecting the determined option, and storing the received observed value and related information of the option in a storage device as learning data; To determine which of the plurality of options to select, a plurality of predictors that learn a relationship between the associated information and the observed value by referring to the learning data are caused to learn the relationship independently from each other, and one of the plurality of learned predictors is used to determine which of the plurality of options is to be selected; The server, acquiring the related information and transmitting the related information to the information processing device; Selecting an option indicated by the information received from the information processing device; and observing a gain obtained by selecting the option and transmitting the observed value to the information processing device; The related information is feature information indicating features of each of the plurality of options.
23. An information processing method comprising:
Citation Information
Patent Citations
Contextual-bandit approach to personalized news article recommendation
US20150051973A1
Learning device, learning method, and learning program
WO2019198408A1
Information processing system, information processing method, and storage medium
WO2020012589A1
