Information processing device, information processing method, information processing system, and program

Abstract

In order to prevent a reward that is obtained from becoming excessively low in a state in which selectable options change every round, an information processing device (1) includes an acquisition unit (11) that acquires information indicating a set of selectable options in a certain round, a selection unit (12) that selects a plurality of options from the set, and an accumulation unit (13) that accumulates data including the plurality of options selected by the selection unit (12) and an observation value of a reward obtained from the plurality of options selected by the selection unit (12). The selection unit (12) selects the plurality of options such that a predicted value of the reward obtained by referencing data that the accumulation unit (13) has accumulated satisfies a constraint that is determined in accordance with baseline options selected from the set.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing system, and a program for selecting an option. [Background technology]

[0002] There is known a technique for optimizing an index (also called online optimization) under a condition where a function expressing the index to be optimized can change sequentially. In addition, in the online optimization technique, a conservative optimization technique is known that guarantees a certain degree of accuracy (reward) by setting a baseline (for example, Non-Patent Documents 1 to 3). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Abbas Kazerouni et. al., 'Conservative Contextual Linear Bandits', Mar 4, 2017 [Non-Patent Document 2] Evrard Garcelon et.al., 'Improved Algorithms for Conservative Exploration in Bandits', Feb 8, 2020 [Non-Patent Document 3] Sumeet Katariya et. al., 'Conservative Exploration using Interleaving', Jun 3, 2018 Summary of the Invention [Problem to be solved by the invention]

[0004] In the techniques described in Non-Patent Documents 1 and 2, only one option can be selected in one decision-making. In addition, there is a problem that performance cannot be guaranteed for each decision-making (each round). In the technique described in Non-Patent Document 3, multiple options can be selected in one decision-making, but since the entire options (population of options) is fixed, there is a problem that it cannot be applied to a situation where the population may change over time.

[0005] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its objective is to provide a technology that can prevent the reward obtained from being too low in a situation in which the selectable options may change from round to round. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention comprises an acquisition means for acquiring information indicating a set of options selectable in a certain round, a selection means for selecting a plurality of options from the set, and a storage means for storing data including the plurality of options selected by the selection means and observed values ​​of rewards obtained from the plurality of options selected by the selection means, wherein the selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set.

[0007] An information processing method according to one aspect of the present invention includes an information processing device repeatedly acquiring information indicating a set of options selectable in a certain round, selecting a plurality of options from the set, and accumulating data including the plurality of options selected in the selecting process and observed values ​​of rewards obtained by the plurality of options selected in the selecting process, wherein in the selecting process, the plurality of options are selected such that a predicted value of reward obtained by referring to the accumulated data satisfies a constraint condition determined according to a baseline option selected from the set.

[0008] A program according to one aspect of the present invention is a program that causes a computer to function as an information processing device, the program causing the computer to function as an acquisition means for acquiring information indicating a set of options selectable in a certain round, a selection means for selecting a plurality of options from the set, and a storage means for storing data including the plurality of options selected by the selection means and observed values ​​of rewards obtained from the plurality of options selected by the selection means, the selection means selecting the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set.

[0009] An information processing system according to one aspect of the present invention is an information processing system including an information processing device and a terminal device, wherein the information processing device comprises an acquisition means for acquiring information indicating a set of options selectable in a certain round, a selection means for selecting a plurality of options from the set, and a storage means for storing data including the plurality of options selected by the selection means and observed values ​​of rewards obtained by the plurality of options selected by the selection means, wherein the selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set, and the terminal device comprises an option acquisition means for acquiring the plurality of options selected by the selection means, and a reward information provision means for providing the information processing device with reward information indicating rewards obtained by performing actions indicated by the plurality of options.

[0010] An information processing method according to one aspect of the present invention includes an information processing device repeatedly acquiring information indicating a set of options selectable in a certain round, selecting a plurality of options from the set, and accumulating data including the plurality of options selected in the selecting process and observed values ​​of rewards obtained by the plurality of options selected in the selecting process, wherein in the selecting process, the plurality of options are selected under constraints determined according to a baseline option selected from the set so that a predicted value of reward obtained by referring to the accumulated data is larger, and a terminal device acquires the plurality of options selected in the selecting process, and provides reward information indicating rewards obtained by performing actions indicated by the plurality of options to the information processing device. Effect of the Invention

[0011] According to one aspect of the present invention, in a situation where the selectable options may change from round to round, it is possible to prevent the reward obtained from becoming too low. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Diagram 2] 1 is a flowchart showing a flow of an information processing method executed by an information processing device according to a first exemplary embodiment of the present invention. [Diagram 3] 1 is a block diagram showing a configuration of an information processing system according to a first exemplary embodiment of the present invention. [Figure 4] 1 is a flow chart showing the flow of an information processing method executed by an information processing system according to an exemplary embodiment 1 of the present invention. [Diagram 5] FIG. 11 is a block diagram showing a configuration of an information processing system according to a second exemplary embodiment of the present invention. [Figure 6] 11 is a flowchart showing the flow of an information processing method executed by an information processing device according to an exemplary embodiment 2 of the present invention. [Figure 7]FIG. 11 is a diagram showing an example of a screen according to the second exemplary embodiment of the present invention. [Figure 8] FIG. 11 is a block diagram showing a configuration of an information processing device according to an exemplary embodiment 3 of the present invention. [Figure 9] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device in each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Example 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0014] <Outline of information processing device> The information processing device 1 is a device that selects multiple options from a set of selectable options in round t (1≦t≦T; t, T is a natural number). In selecting options, the information processing device 1 selects options such that the predicted value of the reward in round t satisfies the constraint condition.

[0015] (Choices / Rewards) Specific examples of options do not limit this exemplary embodiment, but examples of options are content recommended on a web page, product prices, or product stock quantities. As an example, options are represented by a d-dimensional (d is a natural number equal to or greater than 1) feature vector. Specific examples of rewards do not limit this exemplary embodiment, but examples of rewards are whether or not a content option is selected (clicked, etc.) by a user, product sales, the reciprocal of the product inventory, or a value obtained by subtracting the product inventory quantity from a constant.

[0016] <Configuration of information processing device> Fig. 1 is a block diagram showing the configuration of an information processing device 1. As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, a selection unit 12, and a storage unit 13. The acquisition unit 11 is configured to realize an acquisition means in this exemplary embodiment. The selection unit 12 is configured to realize a selection means in this exemplary embodiment. The storage unit 13 is configured to realize a storage means in this exemplary embodiment.

[0017] (Acquisition Department) The acquisition unit 11 acquires a set of options {x t (i)}, where i ∈ [N], N is the number of options available in round t. [N] is the set of natural numbers from 1 to N. t (i)} may change with round t. For example, if the options are recommended content on a web page, selectable options, i.e., recommended content, may be added or removed as round t progresses. t Each element (option) included in {(i)} is a candidate for one of a plurality of options to be selected by the selection unit 12.

[0018] (Selection section / Storage section) The selection unit 12 selects a set {x t (i)}. The storage unit 13 stores data including the multiple options selected by the selection unit 12 and the observed values ​​of rewards obtained by the multiple options selected by the selection unit 12. The selection unit 12 calculates whether a predicted value of reward obtained by referring to the data stored by the storage unit 13 is included in a set {x t Baseline Option I selected from {(i)} t b A set of options selected by the selection unit 12 in round t is hereinafter referred to as “options I t " Option I t is the set {x t (i)} is a subset of

[0019] (Predicted reward) The predicted reward value in round t is calculated based on the data accumulated by the accumulation unit 13, that is, the options I t and a value obtained by referring to at least a portion of the observed rewards. t The predicted reward value of option I, which is a candidate of option I, is, for example, the sum of the predicted reward values ​​calculated for each option included in option I. t , and the set {x t (i)} is a subset of

[0020] (Baseline Options) Baseline Option I t b is the set {x t (i)}, that is, the set {x t (i)}, which is the set of options that guarantee a certain reward in round t. t b The reward obtained by does not have to be the maximum, but can be any reward that is not too low. Baseline Option I t b The selection method of the baseline option I is not limited, but may be, for example, a rule-based selection method or a selection method using a predictive model generated by machine learning. t b Although the specific example of Baseline Option I is not intended to limit the present exemplary embodiment, t b As an example, the value is a content having a predetermined attribute among the contents recommended on the Web page (for example, the content of a book), a list price of a product, or an average purchase amount.

[0021] For rule-based selection, baseline option I t b For example, the set {x t(i)}. However, the selection method based on the rule base is not limited to the above example, and other selection methods may be used. In the case of selection using a predictive model, the baseline option I t b As an example, the choice x t (i) is the input, and the choice is x t (i) The predicted reward value r t (i). However, the inputs and outputs of the prediction model are not limited to the above example, and the output may be the baseline option I t b Any method may be used as long as it can be used for the selection of the prediction model. The machine learning method of the prediction model is not limited, and as an example, a decision tree-based, linear regression, or neural network method may be used, or two or more of these methods may be used. Examples of the decision tree-based method include LightGBM (Light Gradient Boosting Machine), random forest, and XGBoost. Examples of the linear regression method include support vector regression, Ridge regression, Lasso regression, and ElasticNet. Examples of the neural network include deep learning.

[0022] (constraints) A constraint is a condition that limits the predicted value of reward. A constraint is a condition that limits the baseline option I t b The constraint condition is expressed, for example, by the following formula (1).

number

[0023] In equation (1), the predicted value r t (i) is the predicted reward obtained by choice i in round t. The parameter α is a real number α∈[0,1], i.e., 0 to 1. E is the operator for expectation.

[0024] More specifically, the constraints can be, for example, (i) the baseline choice I t b (ii) the sum of the confidence upper bounds of the rewards for each round t, t The sum of the confidence lower bounds of the rewards for at least a part of the choice set I that is a candidate of the round t is used for each round t. In this case, the constraint condition is expressed by the following formula (2), for example.

number

number

number

[0025] The constraints are not limited to the above examples and may be set to any of the following baseline options: t b Any constraint condition may be used as long as it is determined according to the above.

[0026] <Effects of information processing devices> According to the information processing device 1 according to this exemplary embodiment, a set of options selectable in a certain round t, {x t (i)}, and obtain the obtained set {x t(i)} and accumulates data including the selected options and the observed values ​​of rewards obtained by the selected options. In addition, the information processing device 1 calculates whether a predicted value of reward obtained by referring to the accumulated data in the selection of options is within a set {x t Baseline Option I selected from {(i)} t b Therefore, according to the information processing device 1 according to the present exemplary embodiment, it is possible to prevent the reward from becoming too low in a situation where the selectable options may change depending on the round.

[0027] <Information processing method flow> The flow of the information processing method S1 executed by the information processing device 1 configured as above will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. The information processing device 1 repeats the information processing method S1 to repeatedly select options. Note that a description of the contents that have already been described will be omitted. As shown in Fig. 2, the information processing method S1 includes steps S11 to S13.

[0028] (Step S11) In step S11, the acquisition unit 11 acquires a set of options {x t The acquisition unit 11 acquires information indicating the set {x (i)} via a communication interface of the information processing device 1, for example. t The information processing device 1 may acquire the set {x t The acquisition unit 11 may acquire information indicating the set {x t The information indicating the ID of the packet may be obtained by reading out the information indicating the ID of the packet (i).

[0029] (Step S12) In step S12, the selection unit 12 selects a set {x tIn step S12 (selection step), the selection unit 12 selects a plurality of options from the set {x t Baseline Option I selected from {(i)} t b A plurality of options are selected so as to satisfy the constraints determined according to the above.

[0030] (Step S13) In step S13, the accumulation unit 13 accumulates data including the multiple options selected in step S12 (selecting step) and the observed values ​​of rewards obtained from the multiple options selected in step S12 (selecting step).

[0031] <Effects of information processing methods> According to the information processing method S1 according to this exemplary embodiment, the information processing device 1 calculates a set of options {x t (i)}, and obtain the obtained set {x t (i)} and accumulates data including the selected options and the observed values ​​of rewards obtained by the selected options. t In the process of selecting, the predicted reward obtained by referring to the accumulated data is selected from the set {x t Baseline Option I selected from {(i)} t b Therefore, according to the information processing method S1 according to the present exemplary embodiment, it is possible to prevent the reward from becoming too low in a situation where the selectable options may change from round to round.

[0032] <Configuration of information processing system> Next, the configuration of the information processing system 100 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing system 100. As shown in Fig. 3, the information processing system 100 includes an information processing device 1 and a terminal device 2. Each component of the information processing device 1 has already been described, and therefore the description will not be repeated here.

[0033] 3, the terminal device 2 includes an option acquisition unit 21 and a reward information provision unit 22. The option acquisition unit 21 is configured to realize an option acquisition means in this exemplary embodiment. The reward information provision unit 22 is configured to realize a reward information provision means in this exemplary embodiment.

[0034] The option acquisition unit 21 acquires the multiple options selected by the selection unit 12. The reward information provision unit 22 provides the information processing device 1 with reward information indicating rewards obtained by performing actions indicated by the multiple options acquired by the option acquisition unit 21.

[0035] <Effects of information processing systems> According to the information processing system 100 according to this exemplary embodiment, a set of options available in a certain round t, {x t (i)}, and obtain the obtained set {x t (i)} and accumulates data including the selected options and the observed values ​​of rewards obtained by the selected options. t In the process of selecting, the predicted reward obtained by referring to the accumulated data is selected from the set {x t Baseline Option I selected from {(i)} t b Therefore, according to the information processing system 100 according to the present exemplary embodiment, it is possible to prevent the reward from becoming too low in a situation where the selectable options may change from round to round.

[0036] <Information processing method flow> The flow of the information processing method S100 executed by the information processing system 100 configured as above will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S100. By executing the information processing method S100, the information processing system 100 selects an option I t The selection of the above is repeated. Note that the contents that have already been described will not be described again.

[0037] As shown in Fig. 4, information processing method S100 includes steps S11 to S13 and S21 to S22 which are repeated. Here, in the reference numbers given to each step in Fig. 4, the degree of repetition is written as a sub-number following a hyphen "-". For example, S21-1 indicates the first repetition, and S21-2 indicates the second repetition. The same applies to the other steps.

[0038] (Step S11-1) In step S11-1, the acquisition unit 11 acquires a set of options {x t (i)} is obtained.

[0039] (Step S12-1) In step S12-1, the selection unit 12 selects a set {x t (i)}. At this time, in step S12-1 (selection step), the selection unit 12 selects a plurality of options from the set {x t Under constraints determined according to the baseline options selected from (i)), multiple options are selected so as to maximize the predicted reward obtained by referring to the accumulated data.

[0040] (Step S21-1) In step S21-1, the option acquisition unit 21 acquires the multiple options selected in step S12-1 (the selecting step). The option acquisition unit 21 acquires the multiple options via a communication interface or an input interface of the terminal device 2, for example.

[0041] (Step S22-1) In step S22-1, the reward information providing unit 22 provides reward information indicating rewards obtained by performing actions indicated by the multiple options to the information processing device 1. As an example, the reward information providing unit 22 provides the reward information by transmitting the reward information via a communication interface or an input interface of the terminal device 2.

[0042] (Step S13-1) In step S13-1, the accumulation unit 13 accumulates data including the multiple options selected in step S12-1 (the selecting step) and the observed values ​​of the rewards obtained from the multiple options selected in step S12-1 (the selecting step).

[0043] <Effects of information processing methods> According to the information processing method S100 according to this exemplary embodiment, a set of options available in a certain round t, {x t (i)}, and obtain the obtained set {x t (i)}, and accumulates data including the selected options and the observed rewards obtained by the selected options. In addition, in the step of selecting the options, the information processing device 1 calculates a predicted reward value obtained by referring to the accumulated data based on a set {x t Baseline Option I selected from {(i)} t b Therefore, according to the information processing method S100 according to the present exemplary embodiment, it is possible to prevent the reward from becoming too low in a situation where the selectable options may change from round to round.

[0044] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals and will not be described repeatedly.

[0045] <Configuration of Information Processing System 100A> The configuration of an information processing system 100A according to this exemplary embodiment will be described with reference to FIG. 5. FIG. 5 is a block diagram showing the configuration of the information processing system 100A. As shown in FIG. 5, the information processing system 100A includes an information processing device 1A and a terminal device 2A. Also, as shown in FIG. 5, the information processing device 1A and the terminal device 2A are configured to be able to communicate with each other via a network N. Here, the specific configuration of the network N does not limit this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0046] <Configuration of information processing device 1A> The configuration of the information processing device 1A according to this exemplary embodiment will be described with reference to FIG. 5. FIG. 5 is a block diagram showing the configuration of the information processing device 1A. As shown in FIG. 5, the information processing device 1A includes a control unit 10A, a storage unit 17A, and a communication unit 19A. The communication unit 19A communicates with devices external to the information processing device 1A. As an example, the communication unit 19A communicates with a terminal device 2A. The communication unit 19A transmits data supplied from the control unit 10A to the terminal device 2A, and supplies data received from the terminal device 2A to the control unit 10A.

[0047] (Control unit 10A) As shown in FIG. 5, the control unit 10A includes an acquisition unit 11, a selection unit 12, a storage unit 13, a baseline selection unit 14A, and a reward prediction unit 15A. The acquisition unit 11 is configured to realize an acquisition means in this exemplary embodiment. The selection unit 12 is configured to realize a selection means in this exemplary embodiment. The storage unit 13 is configured to realize a storage means in this exemplary embodiment. The baseline selection unit 14A is configured to realize a baseline selection means in this exemplary embodiment. The reward prediction unit 15A is configured to realize a reward prediction means in this exemplary embodiment.

[0048] The acquisition unit 11 acquires a set of options {x t Although specific examples of the options do not limit the present exemplary embodiment, the options are, for example, content to be recommended on a web page, the price of a product, or the stock amount of a product, as in the first exemplary embodiment.

[0049] The selection unit 12 selects a set {x t The selection unit 12 selects multiple options from the set {x t Baseline Option I selected from {(i)} t b A plurality of options are selected so as to satisfy a constraint condition determined according to the above. Although a specific example of the reward does not limit the present exemplary embodiment, the reward may be, for example, whether or not a content option is selected (clicked, etc.) by a user, sales of a product, the reciprocal of the amount of stock of the product, or a value obtained by subtracting the amount of stock of the product from a constant.

[0050] The accumulation unit 13 accumulates data including the multiple options selected by the selection unit 12 and the observed values ​​of rewards obtained from the multiple options selected by the selection unit 12, as in the first exemplary embodiment.

[0051] The baseline selection unit 14A selects a set {x t (i)} to Baseline Option I t b The baseline selection unit 14A selects a baseline option I based on a rule, for example. t b Alternatively, a predictive model generated by machine learning may be used to select baseline option I t b may be selected.

[0052] Reward prediction unit 15A calculates a predicted value of reward by referring to the data stored in storage unit 13. The predicted value of reward for option i∈I includes, as an example, at least one of an upper confidence limit and a lower confidence limit of the reward for option i∈I.

[0053] The storage unit 17A stores various types of data referenced by the control unit 10A. As an example, the storage unit 17A stores option information II and an observation value RI of a reward, as shown in Fig. 5. The various types of data stored in the storage unit 17A will be described later with reference to different drawings.

[0054] <Configuration of Terminal Device 2A> 5, the terminal device 2A includes a control unit 20A, a display unit 24A, an input receiving unit 28A, and a communication unit 29A. As an example, the terminal device 2A can be specifically realized as a user terminal used by an end user who browses web pages, a checkout terminal installed in a store, an inventory management terminal installed in a warehouse, or the like, but this is not intended to limit the present exemplary embodiment.

[0055] The communication unit 29A communicates with devices external to the terminal device 2A. As an example, the communication unit 29A communicates with the information processing device 1A. The communication unit 29A transmits data supplied from the control unit 20A to the information processing device 1A, and supplies data received from the information processing device 1A to the control unit 20A. The display unit 24A displays various information under the control of the control unit 20A. As an example, the display unit 24A includes a display panel and a drive circuit that drives the display panel.

[0056] 5, the control unit 20A includes an option acquisition unit 21, a reward information provision unit 22, and an action execution unit 23A. As in the exemplary embodiment 1, the option acquisition unit 21 acquires a plurality of options selected by the selection unit 12. The reward information provision unit 22 provides the information processing device 1 with reward information indicating rewards obtained by executing actions indicated by the plurality of options.

[0057] The action execution unit 23A executes an action based on a plurality of options selected by the information processing device 1. As an example, when the options selected by the information processing device 1 are "Product P1", "Product P2", and "Product P3", the action execution unit 23A displays information recommending "Product P1", "Product P2", and "Product P3" on a Web page on the display unit 24A. As another example, when the option selected by the information processing device 1 is "Set the price of the product to a certain value", the action execution unit 23A sets the price associated with the product to that value. As another example, when the action selected by the information processing device 1 is "Set the purchase quantity of the product to a certain value", the action execution unit 23A sets the purchase quantity associated with the product to that value.

[0058] The input reception unit 28A receives various inputs to the terminal device 2A. Although the specific configuration of the input reception unit 28A does not limit this exemplary embodiment, as an example, the input reception unit 28A includes an input device such as a keyboard and a touch pad. The input reception unit 28A may also include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves. The input reception unit 28A acquires various information via the above-mentioned input device and data scanner, and supplies the acquired information to the control unit 20A. Here, the information acquired by the input reception unit 28A may include information indicating that a user has clicked on content, or an observation value of a reward such as "sales" or "information related to inventory" regarding a product. In other words, the input reception unit 28A is an example of an input device that accepts an observation value of a reward.

[0059] (Example of data stored in the memory unit) As shown in FIG. 5, the storage unit 17A stores option information II and reward observation value RI. The option information II is the option I selected by the selection unit 12 in round t. t The reward observation value RI is information that represents the option I selected by the selection unit 12 in round t. tThe reward observation value RI is, for example, acquired by the acquisition unit 11 and stored in the storage unit 17A. The observation value is, for example, a numerical value indicating whether or not the user selected the recommended content. For example, if the user did not select a content in round t, the observation value r t (i) is "0". On the other hand, if a user selects a content in round t, the observed value r t (i) is "1". However, the observed value r t (i) is not limited to the above example, and may be other values. For example, the observed value r t (i) may be a numerical value indicating the sales amount of a product, the reciprocal of the number of products in stock, a value obtained by subtracting the number of products in stock from a constant, etc.

[0060] <Flow of information processing method by information processing device> The flow of the information processing method S1A executed by the information processing device 1A configured as above will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing the flow of the information processing method S1A. By executing the information processing method S1A, the information processing device 1A selects options I such that the predicted value of the reward in round t satisfies the constraint condition. t The round t is, for example, the number of times the content is estimated in a web page. In this operation example, the reward observation value r t (i) is r t (i)∈[0,1].

[0061] In the information processing method S1A shown in FIG. 6, the information processing device 1A selects an option x t (i), matrix V t , vector b t , parameter λ, and parameter β are used to perform the calculation. t (i) is a d-dimensional feature vector (d is a natural number greater than or equal to 1). t is a matrix with d rows and d columns. Vector b tis a d-dimensional vector. The parameters λ and β are expressed, for example, by the following formula (3).

number

[0062] (Step S101) In step S101, the selection unit 12 initializes various parameters. As an example, the selection unit 12 sets λI to the matrix V0 and sets a zero vector to the vector b0, as shown in equation (4). Here, I in equation (4) is a unit matrix.

number

[0063] (Step S102) Step S102 is the start point of the loop process for round t. Here, the loop variable in the loop process for the relevant round is t=1, 2, …, T It is.

[0064] (Step S103) In step S103 in the loop, the acquisition unit 11 acquires a selectable option x t (i) set {x t The acquisition unit 11 acquires information indicating the set {x (i)} from another device (for example, a predetermined cloud server) via the communication unit 19A, for example. t (i)} may be obtained from the storage unit 17A. t The information indicating the ID of the packet may be obtained by reading out the information indicating the ID of the packet (i).

[0065] (Step S104) Next, in step S104 in the loop, the acquisition unit 11 acquires the option I selected by the selection unit 12 in round t. t A family S of choice sets I that are candidates for tAs an example, when the information processing system 100A is a system for recommending contents of a Web page and recommending three items from ten options, the acquisition unit 11 identifies the option I t A set of options I, i.e., the set {x t (i)} contains three options x t A family S of choice set I that represents the combination of (i) t Identify.

[0066] (Step S105) In step S105, the baseline selection unit 14A selects a set {x t (i)} to Baseline Option I t b The baseline selection unit 14A selects a baseline option I based on a rule, for example. t b Alternatively, a predictive model generated by machine learning may be used to select baseline option I t b may be selected.

[0067] Rule-based baseline option I t b When selecting a baseline, the baseline selection unit 14A selects a baseline option I that satisfies a predetermined condition from among a plurality of option sets I, for example. t b The predetermined condition is, for example, that some or all of the options included in the option set I have a predetermined attribute. The predetermined attribute is, for example, an attribute corresponding to an end user who uses a Web service. More specifically, for example, the baseline selection unit 14A selects a set of options {x t Among the options (i), we define the option set I that contains options recommending book content at a ratio equal to or greater than the threshold as the baseline option set I. t b may be selected as

[0068] In addition, the baseline selection unit 14A selects the baseline option I using a prediction model constructed by machine learning. t b In this case, the input of the prediction model may be, for example, the option x t (i) The output of the predictive model is, for example, option x t (i) The predicted reward value r t In this case, the baseline selection unit 14A may, for example, add an option x t (i) Input the reward prediction r t Based on (i), for each option x t Score (i) and select baseline option I so that the sum of the scores is maximized. t b However, baseline option I t b The method of selecting the baseline option I is not limited to the above example, and the baseline selection unit 14A may select the baseline option I by other methods. t b may be selected.

[0069] (Step S106) In step S106, the reward prediction unit 15A calculates a predicted value of the reward in round t by referring to the data accumulated by the accumulation unit 13. As an example, the reward prediction unit 15A calculates the upper confidence limit and the lower confidence limit of the option in round t as the predicted value of the reward. In this case, as an example, the reward prediction unit 15A calculates the upper confidence limit in round t by the following formula (5) and calculates the lower confidence limit in round t by the following formula (6).

number

number

[0070] In the formulas (5) and (6),

number

number

number

number

number

[0071] (Step S107) In step S107, the selection unit 12 determines whether the predicted reward value obtained by referring to the data stored in the storage unit 13 is the baseline option I t b Option I is chosen to satisfy the constraints determined according to t In other words, the selection unit 12 selects an option I for each round t such that the predicted value of the reward satisfies the constraint. t The constraint condition is expressed by the above formula (1) as an example. In formula (1), when the parameter α=0.5, the option I is selected based on the constraint expressed by formula (1). t By selecting the baseline option I t b In formula (1), the expected value E[r t (i)] is, for example, a matrix θ∈R that represents the linear contribution of choices to rewards. d Using

number

[0072] More specifically, the constraints may be, for example, (i) the baseline option I t b (ii) the sum of the confidence upper bounds of the rewards by the selection unit 12 in each round, t The sum of the confidence lower bounds of the rewards for at least a part of the choice set I that is a candidate for each round is used. In this case, the constraint condition is expressed by the above formula (2) as an example.

[0073] The selection unit 12 also selects the option I so that the predicted value of the reward is larger under the constraints. t In this case, the selection unit 12 may select, for example, the option i ∈ I t The predicted reward value r t (i) Sum of

number

[0074] More specifically, when the constraint condition is expressed by the above-mentioned formula (2), the selection unit 12 selects the option I by the following formula (9), for example: t Select .

number

[0075] (Step S108) In step S108, the selection unit 12 selects the selected option I t is provided to the terminal device 2A by transmitting it via the communication unit 19A.

[0076] The option acquisition unit 21 acquires the option I from the information processing device 1A. t The action execution unit 23A acquires the option I acquired by the option acquisition unit 21. tAs an example, when the options selected by the information processing device 1 are "Product P1", "Product P2", and "Product P3", the action execution unit 23A displays information recommending "Product P1", "Product P2", and "Product P3" on the Web page on the display unit 24A. In addition, the reward information provision unit 22 provides reward information indicating the observed value of the reward obtained by the action of the action execution unit 23A to the information processing device 1A by transmitting the reward information to the information processing device 1A via the communication unit 29A.

[0077] (Step S109) In step S109, the acquisition unit 11 acquires the observed value of the reward, which is the reward information provided by the terminal device 2A, via the communication unit 19A. In addition, the acquisition unit 11 updates the matrix Vt and the vector bt according to the following equations.

number

number

[0078] (Step S110) In step S110, the storage unit 13 stores the option I selected by the selection unit 12. t And, Option I t The data including the reward observation value obtained by the above is stored in the memory unit 17A.

[0079] (Step S111) Step S111 is the end of the loop process for the round.

[0080] <Example of display by information processing system> In this exemplary embodiment, the information processing system 100A may output information including the option It selected by the selection unit 12 to an output device such as the display unit 24A of the terminal device 2A.

[0081] FIG. 7 is a diagram showing a display example by the information processing system 100A. The example shown in FIG. 7 is a display example in which (the total amount of) sales of multiple products is used as the predicted value of the reward, and one round is set to one day. That is, in the example shown in FIG. 7, the selection unit 12 of the information processing device 1B selects an option I on a certain day (round t). t is selected so as to satisfy the constraints determined according to the baseline choice Itb.

[0082] Then, as shown in Fig. 7, the display unit 24A of the terminal device 2A displays each observed value of the reward (sales amount in Fig. 7) for each round (day in Fig. 7). Furthermore, in the example shown in Fig. 7, the display unit 24A of the terminal device 2A displays information (prices of products A to C) related to the option selected in round t.

[0083] By displaying the above information, the information processing system 100A can provide the user with the sales amount (observed value of reward) and the price of the product (option I t ) can be presented.

[0084] <Examples of application of information processing systems> (Web Marketing) An example will be described in which the information processing system 1001A according to this exemplary embodiment is applied to advertising behavior (marketing measures) in an operating company of a certain electronic commerce site. In this case, advertising (online (banner) advertising, advertising by e-mail, direct mail, sending of discount coupons by e-mail, etc.) for a plurality of customers for an object such as a product or service that the operating company intends to sell is set as the behavioral measure X (option I). In other words, in this application example, the selectable options include advertising for the object. For example, when the behavioral measure X=[1, 0, 2, . . .], the first element indicates banner advertising for customer A, the second element indicates no advertising for customer B, and the third element indicates sending of discount coupons by e-mail to customer C. The objective function takes the behavioral measure X as input, and outputs the result of applying the behavioral measure X to the advertising behavior for each customer (predicted value of reward). The reward includes at least one of information indicating whether or not an advertisement was selected on a web page (whether or not a banner advertisement was clicked), the sales amount of the object, the expected value of the sales amount, and the purchase probability.

[0085] In this application example, by applying the information processing method according to this exemplary embodiment, it is possible to derive suitable advertising behavior for each customer in the operating company.

[0086] (Dynamic Pricing) An example in which the information processing system 1001A according to this exemplary embodiment is applied to determining the price of a product will be described. In this case, the discount on the price of beer from each company at a certain store is set as the behavioral measure (option). For example, when the behavioral measure X=[0, 2, 1, . . .], the first element indicates that the price of beer from company A is the regular price, the second element indicates that the price of beer from company B is increased by 10% from the regular price, and the third element indicates that the price of beer from company C is discounted by 10% from the regular price. The objective function takes the execution measure X as input, and outputs the result of applying the execution measure X to the price of beer from each company each month (predicted value of remuneration). In other words, in this application example, the selectable options include the price setting of the object, and the remuneration includes information indicating the sales result of the object. The sales result includes, for example, at least one of the sales amount of the product, the expected value of the sales amount, and the purchase probability.

[0087] In this case, by applying the information processing method according to this exemplary embodiment, it is possible to derive optimal pricing for the beer prices of each company at the store.

[0088] (Investment Portfolio) An example in which the information processing system 100A according to this exemplary embodiment is applied to an investor's investment behavior will be described. In this case, the investment (purchase, capital increase), sale, and holding of multiple financial products (stock brands, etc.) that the investor holds or intends to hold are defined as action measures X (options I). For example, when the action measure X=[1, 0, 2, . . .], the first element indicates additional investment in the stock of company A, the second element indicates holding (neither purchase nor sale) the bond of company B, and the third element indicates selling the stock of company C. The objective function takes the execution measure X (options) as input, and outputs the result of applying the execution measure X to the behavior for each financial product (predicted value of remuneration). In this case, the predicted value of remuneration is, for example, the expected value of the market value of the financial product. In other words, in this application example, the selectable options include investment behavior for financial products, and the remuneration includes information representing the market value of the financial products.

[0089] In this case, by applying the information processing method according to this exemplary embodiment, it is possible to derive suitable investment behavior for each financial product by the investor.

[0090] (Clinical Trials) An example will be described in which the information processing system 1001A according to this exemplary embodiment is applied to medication behavior for a clinical trial of a certain drug in a pharmaceutical company. In this case, the dosage of medication to a plurality of subjects and the presence or absence of medication are taken as execution measures X (options I). For example, when the behavioral measures X=[1, 0, 2, . . .], the first element indicates that dosage 1 is administered to subject A, the second element indicates that medication is not administered to subject B, and the third element indicates that dosage 2 is administered to subject C. The objective function takes the behavioral measures X (options) as input, and outputs the result (predicted value of reward) of applying the execution measures X to the medication behavior for each subject. The predicted value of reward is, for example, information on the occurrence of side effects (for example, the incidence rate of side effects). In this case, by applying the information processing method according to this exemplary embodiment, a medication behavior that reduces the incidence rate of side effects can be derived.

[0091] <Effects of Exemplary Embodiment 2> As described above, the information processing device 1A according to this exemplary embodiment selects the baseline option I t b Option I is chosen to satisfy the constraints determined according to t Select Baseline Option I t b is a set of options that can change with round t. In this exemplary embodiment, option I t In this manner, according to the present exemplary embodiment, for each round t, the baseline option I t b Based on the constraints determined according to Option I t By selecting , it is possible to guarantee a certain amount of reward in each round t, even in a situation where the selectable options may change from round t to round t.

[0092] Also, according to this exemplary embodiment, Option I tSince there is no need to refer to the rewards observed in past rounds in selecting t This reduces the search space for the selection of , and improves computational efficiency.

[0093] In addition, the information processing device 1A according to the present exemplary embodiment selects the baseline option I t b Under the constraints determined by t Therefore, according to the information processing device 1A according to this exemplary embodiment, in a situation where selectable options may change for each round t, it is possible to prevent the reward for each round t from becoming too low, and to increase the reward obtained.

[0094] Moreover, the selection unit 12 of the information processing device 1A according to this exemplary embodiment selects a set {x t (i)} to Baseline Option I t b and a reward prediction unit 15A that calculates a predicted value of reward by referring to the data stored in the storage unit 13. By selecting an option based on a constraint condition determined according to the baseline option selected by the information processing device 1A, it is possible to prevent the reward obtained from being too low.

[0095] In addition, the information processing device 1A according to the present exemplary embodiment acquires information indicating a set of selectable options for each round, and selects the options I for each round so that the predicted value of the reward satisfies the constraint condition. t By obtaining a set of selectable options for each round, it is possible to prevent the reward obtained in each of multiple rounds from becoming too low.

[0096] Also, in this exemplary embodiment, the baseline option I t bBy using constraints expressed for each round, including at least the sum of the upper confidence limits of the rewards from I and the sum of the lower confidence limits of the rewards from at least a portion of the choice set I for each round, it is possible to prevent the rewards obtained in each of the multiple rounds from being too low.

[0097] Exemplary embodiment 3 The third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and the description thereof will not be repeated.

[0098] <Configuration of information processing device 1B> The configuration of an information processing device 1B according to this exemplary embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the information processing device 1B.

[0099] 8, the information processing device 1B includes a display unit 18B and an input receiving unit 16B in addition to the components included in the information processing device 1A according to the exemplary embodiment. The display unit 18B is configured to be able to display various data to be processed by the information processing device 1B. The display content of the display unit 18B is controlled by the control unit 10A. As an example, the display unit 18B includes a display panel and a drive circuit that drives the display panel.

[0100] The input reception unit 16B receives various inputs to the information processing device 1B. Although the specific configuration of the input reception unit 16B does not limit this exemplary embodiment, as an example, the input reception unit 16B may include an input device such as a keyboard and a touchpad. The input reception unit 16B may also include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves. The input reception unit 16B acquires the observation value of the above-mentioned reward via the above-mentioned input device, data scanner, etc., and supplies the acquired information to the control unit 10A.

[0101] In the above-described exemplary embodiment 2, the selection unit 12 of the information processing device 10A selects the option It In this exemplary embodiment, the selection unit 12 of the information processing device 1B provides the selected option I by transmitting the selected option I to the terminal device 2A (step S108 in FIG. 6). t The selection unit 12 outputs information including the selected option I to an output device such as the display unit 18B. As an example, the selection unit 12 displays the screen example illustrated in FIG. 7 on the display unit 18B. t may perform actions based on

[0102] In the above-described exemplary embodiment 2, the acquisition unit 11 of the information processing device 12A acquires the observed value of the reward by receiving it from the terminal device 2A (step S109 in FIG. 6), whereas in this exemplary embodiment, the acquisition unit 11 of the information processing device 10B acquires, as an example, the observed value of the reward accepted by the input acceptance unit 16B. Also, as an example, the acquisition unit 11 may acquire the observed value of the reward received via the communication unit 19A.

[0103] In this exemplary embodiment, similarly to the above-described exemplary embodiment 2, the information processing device 1B displays the display as shown in FIG. 7 to provide the user with the sales amount (observed value of remuneration) and the price of the product (option I t ) can be presented.

[0104] [Software implementation example] Some or all of the functions of the information processing devices 1, 1A, and 1B may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0105] In the latter case, the information processing devices 1, 1A, and 1B are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 9. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1, 1A, and 1B. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1, 1A, and 1B.

[0106] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0107] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0108] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0109] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0110] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0111] (Appendix 1) An acquisition means for acquiring information indicating a set of options selectable in a certain round; A selection means for selecting a plurality of options from the set; A storage means for storing data including a plurality of options selected by the selection means and observed values ​​of rewards obtained by the plurality of options selected by the selection means; Equipped with the selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set. 23. An information processing apparatus comprising:

[0112] According to the above configuration, in a situation where the selectable options may change from round to round, it is possible to prevent the reward obtained from becoming too low.

[0113] (Appendix 2) The selection means selects the plurality of options so as to maximize the predicted value of the reward under the constraint condition. 2. The information processing device according to claim 1 .

[0114] According to the above configuration, in a situation where the selectable options may change from round to round, it is possible to prevent the reward for each round from becoming too low, while making it possible to increase the reward that can be obtained.

[0115] (Appendix 3) The selection means is a baseline selection means for selecting said baseline option from said collection; a reward prediction means for calculating a predicted value of the reward by referring to the data stored by the storage means; Equipped with 3. The information processing device according to claim 1 or 2.

[0116] According to the above configuration, by selecting an option based on a constraint condition determined according to the selected baseline option, it is possible to prevent the reward obtained from becoming too low.

[0117] (Appendix 4) the acquiring means acquires the information indicating a set of selectable options for each round; The selection means selects the plurality of options for each round such that the predicted value of the reward satisfies the constraint condition. 4. The information processing device according to claim 1,

[0118] According to the above configuration, by obtaining a set of selectable options for each round, it is possible to prevent the reward obtained in each of the multiple rounds from becoming too low.

[0119] (Appendix 5) The constraint is: The sum of the confidence limits of the rewards of the baseline choices per round; and A sum of lower confidence limits of rewards for at least a portion of the candidates of the plurality of options selected by the selection means for each round; is expressed for each round using at least 5. The information processing device according to claim 4.

[0120] According to the above configuration, by using at least the sum of the upper confidence limits of the rewards from the baseline options for each round and the sum of the lower confidence limits of the rewards from at least some of the multiple option candidates selected by the selection means for each round, it is possible to prevent the rewards obtained in each of the multiple rounds from being too low by using constraints expressed for each round.

[0121] (Appendix 6) and an input device for receiving the reward observation. 6. The information processing device according to claim 1,

[0122] According to the above configuration, by selecting an option so that the predicted value of the reward obtained by referring to the observed value of the reward received by the input device satisfies the constraint condition, it is possible to prevent the reward obtained from being too low in a situation where the selectable options may change from round to round.

[0123] (Appendix 7) An information processing device, obtaining information indicative of a set of options available for selection in a given round; selecting a plurality of options from the set; and accumulating data including the multiple options selected in the selecting step and observed values ​​of rewards obtained by the multiple options selected in the selecting step; Repeatedly includes In the selecting step, the plurality of options are selected such that a predicted value of reward obtained by referring to the accumulated data satisfies a constraint condition determined according to a baseline option selected from the set. 23. An information processing method comprising:

[0124] According to the above method, the same effects as those of the above-mentioned information processing device can be achieved.

[0125] (Appendix 8) A program for causing a computer to function as an information processing device, The program causes the computer to An acquisition means for acquiring information indicating a set of options selectable in a certain round; A selection means for selecting a plurality of options from the set; a storage means for storing data including a plurality of options selected by the selection means and observed values ​​of rewards obtained by the plurality of options selected by the selection means; the selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set. A program characterized by:

[0126] According to the above program, the same effects as those of the above information processing device can be achieved.

[0127] (Appendix 9) An information processing system including an information processing device and a terminal device, The information processing device includes: An acquisition means for acquiring information indicating a set of options selectable in a certain round; A selection means for selecting a plurality of options from the set; A storage means for storing data including a plurality of options selected by the selection means and observed values ​​of rewards obtained from the plurality of options selected by the selection means; Equipped with the selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set; The terminal device an option acquisition means for acquiring the plurality of options selected by the selection means; a reward information providing means for providing, to the information processing device, reward information indicating a reward obtained by performing the actions indicated by the plurality of options; An information processing system comprising:

[0128] According to the above information processing system, the same effects as those of the above information processing device are achieved.

[0129] (Appendix 10) An information processing device, obtaining information indicative of a set of options available for selection in a given round; selecting a plurality of options from the set; and accumulating data including the multiple options selected in the selecting step and observed values ​​of rewards obtained by the multiple options selected in the selecting step; Repeatedly includes In the selecting step, the plurality of options are selected under a constraint condition determined according to a baseline option selected from the set so that a predicted value of reward obtained by referring to the accumulated data is larger; The terminal device acquiring the plurality of options selected in the selecting step; and providing reward information indicating a reward obtained by performing the actions indicated by the plurality of options to the information processing device; 13. An information processing method comprising:

[0130] According to the above information processing method, the same effects as those of the above information processing device are achieved.

[0131] (Appendix 11) the selectable options include advertisements for the subject matter; The reward includes at least one of information indicating whether the advertisement was selected on the web page, the sales amount of the object, an expected value of the sales amount, and a purchase probability. 7. An information processing device according to any one of claims 1 to 6.

[0132] (Appendix 12) the selectable options include pricing for the object; The reward includes information indicating the sales result of the object. An information processing device according to any one of Supplementary Notes 1 to 6 and Supplementary Note 11.

[0133] (Appendix 13) the selectable options include investment actions with respect to financial products; The compensation includes information representative of the market value of the financial instrument. 13. An information processing device according to any one of appendixes 1 to 6 and 11 to 12.

[0134] (Appendix 14) the selectable options include administering a medication to the subject; The reward includes information regarding the occurrence of side effects. 14. An information processing device according to any one of appendixes 1 to 6 and 11 to 13.

[0135] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0136] An information processing device having at least one processor, the processor executing an acquisition process for acquiring information indicating a set of options selectable in a certain round, a selection process for selecting a plurality of options from the set, and an accumulation process for accumulating data including the plurality of options selected in the selection process and observed values ​​of rewards obtained by the plurality of options selected in the selection process, wherein the plurality of options are selected in the selection process such that a predicted value of reward obtained by referring to the data accumulated in the accumulation process satisfies a constraint condition determined according to a baseline option selected from the set.

[0137] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process, the selection process, and the accumulation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0138] 1, 1A, 1B Information processing device 11 Acquisition Department 12 Selection section 13 Storage section 14A Baseline Selection Section 15A Reward Prediction Section 16B Input reception section 17A Storage section 18B Display section 2,2A Terminal Equipment 21 Choice Acquisition Section 22 Compensation Information Department 23A Action Execution Department 100,100A Information Processing System

Claims

1. An acquisition means for acquiring information indicating a set of options selectable in a certain round; A selection means for selecting a plurality of options from the set; A storage means for storing data including a plurality of options selected by the selection means and observed values ​​of rewards obtained from the plurality of options selected by the selection means; Equipped with The selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set.

23. An information processing apparatus comprising:

2. The selection means selects the plurality of options so as to maximize the predicted value of the reward under the constraint condition.

2. The information processing apparatus according to claim 1,

3. The selection means is a baseline selection means for selecting said baseline option from said collection; a reward prediction means for calculating a predicted value of the reward by referring to the data stored by the storage means; Equipped with 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the acquiring means acquires the information indicating a set of selectable options for each round; The selection means selects the plurality of options for each round such that the predicted value of the reward satisfies the constraint condition.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises: a first input section;

5. The constraint is: The sum of the confidence limits of the rewards of the baseline choices per round; and A sum of lower confidence limits of rewards for at least a portion of the candidates of the plurality of options selected by the selection means for each round. is expressed for each round using at least 5. The information processing apparatus according to claim 4.

6. and an input device for receiving the reward observation.

6. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

7. An information processing device, obtaining information indicative of a set of options available for selection in a given round; selecting a plurality of options from the set; and accumulating data including the multiple options selected in the selecting step and observed values ​​of rewards obtained from the multiple options selected in the selecting step; Repeatedly includes In the selecting step, the plurality of options are selected such that a predicted value of reward obtained by referring to the accumulated data satisfies a constraint condition determined according to a baseline option selected from the set.

23. An information processing method comprising:

8. A program for causing a computer to function as an information processing device, The program causes the computer to An acquisition means for acquiring information indicating a set of options selectable in a certain round; A selection means for selecting a plurality of options from the set; a storage means for storing data including a plurality of options selected by the selection means and observed values ​​of rewards obtained by the plurality of options selected by the selection means; The selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set. A program characterized by:

9. An information processing system including an information processing device and a terminal device, The information processing device includes: An acquisition means for acquiring information indicating a set of options selectable in a certain round; A selection means for selecting a plurality of options from the set; A storage means for storing data including a plurality of options selected by the selection means and observed values ​​of rewards obtained from the plurality of options selected by the selection means; Equipped with the selection means selects the plurality of options such that a predicted value of reward obtained by referring to the data stored by the storage means satisfies a constraint condition determined according to a baseline option selected from the set; The terminal device an option acquisition means for acquiring the plurality of options selected by the selection means; a reward information providing means for providing, to the information processing device, reward information indicating a reward obtained by performing the actions indicated by the plurality of options; An information processing system comprising:

10. An information processing device, obtaining information indicative of a set of options available for selection in a given round; selecting a plurality of options from the set; and accumulating data including the multiple options selected in the selecting step and observed values ​​of rewards obtained from the multiple options selected in the selecting step; Repeatedly includes In the selecting step, the plurality of options are selected under a constraint condition determined according to a baseline option selected from the set so that a predicted value of reward obtained by referring to the accumulated data is larger; The terminal device obtaining the plurality of options selected in the selecting step; and providing reward information indicating a reward obtained by performing the actions indicated by the plurality of options to the information processing device; 13. An information processing method comprising:

Images