Information processing device, information processing method, information processing system, and program

By weighting learning data based on reliability to predict the reward sum, the information processing device enhances action determination in reinforcement learning, addressing the issue of unequal treatment of reliable and unreliable data in existing techniques.

JP7586311B2Active Publication Date: 2024-11-19NEC CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023523808
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-05-26
Publication Date
2024-11-19
Estimated Expiration
2041-05-26

AI Technical Summary

Technical Problem

Existing reinforcement learning techniques, such as the Upper-Confidence Bounds (UCB) algorithm, do not adequately differentiate between highly reliable and low reliable learning data, leading to suboptimal action determination.

Method used

An information processing device and method that weights learning data based on reliability to calculate a function predicting the reward sum, allowing for more suitable action determination by incorporating a first function that accounts for data variation and reliability.

Benefits of technology

Enables the determination of more suitable actions by accurately predicting the reward sum, improving the efficiency and effectiveness of reinforcement learning processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586311000041
    Figure 0007586311000041
  • Figure 0007586311000042
    Figure 0007586311000042
  • Figure 0007586311000043
    Figure 0007586311000043
Patent Text Reader

Abstract

To determine a better action, an information processing device (1) comprises an acquisition unit (11) that acquires a state, a determination unit (12) that refers to the state to determine an action, and an accumulation unit (13) that accumulates learning data including the state and a reward obtained through the action determined by the determination unit. The determination unit calculates a first function for predicting a sum of rewards from the state and the action by weighting the learning data, a determines an action using the first function.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing system, and a program for determining an action. [Background technology]

[0002] There is a known technique for sequentially determining an action that maximizes the total reward while observing the reward in a state in which the relationship between the action and the reward is unknown. For example, Non-Patent Document 1 discloses a technique using the so-called Upper-Confidence Bounds (UCB) algorithm as an example of such a technique. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Chi Jin et.al. “Provably Efficient Reinforcement Learning with Linear Function Approximation” arXiv:1907.05388v2 [cs.LG], Aug 8, 2019 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Non-Patent Document 1 has room for improvement in terms of determining more suitable actions. Generally, learning data referred to in determining actions may contain a mixture of highly reliable data and low reliable data, but the technology in Non-Patent Document 1 treats these data on the same level.

[0005] One aspect of the present invention has been made in consideration of the above problems, and one example of a purpose of the present invention is to provide a technique capable of determining a more suitable action. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention includes an acquisition means for acquiring a state, a decision means for determining an action by referring to the state, and a storage means for storing learning data including the state and a reward obtained by the action determined by the decision means, wherein the decision means calculates a first function that predicts a reward sum from the state and the action by weighting the learning data, and determines an action using the first function.

[0007] An information processing method according to one embodiment of the present invention includes an information processing device repeatedly acquiring a state, determining an action by referring to the state, and accumulating learning data including the state and a reward obtained by the determined action, and in the step of determining the action, the action is determined using a first function that predicts a reward sum from the state and the action, the first function being calculated by weighting the learning data.

[0008] A program according to one embodiment of the present invention is a program that causes a computer to function as an information processing device, the program causing the computer to function as an acquisition means for acquiring a state, a decision means for determining an action by referring to the state, and a storage means for storing learning data including the state and a reward obtained by the action determined by the decision means, the decision means calculating a first function that predicts the sum of rewards from the state and the action by weighting the learning data, and determining an action using the first function.

[0009] An information processing system according to one embodiment of the present invention is an information processing system including an information processing device and a terminal device, wherein the information processing device comprises an acquisition means for acquiring a state, a decision means for determining an action by referring to the state, and a storage means for storing learning data including the state and a reward obtained by the action determined by the decision means, wherein the decision means calculates a first function that predicts a reward sum from the state and the action by weighting the learning data and determines an action using the first function, and the terminal device comprises a state information providing means for acquiring a state and providing it to the information processing device, and a reward information providing means for providing reward information indicating the reward obtained by performing the action determined by the information processing device.

[0010] An information processing method according to one aspect of the present invention includes: The process includes repeatedly determining an action by referring to the state and accumulating learning data including the state and a reward obtained by the determined action, and in the process of determining the action, the action is determined using a first function that predicts a reward sum from the state and the action, the first function being calculated by weighting the learning data, and the terminal device acquires the state and provides it to the information processing device, and provides reward information indicating the reward obtained by the information processing device performing the determined action to the information processing device. Effect of the Invention

[0011] According to one aspect of the present invention, a more suitable action can be determined. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Diagram 2] 1 is a flowchart showing a flow of an information processing method executed by an information processing device according to a first exemplary embodiment of the present invention. [Diagram 3]1 is a block diagram showing a configuration of an information processing system according to a first exemplary embodiment of the present invention. [Figure 4] 1 is a flow chart showing the flow of an information processing method executed by an information processing system according to an exemplary embodiment 1 of the present invention. [Diagram 5] FIG. 11 is a block diagram showing a configuration of an information processing system according to a second exemplary embodiment of the present invention. [Figure 6] FIG. 11 is a diagram showing an example of various data stored in a storage unit of an information processing device according to an exemplary embodiment 2 of the present invention. [Figure 7] 11 is a flowchart showing the flow of an information processing method executed by an information processing device according to an exemplary embodiment 2 of the present invention. [Figure 8] FIG. 11 is a block diagram showing a configuration of an information processing device according to an exemplary embodiment 3 of the present invention. [Figure 9] FIG. 11 is a diagram showing an example of a display screen displayed by an information processing device according to an exemplary embodiment 3 of the present invention. [Figure 10] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device in each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Example 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0014] <Overview of information processing device 1> The information processing device 1 according to this exemplary embodiment is, roughly speaking, a device that selects an action that maximizes the value of a certain prediction function in a given state. Here, the prediction function is, for example, a function that calculates a predicted value of the sum of a target quantity. More specifically, the information processing device 1 is, for example, Past state, - past choices, and - Observed values ​​of the target quantity obtained from past states and past actions are sequentially accumulated as learning data, and a prediction function for predicting the sum of target quantities is sequentially updated by referring to the learning data. Here, the information processing device 1 is configured to select an action that maximizes the prediction function in a given state.

[0015] In other words, the information processing device 1, for example, Get status Select the action that maximizes the prediction function under the obtained state Obtain the observed value of the target quantity obtained by the selected action -The observed values ​​of the above states, actions, and target quantities are stored as learning data. Update the prediction function using the training data The apparatus is configured to repeat the steps of:

[0016] As an example of the target quantity, a reward obtained by an action can be given. As an example of the prediction function, a reward sum function that calculates a predicted value of the sum of rewards can be given. Here, in this exemplary embodiment, "state", "action", and "reward" are interpreted as concepts that do not include any particular limitations in the information processing sense, unless otherwise specified.

[0017] In addition, the expression "learning data" in this specification is not limited to data that is referenced for updating (learning) a prediction function. Instead of the expression "learning data" in this specification, expressions such as "learning data," "teacher data," and "reference data" may be used.

[0018] <Configuration of information processing device 1> The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1.

[0019] As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, a determination unit 12, and a storage unit 13. The acquisition unit 11 is configured to realize an acquisition means in this exemplary embodiment. The determination unit 12 is configured to realize a determination means in this exemplary embodiment. The storage unit 13 is configured to realize a storage means in this exemplary embodiment.

[0020] The acquisition unit 11 acquires a state. As an example, the acquisition unit 11 acquires state information including information related to the state, and identifies the state indicated by the state information. Here, specific examples of the "state" do not limit the present exemplary embodiment, but examples include environmental conditions such as temperature and weather.

[0021] The decision unit 12 determines an action by referring to the state acquired by the acquisition unit 11. Here, the decision unit 12 calculates a first function that predicts a reward sum from the state and the action by weighting the learning data accumulated by the accumulation unit 13 described later, and determines an action using the first function. Here, the first function is sometimes called a reward sum function because it is a function that predicts the reward sum. In addition, the first function is sometimes called an action value function because it is a function that quantifies the value of an action.

[0022] Furthermore, although a specific example of the weighting process performed on the learning data by the determination unit 12 does not limit this exemplary embodiment, as an example, the determination unit 12 can calculate an index of variation from one or more values ​​included in the learning data, and calculate the first function by applying a smaller weighting coefficient to the one or more values ​​as the calculated index of variation is larger.

[0023] Here, as an example, the index related to the variance can be one that can be interpreted as an index expressing the reliability of each value included in the learning data. It can also be interpreted that the greater the variance, the lower the reliability. Therefore, it can also be expressed that the determination unit 12 calculates the first function by applying a higher weight to a value with a higher reliability.

[0024] In addition, specific examples of "action" do not limit the present exemplary embodiment, but may include, as examples, the "price" and "purchase amount" of the object. In addition, specific examples of "reward" do not limit the present exemplary embodiment, but may include, as examples, the "sales" and "reciprocal of the stock amount" or "constant minus the stock amount" of the object.

[0025] As an example, the decision unit 12 can be configured to select an action that maximizes the first function that includes the state acquired by the acquisition unit 11 as an argument, but this is not a limitation of this exemplary embodiment.

[0026] The storage unit 13 stores learning data including the state acquired by the acquisition unit 11 and the reward obtained by the action determined by the determination unit 12.

[0027] <Effects of information processing device 1> According to the information processing device 1 according to this exemplary embodiment, a first function that predicts the reward sum from the state and the action is calculated by weighting the learning data, and the action is determined using the first function, so that a more suitable action can be determined.

[0028] <Flow of information processing method by information processing device 1> The flow of the information processing method S1 executed by the information processing device 1 configured as above will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. The information processing device 1 repeatedly selects an action by repeating the information processing method S1. Note that the contents that have already been described will not be described.

[0029] As shown in FIG. 2, the information processing method S1 includes steps S11 to S13.

[0030] (Step S11) In step S11, the acquisition unit 11 acquires a state. As an example, the acquisition unit 11 acquires state information including information related to the state, and identifies the state indicated by the state information.

[0031] (Step S12) In step S12, the determination unit 12 determines an action by referring to the state acquired by the acquisition unit 11 in step S11. Here, the determination unit 12 determines an action by using a first function that predicts a reward sum from the state and the action and is calculated by weighting the learning data.

[0032] Here, as the learning data that the determining unit 12 refers to in order to calculate the first function in the nth (n is a natural number) repetition, for example, the learning data accumulated up to the n-1th repetition is used.

[0033] (Step S13) In step S13, the storage unit 13 11 The learning data including the state acquired by the acquisition unit 11 in step S12 and the reward obtained by the action determined by the determination unit 12 in step S13 is accumulated.

[0034] <Effects of information processing method S1> According to the information processing method S1 of this exemplary embodiment, a first function that predicts the reward sum from the state and the action is calculated by weighting the learning data, and the action is determined using the first function, so that a more suitable action can be determined.

[0035] <Configuration of information processing system> Next, the configuration of the information processing system 100 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing system 100.

[0036] 3, the information processing system 100 includes an information processing device 1 and a terminal device 2. Each component of the information processing device 1 has already been described, and therefore description thereof will be omitted here.

[0037] 3, the terminal device 2 includes a status information providing unit 21 and a reward information providing unit 22. The status information providing unit 21 is configured to realize a status information providing means in this exemplary embodiment. The reward information providing unit 22 is configured to realize a reward information providing means in this exemplary embodiment.

[0038] The status information providing unit 21 acquires a status and provides it to the information processing device 1. As an example, the status information providing unit 21 acquires data representing a status and provides the data to the information processing device 1.

[0039] The reward information providing unit 22 provides reward information indicating a reward obtained by performing an action determined by the information processing device 1 to the information processing device 1. As an example, the reward information providing unit 22 may be configured to include an acquisition unit that acquires action information indicating the action determined by the information processing device 1, and an execution unit that executes the action determined by the information processing device 1.

[0040] <Effects of information processing systems> According to the information processing system 100 of this exemplary embodiment, a first function that predicts the reward sum from the state and the action is calculated by weighting the learning data, and the action is determined using the first function, so that a more suitable action can be determined.

[0041] <Flow of information processing method by information processing system 100> The flow of the information processing method S100 executed by the information processing system 100 configured as above will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S100. The information processing system 100 repeatedly selects an action by executing the information processing method S100. Note that the description of the contents that have already been described will be omitted.

[0042] As shown in Fig. 4, information processing method S100 includes steps S11 to S13 and S21 to S22 which are repeated. Here, in the reference numbers given to each step in Fig. 4, the degree of repetition is written as a sub-number following a hyphen "-". For example, S21-1 indicates the first repetition, and S21-2 indicates the second repetition. The same applies to the other steps.

[0043] (Step S21-1) In step S21-1, the status information providing unit 21 of the terminal device 2 acquires data indicating a status, and provides the data to the information processing device 1.

[0044] (Step S11-1) In step S11-1, the acquisition unit 11 of the information processing device 1 acquires the status provided by the status information providing unit 21 of the terminal device 2.

[0045] (Step S12-1) In step S12-1, the decision unit 12 of the information processing device 1 decides on an action by referring to the state acquired in step S11-1 by the acquisition unit 11. Then, the information processing device 1 provides the terminal device 2 with action information indicating the decided action.

[0046] (Step S22-1) In step S22-1, the reward information providing unit 22 of the terminal device 2 provides to the information processing device 1 reward information indicating the reward obtained by performing the action determined by the determination unit 12 of the information processing device 1 in step S12-1.

[0047] (Step S13-1) In step S13-1, the storage unit 13 of the information processing device 1 stores learning data including the state acquired by the acquisition unit 11 in step S12-1 and the reward obtained by the action determined by the determination unit 12 in step S12-1.

[0048] (Step S21-2) Next, in step S21-2, the status information providing unit 21 of the terminal device 2 acquires data representing the status, and provides the data to the information processing device 1. The information acquired in this step may be different from the status acquired in step S21-1.

[0049] (Step S11-2) Step S11 - 2 In the process, the acquisition unit 11 of the information processing device 1 acquires a status provided by a status information providing unit 21 of the terminal device 2.

[0050] (Step S12-2) In step S12-2, the decision unit 12 of the information processing device 1 determines an action by referring to the state acquired by the acquisition unit 11 in step S11-2. Here, the decision unit 12 calculates a first function that predicts a reward sum from the state and the action by weighting the learning data, and determines an action using the first function. Then, the information processing device 1 provides the terminal device 2 with action information indicating the determined action.

[0051] Here, in this step, as the learning data to which the determining unit 12 refers in order to calculate the first function, the learning data accumulated up to step S13-1 is used, for example.

[0052] (Step S22-2) In step S22-2, the reward information providing unit 22 of the terminal device 2 provides to the information processing device 1 reward information indicating the reward obtained by performing the action determined by the determination unit 12 of the information processing device 1 in step S12-2.

[0053] (Step S13-2) In step S13-2, the storage unit 13 of the information processing device 1 11 The learning data including the state acquired by the acquisition unit 11 in step S12-2 and the reward obtained by the action determined by the determination unit 12 in step S12-2 is accumulated.

[0054] <Effects of the information processing method S100> According to the information processing system 100 of this exemplary embodiment, a first function that predicts the reward sum from the state and the action is calculated by weighting the learning data, and the action is determined using the first function, so that a more suitable action can be determined.

[0055] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0056] <Configuration of Information Processing System 100A> The configuration of an information processing system 100A according to this exemplary embodiment will be described with reference to FIG. 5. FIG. 5 is a block diagram showing the configuration of the information processing system 100A. As shown in FIG. 5, the information processing system 100A includes an information processing device 1A and a terminal device 2A. Also, as shown in FIG. 5, the information processing device 1A and the terminal device 2A are configured to be able to communicate with each other via a network N. Here, the specific configuration of the network N does not limit this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0057] <Configuration of information processing device 1A> The configuration of an information processing device 1A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 1A.

[0058] As shown in FIG. 5, the information processing device 1A includes a control unit 10A, a storage unit 17A, and a communication unit 19A.

[0059] The communication unit 19A communicates with devices external to the information processing device 1A. As an example, the communication unit 19A communicates with the terminal device 2A. The communication unit 19A transmits data provided from the control unit 10A to the terminal device 2A, and provides data received from the terminal device 2A to the control unit 10A.

[0060] (Control unit 10A) As shown in Fig. 5, the control unit 10A includes an acquisition unit 11, a determination unit 12, and a storage unit 13. The acquisition unit 11 is configured to realize an acquisition means in this exemplary embodiment. The determination unit 12 is configured to realize a determination means in this exemplary embodiment. The storage unit 13 is configured to realize a storage means in this exemplary embodiment.

[0061] The acquisition unit 11 acquires a status, similarly to the exemplary embodiment 1. As an example, the acquisition unit 11 acquires status information including information related to the status from the status information providing unit 21 of the terminal device 2A via the communication unit 19A. Then, the acquisition unit 11 identifies the status indicated by the acquired status information. Here, specific examples of the "status" do not limit this exemplary embodiment, but as an example, similarly to the exemplary embodiment 1, there is an environmental state such as temperature and weather.

[0062] The determination unit 12 determines an action by referring to the state acquired by the acquisition unit 11, as in the first exemplary embodiment. Here, the determination unit 12 calculates a first function that predicts a reward sum from a state and an action by weighting the learning data accumulated by the accumulation unit 13, and determines an action using the first function. Here, the first function is a function that predicts a reward sum, as in the first exemplary embodiment, and is therefore also called a reward sum function. The first function is also a function that quantifies the value of an action, and is therefore also called an action value function. A specific example of the weighting process for the learning data performed by the determination unit 12 will be described later, and therefore will not be described here.

[0063] Note that specific examples of "action" do not limit this exemplary embodiment, but may include, as examples, the "price" and "purchase amount" of the object. Also, specific examples of "reward" do not limit this exemplary embodiment, but may include, as examples, the "sales" and "reciprocal of inventory amount" or "constant minus inventory amount" of the object.

[0064] The accumulation unit 13 accumulates learning data including the state acquired by the acquisition unit 11 and the reward obtained by the action determined by the determination unit 12. As an example, the accumulation unit 13 stores the learning data including the state acquired by the acquisition unit 11 and the reward obtained by the action determined by the determination unit 12 in the memory unit 17A.

[0065] The storage unit 17A stores various data referenced by the control unit 10A. As an example, the storage unit 17A stores state information SI, action information AI, reward observation value RI, and reward sum function RSF, as shown in Fig. 5. The various data stored in the storage unit 17A will be described later with reference to different drawings.

[0066] <Configuration of Terminal Device 2A> 5, the terminal device 2A includes a control unit 20A, an action execution unit 26A, an input reception unit 28A, and a communication unit 29A. As an example, the terminal device can be specifically realized as a checkout terminal installed in a store, an inventory management terminal installed in a warehouse, or the like, but this is not intended to limit the present exemplary embodiment.

[0067] The communication unit 29A communicates with devices external to the terminal device 2A. As an example, the communication unit 29A communicates with the information processing device 1A. The communication unit 29A transmits data provided from the control unit 20A to the information processing device 1A, and provides data received from the information processing device 1A to the control unit 20A.

[0068] The control unit 20A includes a status information providing unit 21 and a remuneration information providing unit 22, as shown in FIG.

[0069] The status information providing unit 21 acquires a status and provides it to the information processing device 1. As an example, the status information providing unit 21 accepts input of data representing a status via the input accepting unit 28A and provides the information processing device 1 with the data.

[0070] The reward information providing unit 22 provides reward information indicating a reward obtained by the information processing device 1 performing the action determined by the information processing device 1 to the information processing device 1. Here, the reward information providing unit 22 can be configured to acquire the reward information indicating a reward obtained by the information processing device 1 performing the action determined by the information processing device 1 via the input receiving unit 28A.

[0071] The action executing unit 26A executes the action determined by the information processing device 1. As an example, when the action determined by the information processing device 1 is "to set the price of the object to a certain value", the action executing unit 26A sets the price associated with the object to that value. Also, when the action determined by the information processing device 1 is "to set the purchase amount of the object to a certain value", the action executing unit 26A sets the purchase amount associated with the object to that value.

[0072] The input reception unit 28A receives various inputs to the terminal device 2A. Although the specific configuration of the input reception unit 28A does not limit this exemplary embodiment, as an example, the input reception unit 28A may be configured to include input devices such as a keyboard and a touchpad. The input reception unit 28A may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves, and a sensor that senses the state of the environment.

[0073] The input reception unit 28A acquires the above-mentioned status information and the above-mentioned remuneration information via the above-mentioned input device, data scanner, sensor, etc., and supplies the acquired information to the control unit 20A. Here, the remuneration information acquired by the input reception unit 28A may include "sales" and "information related to inventory amount" regarding the target object.

[0074] (Example of data stored in storage unit 17A) Next, various types of data stored in the storage unit 17A of the information processing device 1A will be described with reference to FIG.

[0075] As shown in FIG. 6, the storage section 17A includes: Status information SI ·Behavior information AI The observed reward RI, and Reward sum function RSF The state information SI, the action information AI, and the reward observation value RI constitute, as an example, learning data referred to by the determination unit 12 in this exemplary embodiment.

[0076] (Status information SI) More specifically, as shown in FIG. 6, the state information SI is a set of state parameters s having a first index k (k=1, 2, . . . K: K is a natural number) and a second index h (h=1, 2, . . . H: H is a natural number). k h Here, the first index k is described as an index representing a date as an example, but the present exemplary embodiment is not limited to this. Also, the second index h is described as an index representing a time period as an example, but the present exemplary embodiment is not limited to this.

[0077] As shown in FIG. 6, the state information SI includes, for example, a set of state parameters s 1 1~s 1 12 , k=2 state parameter set s 2 1~s 2 12 In the example shown in FIG. 6, the state information SI includes a state parameter set s 3 1~s 3 2 is included.

[0078] State parameters s included in state information SI k hThe specific value of is acquired by the acquisition unit 11 and stored in the storage unit 17A. For example, in a configuration in which temperature is used as the state, the state parameter s k h The values ​​of temperature or values ​​obtained by converting the temperature according to a predetermined conversion rule are used for each value. In addition, in the case of a configuration in which the weather is used as the state, the state parameter s k h The values ​​of are obtained by quantifying the weather. Note that unless there is any confusion, state parameters are sometimes simply called states.

[0079] (Behavior information AI) Similarly, the behavior information AI includes a behavior parameter а having a first index k and a second index h, as shown in FIG. k h As shown in FIG. 6, the behavior information AI includes, for example, a behavior parameter set а 1 1~a 1 12 , k=2 behavioral parameter set а 2 1~a 2 12 In the example shown in Figure 6, action The information AI has a set of k=3 behavioral parameters. 3 1~a 3 2 is included.

[0080] Behavioral parameters included in behavioral information AI k h The specific value of is determined by the determination unit 12 and stored in the storage unit 17A. For example, the behavior parameter a k h As the values ​​of the parameters, values ​​indicating "price" and "purchase amount" are determined by the determination unit 12 and stored in the storage unit 17A. Unless otherwise confused, the behavior parameters may be simply called "behavior."

[0081] (Reward Observation RI) The reward observation value RI is represented by a first index k and a second index h as shown in FIG. 6, and each value is stored in the storage unit 17A. More specifically, the state s k h Act under k h The reward observation RI obtained by executing the above is expressed as r(s k h ,a k h ) and each value is stored in the storage unit 17A. As shown in FIG. 6, the reward observation value RI includes, for example, a reward observation value group r(s 1 1,a 1 1)~r(s 1 12 ,a 1 12 ), k=2 reward observations r(s 2 1,a 2 1)~r(s 2 12 ,a 2 12 In the example shown in Figure 6, the reward observation RI includes k=3 News of The set of reward observations r(s 3 1,a 3 1)~r(s 3 2,a 3 2) is included.

[0082] The reward observation RI contains r(s k h ,a k h ) is acquired by the acquisition unit 11 and stored in the storage unit 17A. For example, r(s k h ,a k h ) are acquired by the acquisition unit 11 as values ​​indicating the "sales", "reciprocal of stock amount", or "constant minus stock amount" of the object, and are stored in the memory unit 17A.

[0083] (Reward sum function RSF) As shown in FIG. 6, the storage unit 17A also stores each function form of the reward sum function RSF. Each function form of the reward sum function RSF is expressed as Q k h It is expressed as: Q k h is a function that takes two arguments as input and outputs the predicted value of the sum of rewards, Q k h It is also written as (·,·). Here, the sum of rewards refers to the total sum of rewards in a given period, for example. Q k h The two variables that it takes as arguments are, for example, state and action.

[0084] The reward sum function RSF may also be called a reward sum function Q, a Q function, or an action value function. Each functional form of the reward sum function RSF is determined by the determination unit 12 and stored in the storage unit 17A.

[0085] <Flow of information processing method by information processing device 1A> The flow of the information processing method S1A executed by the information processing device 1A configured as above will be described with reference to Fig. 7. Fig. 7 is a flow diagram showing the flow of the information processing method S1A. A By executing the information processing method S1A, the sum of the observed rewards in a predetermined period

number

[0086] In the following explanation, the set of states is

number

number

[0087] (Step S1 0 1) In step S101, the determination unit 12 initializes various parameters. As an example, the determination unit 12 acquires values ​​to be set for the parameters H and d via the acquisition unit 11, and sets the acquired values ​​as the values ​​of the parameters H and d.

[0088] Here, the parameter H is a parameter that defines the upper limit of the second index h, as described above. It can also be said that the parameter H is the total number of second indexes h that can be taken for each value of the first index k.

[0089] On the other hand, the parameter d is the dimension of the vector that represents the state and the action. In other words, the mapping for expressing the state and the action as a vector is

number

[0090] In step S101, the determination unit 12 further determines the parameters λ and β as follows, for example:

number

[0091] In step S101, the determination unit 12 performs the following initialization process.

number

number

number

number

number

number

number

[0092] Moreover, in step S101, the determination unit 12 initializes the Q function as follows.

number

number

number

number

[0093] (Step S102) Step S102 is the start point of a loop process related to the date by the determination unit 12. Here, the loop variable in the loop process related to the date is k=1, 2, . . . , K It is.

[0094] (Step S111) In step S111 in the loop on the date, the determination unit 12 determines the state s k In other words, the determination unit 12 observes the state s k Gets the value 1.

[0095] (Step S103) Step S103 is the start point of a first loop process for the time period by the determination unit 12. Here, the loop variable in the loop process for the time period is h=1, 2, . . . , H It is.

[0096] (Step S12) In step S12 in the first loop regarding the time period, the decision unit 12 determines whether the action k h As an example, the determination unit 12 selects

number

[0097] (Step S104) Next, in step S104 in the first loop regarding the time period, the determination unit 12 determines the reward r(s k h ,a k h In other words, the determination unit 12 observes the reward r(s k h ,a k h) value.

[0098] (Step S13) Next, in step S13 in the first loop regarding the time period, the storage unit 13 stores the state s k h , action a k h , and reward r(s k h ,a k h ) is stored in the memory unit 17. A Accumulate.

[0099] (Step S112) Next, in step S112 in the first loop regarding the time period, the determination unit 12 determines the state s k h+1 In other words, the determination unit 12 observes the state s k h+1 Get the value of .

[0100] (Step S105) Step S105 is the end of the first loop process by the decision unit 12 regarding the time slot.

[0101] (Step S106) Step S106 is the start point of a second loop process for the time period by the determination unit 12. The loop variable in the loop process for the time period is h=H, H-1, . . ., 1 It is.

[0102] Before entering the second loop process for the time slot, the determination unit 12

number

[0103] (Step S107) In step S107 in the second loop on the time period, the determination unit 12 updates various parameters. More specifically, the determination unit 12 performs the following update process.

number

number

number

number

[0104] In step S107, the determination unit 12

number

number

number

number

number

number

[0105] In step S107, the determination unit 12

number

number

number

number

number

[0106] (Step S108) In step S108, the determination unit 12 calculates the reward sum function Q k h More specifically, by using the various parameters updated in step S107,

number

[0107] (Step S109) Step S109 is the end of the second loop process by the decision unit 12 regarding the time slot.

[0108] (Step S110) Step S110 is the end of the loop process performed by the determination unit 12 on the date.

[0109] (Detailed explanation of information processing method S1A) The flow of the information processing method S1A has been described above, but a more detailed explanation is as follows.

[0110] First, as described above, the information processing method S1A includes: Obtaining the status (steps S111 and S112) Deciding an action by referring to the state (step S12); and and repeatedly accumulating learning data including the state and a reward obtained by the determined action (S13). In the step of determining the action (step S12), the action is determined using a first function that predicts a reward sum from a state and an action, and that is calculated by weighting the learning data. Therefore, according to the information processing method S1A, a more suitable action can be determined.

[0111] Furthermore, as described above with reference to (Formula A1), the determination unit 12 selects an action that maximizes the reward sum function that includes the state acquired by the acquisition unit 11 as an argument, and therefore can preferably select an action that maximizes the observed value of the reward in a specified period of time.

[0112] In addition, in the first and second lines of (Formula A3),

number

number

number

[0113] In addition, the second line of (Equation A3) shows that the larger the variance value of the state evaluation function, the more the vector

number

number

[0114] Therefore, the decision unit 12 that executes the information processing method S1A: Calculating an index of variation from one or more values ​​included in the learning data; The larger the calculated index of variability is, the smaller the weighting coefficient is applied to the one or more values ​​to calculate the reward sum function (also referred to as a first function).

[0115] As described above, the determining unit 12 calculates, as an index relating to the variance, the variance of the state evaluation function (also referred to as the second function) obtained by referring to the state and the action.

[0116] Here, the variance of the state evaluation function can be interpreted as an index expressing the reliability of each value included in the learning data. Therefore, it can be interpreted that the greater the variance, the lower the reliability. Therefore, it can be expressed that the determination unit 12 calculates the reward sum function by applying a higher weight to a value with higher reliability.

[0117] Therefore, according to the above configuration, by incorporating a greater contribution from more reliable learning data, it is possible to suitably determine an action that maximizes the sum of the observed reward values.

[0118] As described above, the determination unit 12 also generates a feature map that maps states and actions to vectors.

number

[0119] In the above example, the variance value of the state evaluation function V is used as an index of variation, but this does not limit the present exemplary embodiment. An index other than the variance value, such as the standard deviation of the state evaluation function V, may be used as an index of variation.

[0120] In addition, when the information processing device 1A is configured to determine the price of an object as an action, it may be expressed as a price determination device or an object management device, and when it is configured to determine the purchase quantity of an object as an action, it may be expressed as a purchase quantity determination device or an inventory management device.

[0121] Exemplary embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0122] <Configuration of information processing device 1B> The configuration of an information processing device 1B according to this exemplary embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the information processing device 1B.

[0123] As shown in FIG. 8, the information processing device 1B is 2 In addition to the components of the information processing device 1A, the information processing device 1A includes a display unit 15B and an input receiving unit 16B. The display unit 15B is a component that realizes the display means in this exemplary embodiment. The input receiving unit 16B is a component that realizes the input receiving means in this exemplary embodiment.

[0124] (Display section 15B) The display unit 15B is configured to be capable of displaying various data to be processed by the information processing device 1B. The display contents of the display unit 15B are controlled by the control unit 10A. As an example, the display unit 15B is configured to include a display panel and a drive circuit that drives the display panel.

[0125] The display unit 15B, for example, displays the state s kh , action a k h , the reward r(s k h ,a k h ), and / or the value of the reward sum function Q, together with the variance of the state evaluation function V.

[0126] The upper part of Fig. 9 is a diagram showing an example of a display screen displayed by the display unit 15B. In the example shown in the upper part of Fig. 9, the display unit 15B displays the value of the reward sum function Q on a certain day and the variance of the corresponding state evaluation function V for each time period.

[0127] In the example shown in the upper part of FIG. 9, the black circles represent the values ​​of the reward sum function Q, and the vertical bars represent the variance of the state evaluation function V.

[0128] In addition, the display unit 15 B As an example, state s k h , action a k h , the reward r(s k h ,a k h ), and the value of the reward sum function Q, values ​​for which the variance of the corresponding state evaluation function V is equal to or less than a threshold value may be highlighted.

[0129] The lower part of Fig. 9 is a diagram showing an example of a display screen displayed by the display unit 15 B. In the example shown in the lower part of Fig. 9, the display unit 15 B displays the value (price) of the action a on a certain day and the variance of the corresponding state evaluation function V for each time period.

[0130] In the example shown in the lower part of Fig. 9, the black circles are the values ​​(prices) of the action a, and the vertical bars indicate the variance of the state evaluation function V. Furthermore, the variance of the state evaluation function V can be interpreted as expressing the reliability of each value of the data, as described above.

[0131] By displaying as described above on display unit 15B, the value of each data item can be visually presented together with its reliability to the user of information processing device 1B, thereby improving the usability and explainability of information processing device 1B.

[0132] In the example shown in the lower part of Fig. 9, values ​​for which the variance of the corresponding condition evaluation function V is equal to or less than the threshold are highlighted. More specifically, the highlighted display is achieved by displaying black circles corresponding to prices for which the variance of the corresponding condition evaluation function V is equal to or less than the threshold and the corresponding bars surrounded by dashed lines.

[0133] By being equipped with the display unit 15B as described above, the information processing device 1B can visually present to a user of the information processing device 1B data in which the variance of the state evaluation function V is below a threshold value (in other words, the reliability is above a threshold value), thereby further improving the usability and explainability of the information processing device 1B.

[0134] In this exemplary embodiment, the information processing device 1B may be configured to include a recommended value calculation unit that calculates recommended values ​​of at least any of the parameters d, H, λ, and β described in exemplary embodiment 2, and to present the calculated recommended values ​​to a user of the information processing device 1B via the display unit 15B.

[0135] (Input reception unit 16B) The input reception unit 16B receives various inputs to the information processing device 1B. Although the specific configuration of the input reception unit 16B does not limit this exemplary embodiment, as an example, the input reception unit 16B may be configured to include input devices such as a keyboard and a touch pad. The input reception unit 16B may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays and radio waves, and a sensor that senses the state of the environment.

[0136] The input receiving unit 16B acquires the observed values ​​of the above-mentioned states and rewards via the above-mentioned input devices, data scanners, sensors, etc., and supplies the acquired information to the control unit 10A.

[0137] The information received by the input receiving unit 16B is not limited to the above example. As an example, the correction information for correcting the action determined by the determination unit 12 may be received from the user of the information processing device 1B. For example, the display unit 15B may display the above-mentioned information, and the user who recognizes the displayed content may input the correction information for correcting the action (price) to the input receiving unit 16B.

[0138] In the case of the above configuration, the determination unit 12 determines a corrected action by correcting the action (price) determined in step S12 in the exemplary embodiment 2 by the correction amount indicated by the correction information. Then, the determination unit 12 observes the reward obtained by performing the corrected action, and executes the remaining processing described in the exemplary embodiment 2.

[0139] According to the above configuration, since a correction by a user can be reflected in the action determined by the determination unit 12, it is possible to improve usability and explainability.

[0140] (Application example of information processing device 1B) An application example of the information processing device 1B will be described below. The following application example is an example in which the information processing device 1B is used to determine the price of beer from each company at a certain store. More specifically, the discount rate of beer from each company at a certain store is determined as an action (execution measure).

[0141] In this example, execution measure X is X=[0, 2, 1, . . .] Here, the first element being 0 indicates that the price of beer from company A is the regular price, the second element being 2 indicates that the price of beer from company B is a 10% premium over the regular price, and the third element being 1 indicates that the price of beer from company C is a 10% discount off the regular price.

[0142] As the reward sum function Q in this example, a reward sum function Q relating to the beer sales of Company A, Company B, and Company C may be prepared separately and updated separately. Alternatively, as the reward sum function Q, a reward sum function relating to the total sales of Company A's beer, Company B's beer, and Company C's beer may be prepared and updated.

[0143] In this example, the display unit 15B visually presents the beer sales of each company.

[0144] According to this application example, it is possible to derive optimal pricing for beer from each company at the store.

[0145] [Software implementation example] Some or all of the functions of the information processing devices 1, 1A, and 1B may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0146] In the latter case, the information processing devices 1, 1A, and 1B are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 10. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1, 1A, and 1B. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1, 1A, and 1B.

[0147] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0148] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0149] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0150] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0151] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0152] (Appendix 1) An acquisition means for acquiring a state; A decision means for deciding an action by referring to the state; A storage means for storing learning data including the state and a reward obtained by the action determined by the determination means; Equipped with The determining means is A first function for predicting a reward sum from a state and an action is calculated by weighting the learning data, and an action is determined using the first function. 23. An information processing apparatus comprising:

[0153] According to the above configuration, a first function that predicts the reward sum from the state and the action is calculated by weighting the learning data, and the action is determined using the first function, so that a more suitable action can be determined.

[0154] (Appendix 2) The determining means is Calculating an index of variation from one or more values ​​included in the learning data; The first function is calculated by applying a smaller weighting factor to the one or more values ​​when the calculated variability index is larger. The appended claim characterized by 1 The information processing device described in

[0155] According to the above configuration, the first function is calculated by applying a smaller weighting coefficient to the one or more values ​​as the index related to the variability becomes larger, so that a more suitable action can be determined.

[0156] (Appendix 3) The determining means is Calculating the variance of a second function obtained by referring to the state and the behavior as an index of the variance. 3. The information processing device according to claim 2.

[0157] According to the above configuration, the variance of the second function obtained by referring to the state and the action is calculated as the index related to the variation, so that a more suitable action can be determined.

[0158] (Appendix 4) 4. The information processing device according to claim 2 or 3, further comprising a display means for displaying at least one of the state, the action, the reward, and the value of the first function, and an index related to the variability.

[0159] According to the above configuration, usability and explainability are improved.

[0160] (Appendix 5) The display means includes: Highlighting a value of the state, the action, the reward, and the value of the first function, the value of which has a variability index equal to or less than a threshold value. 5. The information processing device according to claim 4.

[0161] According to the above configuration, usability and explainability are improved.

[0162] (Appendix 6) The determining means is Calculating the first function using a feature map that maps the state and the action to a vector. 6. The information processing device according to claim 1,

[0163] According to the above configuration, a more suitable action can be determined.

[0164] (Appendix 7) The decision means selects an action that maximizes the first function that includes the state acquired by the acquisition means as an argument. 7. The information processing device according to claim 1,

[0165] According to the above configuration, an action that maximizes the first function that includes the state acquired by the acquisition means as an argument is selected, so that a suitable action that maximizes the observed value of the reward in a specified period of time can be selected.

[0166] (Appendix 8) The game machine further includes an input device for receiving the state and the reward. 8. The information processing device according to claim 1,

[0167] According to the above configuration, the state and the reward can be suitably input via the input device.

[0168] (Appendix 9) An information processing device, Obtaining the state, determining an action with reference to said state; and accumulating learning data including the state and a reward obtained by the determined action; Repeatedly includes In the step of determining the action, The action is determined using a first function that predicts a reward sum from a state and an action, the first function being calculated by weighting the learning data. 23. An information processing method comprising:

[0169] According to the above method, the same effects as those of the above-mentioned information processing device can be achieved.

[0170] (Appendix 10) A program for causing a computer to function as an information processing device, The program causes the computer to An acquisition means for acquiring a state; A decision means for deciding an action by referring to the state; a storage means for storing learning data including the state and a reward obtained by the action determined by the determination means; The determining means is A first function for predicting a reward sum from a state and an action is calculated by weighting the learning data, and an action is determined using the first function. A program characterized by:

[0171] According to the above program, the same effects as those of the above information processing device can be achieved.

[0172] (Appendix 11) An information processing system including an information processing device and a terminal device, The information processing device includes: An acquisition means for acquiring a state; A decision means for deciding an action by referring to the state; A storage means for storing learning data including the state and a reward obtained by the action determined by the determination means; Equipped with The determining means is Calculating a first function that predicts a reward sum from a state and an action by weighting the learning data, and determining an action using the first function; The terminal device A status information providing means for acquiring a status and providing the status to the information processing device; a reward information providing means for providing the information processing device with reward information indicating a reward obtained by the information processing device performing the determined action; Equipped with An information processing system comprising:

[0173] According to the above information processing system, the same effects as those of the above information processing device are achieved.

[0174] (Appendix 12) An information processing device, To get the state, determining an action with reference to said state; and accumulating learning data including the state and a reward obtained by the determined action; Repeatedly includes In the step of determining the action, determining an action using a first function that predicts a reward sum from a state and an action, the first function being calculated by weighting the learning data; The terminal device acquiring a status and providing it to the information processing device; and providing the information processing device with reward information indicating a reward obtained by the information processing device performing the determined action; Includes 23. An information processing method comprising:

[0175] According to the above information processing method, the same effects as those of the above information processing device are achieved.

[0176] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0177] At least one processor, the processor comprising: An acquisition process for acquiring a state; A decision process for deciding an action by referring to the state; an accumulation process for accumulating learning data including the state and a reward obtained by the action determined by the determination process; Run The processor, A first function for predicting a reward sum from a state and an action is calculated by weighting the learning data, and an action is determined using the first function.

[0178] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process, the determination process, and the accumulation process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0179] 1, 1A, 1B Information processing device 11 Acquisition Department 12 Decision Section 13 Storage section 15B Display section 16B Input reception section 17A Storage section 100,100A Information Processing System

Claims

1. An acquisition means for acquiring a state; A decision means for deciding an action by referring to the state; A storage means for storing learning data including the state and a reward obtained by the action determined by the determination means; Equipped with The determining means is Calculating an index related to variation from one or more values ​​included in the learning data; A first function for predicting the sum of rewards from the state and the action is calculated by applying a smaller weighting coefficient to the one or more values ​​as the calculated index of variability is larger, and the action is determined using the first function.

23. An information processing apparatus comprising:

2. The determining means comprises: Calculating the variance of a second function obtained by referring to the state and the behavior as the index of the variance.

2. The information processing apparatus according to claim 1,

3. An information processing device as described in claim 1 or 2, characterized in that it is provided with a display means for displaying at least any of the state, the action, the reward, and the value of the first function, as well as an index regarding the variability.

4. The display means Among the at least any of the state, the action, the reward, and the value of the first function, a value having an index of variability equal to or less than a threshold is highlighted.

4. The information processing apparatus according to claim 3.

5. The determining means comprises: Calculating the first function using a feature map that maps the states and the actions to vectors.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises: a first input section;

6. An information processing device comprising: Obtaining the state, determining an action with reference to said state; and accumulating learning data including the state and a reward obtained by the determined action; Repeatedly includes In the step of determining the action, The action is determined using a first function that predicts a reward sum from a state and an action, the first function being calculated by applying a smaller weighting coefficient to one or more values ​​as an index of variability calculated from the one or more values ​​included in the learning data increases.

23. An information processing method comprising:

7. A program for causing a computer to function as an information processing device, comprising: The program causes the computer to An acquisition means for acquiring a state; A decision means for deciding an action by referring to the state; a storage means for storing learning data including the state and a reward obtained by the action determined by the determination means; The determining means is Calculating an index related to variation from one or more values ​​included in the learning data; A first function for predicting the sum of rewards from the state and the action is calculated by applying a smaller weighting coefficient to the one or more values ​​as the calculated index of variability is larger, and the action is determined using the first function. A program characterized by:

8. An information processing system including an information processing device and a terminal device, The information processing device includes: An acquisition means for acquiring a state; A decision means for deciding an action by referring to the state; A storage means for storing learning data including the state and a reward obtained by the action determined by the determination means; Equipped with The determining means is Calculating an index related to variation from one or more values ​​included in the learning data; calculating a first function for predicting a reward sum from a state and an action by applying a smaller weighting coefficient to the one or more values ​​as the calculated index of variability is larger, and determining an action using the first function; The terminal device A status information providing means for acquiring a status and providing the status to the information processing device; a reward information providing means for providing the information processing device with reward information indicating a reward obtained by the information processing device performing the determined action; Equipped with An information processing system comprising:

9. An information processing device comprising: Obtaining the state, determining an action with reference to said state; and accumulating learning data including the state and a reward obtained by the determined action; Repeatedly includes In the step of determining the action, determining an action using a first function that predicts a reward sum from a state and an action, the first function being calculated by applying a smaller weighting coefficient to one or more values ​​as an index of variability calculated from the one or more values ​​included in the learning data increases; The terminal device acquiring a status and providing it to the information processing device; and providing the information processing device with reward information indicating a reward obtained by the information processing device performing the determined action; Includes 23. An information processing method comprising:

Citation Information

Patent Citations

  • State estimation method, state estimation device, state estimation system and computer program

    JP2007128318A

  • Information providing device and information providing program

    JP2017162385A

  • Burden estimation device and burden estimation program

    JP2018097399A

  • Smoothed Sarsa: Reinforcement Learning for Robot Delivery Tasks

    US20100094786A1

  • Information processing device and information processing method

    WO2021075107A1