Optimization device, optimization method, and program

The optimization method updates probability distributions using past data as a constraint, enabling optimal decision-making in adversarial environments without requiring prior knowledge of the environment's structure, thus improving decision-making efficiency.

JP7582317B2Active Publication Date: 2024-11-13NEC CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2022553260
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-29
Publication Date
2024-11-13
Estimated Expiration
2040-09-29

AI Technical Summary

Technical Problem

Existing decision-making optimization techniques assume a specific probability distribution for the objective function, which is not feasible in real-world adversarial settings, making it difficult to determine the appropriate algorithm without human judgment and knowledge of the environment's structure.

Method used

An optimization method that determines optimal measures by updating a probability distribution using a weighted sum of past distributions as a constraint, eliminating the need to select an algorithm based on the environment's structure.

Benefits of technology

Adaptively updates the probability distribution to determine optimal measures without prior knowledge of the environment, achieving performance comparable to selecting the best algorithm for the specific environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582317000017
    Figure 0007582317000017
  • Figure 0007582317000018
    Figure 0007582317000018
  • Figure 0007582317000019
    Figure 0007582317000019
Patent Text Reader

Abstract

In an optimization device according to the present invention, an acquisition means acquires a reward obtained by executing a certain policy. An update means updates the probability distribution of policies on the basis of the acquired reward. The update means uses, as a constraint, a weighted sum of probability distributions updated in the past. A determination means determines a policy to be executed on the basis of the updated probability distribution.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to optimization techniques for decision making. [Background technology]

[0002] There is a known technology that selects and implements an appropriate measure from among candidate measures, such as optimizing product prices, and sequentially optimizes the measure based on the obtained reward. Patent Document 1 discloses a technology for making appropriate decisions under constraints. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2020 / 012589 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 assumes a probability distribution for the objective function (probabilistic setting). However, in a real-world environment, it may not be possible to assume a specific probability distribution for the objective function (adversarial setting). Therefore, in practical decision-making, it is difficult to determine which of the above problem settings the objective function is suited to. In addition, various algorithms have been proposed for the adversarial setting, but in order to select an appropriate algorithm, it is necessary to properly grasp the structure of the "environment" (for example, whether or not there is a large variance in the rewards obtained), which requires human judgment and knowledge.

[0005] An object of the present disclosure is to provide an optimization method capable of determining optimal measures without depending on the setting of an objective function or the structure of the "environment." [Means for solving the problem]

[0006] In one aspect of the present disclosure, an optimization apparatus includes: A means for acquiring a reward obtained by executing a certain measure; Based on the rewards received, Among the various measures, which should be implemented? Measures Used to determine an updating means for updating the probability distribution; A decision means for deciding a measure to be taken based on the updated probability distribution; Equipped with The updating means uses a weighted sum of previously updated probability distributions as a constraint.

[0007] In another aspect of the present disclosure, a method for optimizing comprises: Receive the rewards obtained by implementing a certain measure, Based on the rewards received, Among the various measures, which should be implemented? Measures Used to determine Update the probability distribution, Determine the action to be taken based on the updated probability distribution; The probability distribution is updated using a weighted sum of previously updated probability distributions as a constraint.

[0008] In yet another aspect of the disclosure, a program includes: Receive the rewards obtained by implementing a certain measure, Based on the rewards received, Among the various measures, which should be implemented? Measures Used to determine Update the probability distribution, Determine the action to be taken based on the updated probability distribution; The probability distribution is updated using a weighted sum of previously updated probability distributions as a constraint. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of the optimization device. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the optimization device. [Diagram 3] 4 is a flowchart of an optimization process according to the first embodiment. [Figure 4] 13 is a flowchart of an optimization process according to a second embodiment. [Diagram 5]FIG. 13 is a block diagram showing the functional configuration of an optimization device according to a third embodiment. [Figure 6] 13 is a flowchart of a prediction process by an optimization device according to a third embodiment. [Figure 7] 1 illustrates a schematic diagram of a basic embodiment of the optimization process of the present disclosure. [Figure 8] An example in which the optimization method of the embodiment is applied to the retail field will be described. [Figure 9] An example in which the optimization method of the embodiment is applied to the field of investment will be described. [Figure 10] An example in which the optimization method of the embodiment is applied to the medical field will be described. [Figure 11] An example in which the optimization method of the embodiment is applied to marketing will be described. [Figure 12] An example in which the optimization method of the embodiment is applied to the estimation of power demand will be described. [Figure 13] An example in which the optimization method of the embodiment is applied to the field of communications will be described. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Prerequisites] (Bandit Optimization) Bandit optimization is a method of sequential decision making with limited information. In bandit optimization, a player is given a set of actions A and sequentially selects action i t and select the loss l at each time step t. t (i t The player's goal is to observe the regret R T The aim of the present invention is to minimize the above problem.

[0011]

number

[0012] There are mainly two different approaches to bandit optimization. The first approach is for stochastic environments, where the loss l is chosen for every time step t. t follows an unknown probability distribution, i.e., the environment is time-invariant. The second approach is for adversarial or non-stochastic environments. In this environment, the loss l t There is no model for loss l t can become hostile towards the player.

[0013] (Multi-armed bandit problem) In a multi-armed bandit problem, the set of policies is a finite set [K] of size K. At each time step t, a player chooses policies i t ∈[K] and the loss l tit Observe the loss vector l t =(l t1 ,l t2 ,··,l tK ) T ∈[0,1] K can be chosen adversarially by the environment. The player's goal is to minimize the regrets of

[0014]

number

[0015] In the multi-armed bandit problem, a probabilistic model or an adversarial model is used. A probabilistic model is suitable for stationary environments, and the loss obtained by the strategy is t We assume that follows an unknown stationary probability distribution. On the other hand, the adversarial model is used in a non-stationary environment, i.e., the loss obtained by the campaign, l tThis model is suitable for environments where the loss l t It is assumed that enemies may become hostile towards the player.

[0016] Examples of adversarial models include a worst-case evaluation model, a first-order evaluation model, a variance-dependent evaluation model, and a path-length evaluation model. The worst-case evaluation model can guarantee performance when the actual environment is the worst case (the most unfavorable environment for the algorithm), that is, it is possible to keep regret within a predetermined range. In the first-order evaluation model, performance can be expected to improve if measures are taken to reduce cumulative loss. In the variance-dependent evaluation model, performance can be expected to improve if the variance in loss is small. In the path-length-dependent evaluation model, performance can be expected to improve if the time change in loss is small.

[0017] As mentioned above, for the multi-armed bandit problem, several models can be applied depending on whether the real environment is stationary or non-stationary. Therefore, to achieve optimal performance, it is necessary to select an appropriate algorithm according to the real-world environment. However, in reality, it is difficult to select an appropriate algorithm by knowing the structure of the environment (stationary / non-stationary, large or small change) in advance.

[0018] Therefore, in this embodiment, there is no need to select an algorithm depending on the structure of the environment, and a single algorithm is used to obtain results equivalent to those obtained when an appropriate algorithm is selected from multiple algorithms.

[0019] [Hardware configuration] 1 is a block diagram showing a hardware configuration of an optimization device 100. As shown in the figure, the optimization device 100 includes a communication unit 11, a processor 12, a memory 13, a recording medium 14, a database (DB) 15, a display unit 16, and an input unit 17.

[0020] The communication unit 11 inputs and outputs data to and from an external device. Specifically, the communication unit 11 outputs the measure selected by the optimization device 100, and acquires the loss (reward) incurred by the measure.

[0021] The processor 12 is a computer such as a CPU (Central Processing Unit), and controls the entire optimization device 100 by executing a program prepared in advance. The processor 12 may be one of a CPU, a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit), or may use a plurality of them in parallel. Specifically, the processor 12 executes the optimization process described below.

[0022] The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

[0023] The recording medium 14 is a non-volatile and non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the optimization device 100. The recording medium 14 records various programs executed by the processor 12. When the optimization device 100 executes an optimization process, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12.

[0024] The DB 15 stores input data input through the communication unit 11 and data generated during processing by the optimization device 100. The optimization device 100 may also include a display unit such as a liquid crystal display device, an input unit 17 for an administrator to give necessary instructions and input, and the like, as necessary.

[0025] [Function configuration] 2 is a block diagram showing a functional configuration of the optimization device 100. In terms of functions, the optimization device 100 includes an input unit 21, a calculation unit 22, a storage unit 23, and an output unit 24. The input unit 21 acquires a loss obtained as a result of executing a certain measure, and outputs it to the calculation unit 22. The storage unit 23 stores a probability distribution used to determine the measure. The calculation unit 22 updates the probability distribution stored in the storage unit 23 based on the loss input from the input unit 21. Although details will be described later, the calculation unit 22 updates the probability distribution using a weighted sum of previously updated probability distributions as a constraint.

[0026] Furthermore, the calculation unit 22 determines the next measure using the updated probability distribution and outputs it to the output unit 24. The output unit 24 outputs the measure determined by the calculation unit 22. When the output measure is executed, the loss resulting therefrom is input to the input unit 21. In this way, every time a measure is executed, the loss (reward) is fed back to the input unit 21, and the probability distribution stored in the storage unit 23 is updated. This allows the optimization device 100 to determine the next measure using a probability distribution adapted to the actual environment. In the above configuration, the input unit 21 is an example of an acquisition means, and the calculation unit 22 is an example of an update means and a determination means.

[0027] [Optimization process] Fig. 3 is a flowchart of the optimization process according to the first embodiment. This process can be realized by the processor 12 shown in Fig. 1 executing a prepared program and operating as the elements shown in Fig. 2. It is assumed that the number K of selectable measures has been determined.

[0028] First, the predicted value of the loss vector m t is initialized (step S11). Specifically, the predicted value m1 of the loss vector is set to “0.” Next, for time steps t=1, 2, . . . , the following loop processing of steps S12 to S19 is repeatedly executed. First, the calculation unit 22 calculates the probability distribution p t Calculate (step S13).

[0029]

number

[0030]

number

[0031] Next, the calculation unit 22 calculates the calculated probability distribution p t Based on the measures t The output unit 24 outputs the determined policy i t (Step S14). Next, the input unit 21 outputs the measure i output in step S14. t The loss obtained by running tit (Step S15). Next, the calculation unit 22 calculates the obtained loss l tit Using the following equation (5), we obtain an unbiased estimator of the loss vector l^ t Calculate (step S16).

[0032]

number

[0033] Next, the calculation unit 22 calculates the weight parameter γti and updates the regularization term Φt(p) using equation (4) (step S17).

[0034]

number

[0035]

number

[0036] Next, the calculation unit 22 calculates the predicted value m of the loss vector using the following equation (8). t is updated (step S18).

[0037]

number

[0038] In this way, in the optimization process of the first embodiment, in step S17, the degree of error in the prediction of the loss in the past time step α ji The weight parameter γ, which indicates the strength of regularization, is calculated using equation (6) based on the accumulation of ti Then, the weighting parameter γ ti The regularization term Φt(p) is updated by equation (4) based on the above. Therefore, the regularization term Φt(p) is updated using the weighted sum of past probability distributions as a constraint, and the probability distribution p shown in equation (3) is t The regularization strength in is updated appropriately.

[0039] In step S18, the predicted value m of the loss vector is calculated by taking into account the loss obtained by implementing the selected measure, as shown in formula (8). t Specifically, the loss obtained by the selected measure is updated. tit The predicted value of the loss vector for the next time step m t+1 This generates the predicted value m of the loss vector according to the result of the implementation of the measures. t is updated appropriately.

[0040] As described above, in the optimization process of the first embodiment, it is possible to adaptively update the probability distribution of measures in accordance with the actual environment and determine the optimal measure, without the need to pre-select an algorithm based on the target environment.

[0041] <Second embodiment> [Prerequisites] The second embodiment relates to a linear bandit problem. In a linear bandit problem, a set of measures A is a linear space R d At each time step t, the player takes action a t ∈A and the loss l t T at Observe the loss vector l t ∈R d can be chosen adversarially by the environment. For every policy a, the loss l t T Assume that a∈[0,1] holds. Regret is defined by the following formula (9). Note that a * is the best policy.

[0042]

number

[0043] Therefore, in order to achieve optimal performance in linear bandit problems, it is necessary to select an appropriate algorithm according to the real-world environment, but in reality, it is difficult to select an appropriate algorithm by knowing the structure of the environment (steady / unsteady, magnitude of change) in advance. In the second embodiment, for linear bandit problems, it is unnecessary to select an algorithm according to the structure of the environment, and a single algorithm is used to obtain results equivalent to those obtained when an appropriate algorithm is selected from among multiple algorithms.

[0044] [Hardware configuration] The hardware configuration of the optimization device of the second embodiment is similar to that of the optimization device 100 of the first embodiment shown in FIG.

[0045] [Function configuration] The functional configuration of the optimization device of the second embodiment is similar to that of the optimization device 100 of the first embodiment shown in FIG.

[0046] [Optimization process] loss l t For the predicted value of the loss vector m t ∈R d In this setting, the player can choose the policy a t Before selecting, the predicted value of the loss vector m t For all policies a, a∈A, 〈m t Assume that ,a〉∈[1,-1]. Perform the following multiplicative weight update on the convex hull A' of the policy set A:

[0047]

number

[0048]

number

[0049]

number

[0050] Fig. 4 is a flowchart of the optimization process according to the second embodiment. This process is realized by the processor 12 shown in Fig. 1 executing a prepared program and operating as the elements shown in Fig. 2.

[0051] First, the calculation unit 22 calculates the predicted value m of the loss vector. t∈L is arbitrarily set (step S21). The set L is defined as follows.

[0052]

number

[0053] Next, for time steps t=1, 2, . . . , T, the following loop process of steps S22 to S29 is repeatedly executed.

[0054] First, the calculation unit 22 calculates x t The norm of is dβ t 2 The probability distribution p defined by equation (11) is maintained until t (x) to x t is repeatedly selected (step S23).

[0055]

number

[0056]

number

[0057]

number

[0058] Equation (16) uses the coefficients λ and D to calculate the predicted value m t In other words, the predicted value of the loss vector is corrected in a direction that reduces the prediction error with a step size of about the coefficient λ. t -l t ,a t 〉」 is the predicted value of the loss vector m t The deviation between the loss vector and the loss lt is in the opposite direction to the predicted value m t Also, adjust "D(m||m t ) is the predicted value of the loss vector m t That is, like equation (3) in the first embodiment, equation (16) adaptively updates the predicted value m of the loss vector according to the loss caused by the execution of the selected measure. t The strength of regularization in is adjusted. Then, the predicted value m t Using equations (10) and (11), the probability distribution p t As a result, in the optimization process of the second embodiment, it is possible to adaptively update the probability distribution in accordance with the actual environment and determine the optimal measure without the need to select an algorithm in advance based on the target environment.

[0059] <Third embodiment> Next, a third embodiment of the present disclosure will be described. FIG. 5 is a block diagram showing a functional configuration of an optimization device 200 according to the third embodiment. The optimization device 200 includes an acquisition means 201, an update means 202, and a decision means 203. The acquisition means 201 acquires a reward for a measure obtained by executing a certain measure. The update means 202 updates the probability distribution of the measure based on the acquired reward. Here, the update means 202 uses a weighted sum of probability distributions updated in the past as a constraint. The decision means 203 decides a measure to be executed based on the updated probability distribution.

[0060] 6 is a flowchart of a prediction process by the optimization device of the third embodiment. In the optimization device 200, the acquisition means 201 executes a certain measure and acquires the reward of the measure (step S51). The update means 202 updates the probability distribution of the measure based on the acquired reward (step S52). Here, the update means 202 uses a weighted sum of the probability distributions updated in the past as a constraint. The decision means 203 decides the measure to be executed based on the updated probability distribution (step S53).

[0061] According to the third embodiment, by updating the probability distribution using the weighted sum of previously updated probability distributions as a constraint, it is possible to adaptively update the probability distribution of measures in accordance with the actual environment and determine the optimal measure, without the need to pre-select an algorithm based on the target environment.

[0062] <Example> Next, an embodiment of the optimization process of the present disclosure will be described. [Basic Example] FIG. 7 shows a schematic diagram of a basic example of the optimization process of the present disclosure. The objective function f(x) corresponding to the environment in which a measure is selected by decision-making may be probabilistic or adversarial, as described above. At time t1, when a measure A1 is selected and executed based on the probability distribution P1 of the measures, a reward (loss) according to the objective function f(x) is obtained. Using this reward, the probability distribution is updated from P1 to P2, and at time t2, a measure A2 is selected based on the updated probability distribution P2. In this case, by applying the optimization method of the embodiment, it is possible to determine an appropriate measure according to the environment indicated by the objective function f(x).

[0063] [Example 1] 8 shows an example in which the optimization method of the embodiment is applied to the retail field. Specifically, a policy is to discount the price of beer from each company at a certain store. For example, when the execution policy X=[0, 2, 1, . . .], the first element indicates that the price of beer from company A is the regular price, the second element indicates that the price of beer from company B is increased by 10% from the regular price, and the third element indicates that the price of beer from company C is discounted by 10% from the regular price.

[0064] The objective function takes the execution measure X as input and the result of applying the execution measure X to the price of beer of each company and selling the beer as output. In this case, by applying the optimization method of the embodiment, it is possible to derive the optimal price setting of the beer price of each company in the store.

[0065] [Example 2] FIG. 9 shows an example in which the optimization method of the embodiment is applied to the field of investment. Specifically, a case in which the optimization method is applied to the investment behavior of an investor or the like will be described. In this case, the execution measures are investment (purchase, capital increase), sale, and holding of multiple financial products (stock brands, etc.) that the investor holds or intends to hold. For example, if the execution measure X=[1, 0, 2, . . .], the first element indicates additional investment in the stock of company A, the second element indicates holding the bonds of company B (neither purchase nor sale), and the third element indicates selling the stock of company C. The objective function takes the execution measure X as input, and outputs the result of applying the execution measure X to the investment behavior of the financial products of each company.

[0066] In this case, by applying the optimization method of the embodiment, it is possible to derive optimal investment behavior for each stock for the investor.

[0067] [Example 3] FIG. 10 shows an example in which the optimization method of the embodiment is applied to the medical field. Specifically, a case in which the method is applied to medication behavior for a clinical trial of a certain drug at a pharmaceutical company will be described. In this case, the amount of medication or avoidance of medication is set as the execution measure. For example, when execution measure X=[1, 0, 2, . . .], the first element indicates that a dose of 1 is administered to subject A, the second element indicates that no medication is administered to subject B, and the third element indicates that a dose of 2 is administered to subject C. The objective function takes the execution measure X as input and outputs the result of applying the execution measure X to the medication behavior for each subject.

[0068] In this case, by applying the optimization method of the embodiment, it is possible to derive optimal medication behavior for each subject in the pharmaceutical company's clinical trial.

[0069] [Example 4] FIG. 11 shows an example in which the optimization method of the embodiment is applied to marketing. Specifically, a case in which the method is applied to advertising behavior (marketing measures) in an operating company of a certain electronic commerce site will be described. In this case, the execution measures are advertisements (online (banner) advertisements, advertisements by e-mail, direct mail, sending discount coupons by e-mail, etc.) for a product or service that the operating company is trying to sell to multiple customers. For example, if execution measures X=[1, 0, 2, . . .], the first element indicates banner advertisements for customer A, the second element indicates no advertisements for customer B, and the third element indicates sending discount coupons by e-mail to customer C. The objective function takes execution measures X as input and outputs the results of applying execution measures X to advertising behavior for each customer. Here, the execution results may be whether or not a banner advertisement was clicked, the purchase amount, the purchase probability, and the expected value of the purchase amount.

[0070] In this case, by applying the optimization method of the embodiment, it is possible to derive optimal advertising behavior for each customer in the operating company.

[0071] [Example 5] Fig. 12 shows an example in which the optimization method of the embodiment is applied to the estimation of power demand. Specifically, the operation rate of each generator in a power generation facility is set as the execution measure. For example, when the execution measure X = [1, 0, 2, ...], each element indicates the operation rate of each generator. Then, the objective function takes the execution measure X as input and the power demand based on the execution measure X as output.

[0072] In this case, by applying the optimization method of the embodiment, it is possible to derive the optimal operating rate for each generator in the power generation facility.

[0073] [Example 6] Fig. 13 shows an example in which the optimization method of the embodiment is applied to the field of communications. Specifically, a case in which the method is applied to minimizing delays in communications via a communications network will be described. In this case, the execution measure is to select one transmission route from multiple transmission routes. The objective function takes the execution measure X as input and the amount of delay occurring as a result of communications via each transmission route as output.

[0074] In this case, by applying the optimization method of the embodiment, it is possible to minimize communication delays in the communication network.

[0075] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0076] (Appendix 1) A means for acquiring a reward obtained by executing a certain measure; An update means for updating the probability distribution of the measures based on the obtained reward; A decision means for deciding a measure to be taken based on the updated probability distribution; Equipped with The updating means is an optimization device that uses a weighted sum of previously updated probability distributions as a constraint.

[0077] (Appendix 2) 2. The optimization device according to claim 1, wherein the update means updates the probability distribution using an update equation including a regularization term indicating a weighted sum of the probability distribution.

[0078] (Appendix 3) 3. The optimization device according to claim 2, wherein the regularization term is calculated by applying different weighting to each past probability distribution using a weight parameter indicating a strength of regularization.

[0079] (Appendix 4) 4. The optimization device according to claim 3, wherein the weighting parameters are calculated based on the degree of deviation of a predicted value of the loss.

[0080] (Appendix 5) The optimization device according to any one of Appendix 2 to 4, wherein the updating means updates the probability distribution based on a probability distribution based on a sum of an accumulation of estimated amounts of loss at past time steps and a predicted value of the loss at a current time step, and based on the regularization term.

[0081] (Appendix 6) 6. The optimization device according to claim 4 or 5, wherein the predicted value of the loss is calculated by reflecting the obtained reward in the previous time step with a predetermined coefficient.

[0082] (Appendix 7) Receive the rewards obtained by implementing a certain measure, Update the probability distribution of the measures based on the rewards obtained. Determine the action to be taken based on the updated probability distribution; An optimization method in which the probability distribution is updated using a weighted sum of previously updated probability distributions as a constraint.

[0083] (Appendix 8) Receive the rewards obtained by implementing a certain measure, Update the probability distribution of the measures based on the rewards obtained. Determine the action to be taken based on the updated probability distribution; A recording medium having recorded thereon a program for causing a computer to execute a process in which the probability distribution is updated using a weighted sum of previously updated probability distributions as a constraint.

[0084] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-mentioned embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0085] 12 processors 21 Input section 22 Calculation section 23 Memory section 24 Output section 100 Optimizer

Claims

1. A means for acquiring a reward obtained by executing a certain measure; An update means for updating a probability distribution used to determine which of a plurality of measures should be executed based on the acquired reward; A decision means for deciding a measure to be taken based on the updated probability distribution; Equipped with The updating means is an optimization device that uses a weighted sum of previously updated probability distributions as a constraint.

2. 2. The optimization device according to claim 1, wherein the update means updates the probability distribution using an update equation including a regularization term indicating a weighted sum of the probability distribution.

3. 3. The optimization device according to claim 2, wherein the regularization term is calculated by applying different weighting to each past probability distribution using a weight parameter indicating a strength of regularization.

4. The optimization device according to claim 3 , wherein the weighting parameters are calculated based on the degree of deviation of a predicted value of the loss.

5. 5. The optimization device according to claim 2, wherein the update means updates the probability distribution based on a probability distribution based on a sum of an accumulation of estimated amounts of loss at past time steps and a predicted value of the loss at a current time step, and based on the regularization term.

6. 6. The optimization device according to claim 4, wherein the predicted value of the loss is calculated by reflecting the obtained reward in the previous time step with a predetermined coefficient.

7. Receive the rewards obtained by implementing a certain measure, updating a probability distribution used to determine which of a plurality of measures should be executed based on the obtained reward; Determine the action to be taken based on the updated probability distribution; An optimization method in which the probability distribution is updated using a weighted sum of previously updated probability distributions as a constraint.

8. Receive the rewards obtained by implementing a certain measure, updating a probability distribution used to determine which of a plurality of measures should be executed based on the obtained reward; Determine the action to be taken based on the updated probability distribution; A program that causes a computer to execute a process in which the probability distribution is updated using a weighted sum of previously updated probability distributions as a constraint.

Citation Information

Patent Citations

  • Technology for determining optimal measures for obtaining remuneration

    JP2008040522A

  • Multimodal interactive program, system, and method taking account of input / output device information

    JP2012216007A

  • Information processing device, information processing method, and program

    JP2015191374A

  • Measure determination system, measure determination method, and measure determination program

    WO2019220479A1

  • Information processing system, information processing method, and storage medium

    WO2020012589A1