Self-adapting random multi-arm decision-problem calculation method and device thereof

Abstract

The invention discloses a self-adapting random multi-arm decision-problem calculation method and a device thereof. The method includes the steps that: an estimated value of each motion and the number of selection times of each motion are initialized; according to the estimated value of each motion and the number of the selection times of each motion, the number m of the selection times of the motion which has the smallest estimated value is determined; exploration is conducted according to the probability w / (w+m<2>), and utilization is conducted according to the probability 1-w / (w+m<2>); w is a preset algorithm parameter, and the exploration operation means that from the motions which currently have the smallest number of the selection times, a motion is randomly selected as a motion of the next time step, and the utilization operation means that the motion which currently has the greatest estimated value is selected as a motion of the next time step; after the motion selection of the next time step is completed, a random award is generated; according to the random award and the selected motions, the estimated value of each motion and the number of selection times of each motion are updated, the m is determined again, until all the motions are completed, the sum of the obtained random awards in a preset greatest time step is counted, and an accumulative award is obtained. The method and the device can balance the exploration selection and the utilization selection and ensure final decision-making effects, and are wide in application range.

Description

technical field [0001] The invention relates to the field of stochastic multi-way selection learning optimization, in particular to an adaptive stochastic multi-arm decision-making problem calculation method and a device thereof. Background technique [0002] The multi-arm decision-making problem is a classic problem in researching the balance between exploration and utilization in reinforcement learning. It was first used in diagnosis and treatment trials. In recent years, the application of multi-arm decision-making has become more and more extensive, and can be applied to recommender systems, crowdsourcing, and smart grids. [0003] Random multi-arm decision-making problem is a classic multi-arm decision-making problem, and it is the basis of many new multi-arm decision-making problems. A random multi-arm decision-making problem includes K arms, one arm corresponds to an action, and an action is selected at each time step. After each action is selected, a random reward w...

AI Technical Summary

Problems solved by technology

Soft maximization only grades the selection probability of each action based on the estimated value of the currently known action. If an action randomly gets a very low reward in the early stage, the selection probability of the action will be very low, which will lead to the The action may not be selected later, but the action may have a high reward in the later stage, so the cumulative reward obtained at the end is not the best

It can be seen that the decision-making effect of ε-greedy and soft maximization is not ideal

[0006] However, the confidence upper bound action selection method makes full use of the estimated value of the action and the number of times it is selected, and directly calculates the action to be selected each time based on the existing information, and its computational burden is relatively large.

However, the confidence upper bound action selection method must select all actions in turn at the beginning of the experiment, so when the number of experiments is less than or equal to the number of actions, the confidence upper bound action selection method will not be applicable

Images