Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises: an acquisition means for sequentially acquiring elements from series data including a finite number of elements; a calculation means for calculating an index indicating to which of a plurality of classes the series data belongs, each time an element is acquired; a determination means for determining whether to end or continue sequential processing including at least the acquisition process performed by the acquisition means and the calculation operation performed by the calculation means by using a decision-making model; and a classification means for classifying the series data into one of the plurality of classes on the basis of the index when the determination means determines to end the sequential processing. The decision-making model is reinforcement-learned on the basis of a reward for a case where a behavior of either one of the end of the sequential processing or the continuation of the sequential processing is taken in a current state, the reward being represented by a result of the behavior.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] As an apparatus for classifying data, an apparatus that classifies data using a likelihood ratio that the data belongs to a class is known. For example, Patent Document 1 discloses an information processing apparatus including: an acquisition unit that sequentially acquires multiple elements included in sequence data; a first calculation unit that calculates an index indicating to which of multiple classes each of the multiple elements likely belongs, taking into account two or more of the multiple elements; a second calculation unit that integrates the indexes of the multiple elements to calculate an integrated index indicating to which of the multiple classes each of the multiple elements likely belongs; and a classification unit that classifies the sequence data into one of the classes based on the integrated index.

[0003] International Publication No. 2020 / 194497

[0004] This disclosure aims to improve upon the related art discussed above.

[0005] One aspect of the information processing device disclosed herein includes an acquisition means for sequentially acquiring elements from sequence data including a finite number of elements; a calculation means for calculating, each time an element is acquired, an index indicating to which of a plurality of classes the sequence data belongs; a decision means for using a decision-making model to decide whether to terminate or continue a sequential process including at least an acquisition process by the acquisition means and a calculation operation by the calculation means; and a classification means for classifying the sequence data into one of the plurality of classes based on the index when the decision means determines to terminate the sequential process, wherein the decision-making model is subjected to reinforcement learning based on a reward represented by a result of an action taken in a current state, either to terminate the sequential process or to continue the sequential process.

[0006] One aspect of the information processing method disclosed herein is an information processing method executed by an information processing device that includes: acquisition means for sequentially acquiring elements from sequence data containing a finite number of elements; calculation means for calculating, each time an element is acquired, an index indicating to which of multiple classes the sequence data belongs; and classification means for classifying the sequence data into one of the multiple classes based on the index when it is determined to terminate the sequential processing, which includes at least the acquisition process by the acquisition means and the calculation operation by the calculation means.The information processing method determines whether to terminate or continue the sequential processing using a decision-making model that has been reinforced learned based on a reward represented by the result of taking either of the actions of terminating the sequential processing or continuing the sequential processing in the current state.

[0007] One aspect of the recording medium disclosed herein is an information processing method executed by an information processing device including: acquisition means for sequentially acquiring elements from sequence data containing a finite number of elements; calculation means for calculating, for each acquired element, an index indicating to which of multiple classes the sequence data belongs; and classification means for classifying the sequence data into one of the multiple classes based on the index when it is determined to terminate the sequential processing, which includes at least the acquisition process by the acquisition means and the calculation operation by the calculation means. The recording medium has recorded thereon a computer program that causes a computer to execute the information processing method, which determines whether to terminate or continue the sequential processing using a decision-making model that has been reinforced learned based on a reward represented by the outcome of either of the actions of terminating or continuing the sequential processing in a current state.

[0008] FIG. 1 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing the flow of a classification operation in an information processing device according to the present disclosure. FIG. 3 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 4 is a flowchart showing the flow of a classification operation in an information processing device according to the present disclosure. FIG. 5 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 6 is a block diagram showing the configuration of an information processing device according to the present disclosure.

[0009] Hereinafter, an information processing device, an information processing method, and a recording medium according to an embodiment will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 according to this disclosure. [1-1: Configuration of Information Processing Device 1]

[0011] The configuration of an information processing device 1 according to this disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this disclosure.

[0012] 1, the information processing device 1 includes a calculation device 11, a storage device 12, and a communication device 13. The calculation device 11, the storage device 12, and the communication device 13 may be connected via a data bus 16.

[0013] The arithmetic device 11 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0014] The arithmetic device 11 reads a computer program 121 including at least one of computer program code and computer program instructions. For example, the arithmetic device 11 may read the computer program 121 stored in the storage device 12. For example, the arithmetic device 11 may read the computer program 121 stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 1. The computer program 121 read from the recording medium may be stored in the storage device 12. The arithmetic device 11 may acquire (i.e., download or read) the computer program 121 from a device (not shown) located outside the information processing device 1 via the communication device 13 (or another communication device). The downloaded computer program 121 may be stored in the storage device 12.

[0015] The arithmetic device 11 executes the loaded computer program 121. As a result, a logical functional block for executing processing to be performed by the information processing device 1 (e.g., authentication processing described below) is realized within the arithmetic device 11. In other words, the arithmetic device 11, together with the storage device 12 in which the computer program 121 is recorded (in other words, together with the storage device 12 and the computer program 121 recorded in the storage device 12, etc.), can function as a controller or computer for realizing the logical functional block for executing processing to be performed by the information processing device 1. In other words, the at least one processor included in the arithmetic device 11, the memory (recording medium) included in the storage device 12, etc., and the computer program 121 are configured to cause the information processing device 1 to perform processing to be performed by the information processing device 1 (e.g., authentication processing described below). The arithmetic device 11 may output information to another computer, cloud server, or other device (not shown) provided outside the information processing device 1 via the communication device 13 (or other communication device).

[0016] The recording medium for recording the computer program 121 executed by the arithmetic device 11 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, and Blu-ray (registered trademark) optical disk, magnetic medium such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 121 (for example, a general-purpose device or a dedicated device in which the computer program 121 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 121 may be realized by a logical processing block realized within the arithmetic device 11 when the arithmetic device 11 (i.e., processor) executes the computer program 121, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 11, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0017] A decision-making model that can be constructed by reinforcement learning is implemented within the arithmetic device 11 by the arithmetic device 11 executing the computer program 121. An example of a decision-making model that can be constructed by reinforcement learning is a decision-making model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the decision-making model may include learning of parameters of the neural network (e.g., at least one of weights and biases). The arithmetic device 11 makes at least a decision using the decision-making model. A decision-making model that has been constructed by reinforcement learning may be implemented in the arithmetic device 11. A decision-making model that has been constructed by offline reinforcement learning using learning data may be implemented in the arithmetic device 11. Furthermore, the decision-making model implemented in the arithmetic device 11 may be updated by online learning on the arithmetic device 11. Alternatively, the computing device 11 may make decisions using a decision-making model implemented in a device external to the computing device 11 (i.e., a device provided outside the information processing device 1) in addition to or instead of the decision-making model implemented in the computing device 11.

[0018] 1 shows an example of logical functional blocks implemented in a computing device 11 for performing information processing. As shown in FIG. 1, an acquisition unit 111, a calculation unit 112, a determination unit 113, and a classification unit 114 are implemented in the computing device 11. The determination unit 113 makes decisions using the decision-making model described above. Note that the processes performed by the acquisition unit 111, the calculation unit 112, the determination unit 113, and the classification unit 114 will be described with reference to FIG. 2.

[0019] The storage device 12 includes at least one memory capable of storing desired data. In other words, the storage device 12 includes at least one memory containing desired data. For example, the storage device 12 may store a computer program 121 executed by the arithmetic device 11. In this case, the storage device 12 (memory) may be used as the above-mentioned recording medium for recording the computer program 121 executed by the arithmetic device 11. The storage device 12 may temporarily store data used by the arithmetic device 11 when the arithmetic device 11 is executing the computer program 121. The storage device 12 may also store data to be stored long-term by the information processing device 1. The storage device 12 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 12 may include a non-temporary recording medium.

[0020] The communication device 13 can communicate with devices external to the information processing device 1 or 2 via a communication network (not shown). The communication device 13 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus). [1-2: Information Processing Method Executed by the Information Processing Device 1]

[0021] An information processing method executed by the information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 1.

[0022] 2 , the acquiring unit 111 sequentially acquires elements from sequence data containing a finite number of elements (step S11). Each time an element is acquired, the calculating unit 112 calculates an index indicating to which of multiple classes the sequence data belongs (step S12).

[0023] The decision unit 113 uses the decision-making model to decide whether to terminate or continue the sequential processing (step S13). The sequential processing includes at least an acquisition process by the acquisition unit 111 and a calculation operation by the calculation unit 112.

[0024] When an index is input, the decision-making model outputs information indicating at least one of the likelihood that the sequential processing should be terminated and the likelihood that the sequential processing should be continued. In this disclosure, "terminate" refers to the action of making a decision using the index calculated at the current time. In addition, in this disclosure, "continue" refers to the action of not making a decision using the index calculated at the current time, but acquiring further elements and calculating an index.

[0025] The decision-making model is reinforced learning based on the reward for taking either the action of terminating the sequential process or the action of continuing the sequential process in the current state. The reward is expressed by the result of the action.

[0026] If it is determined that the sequential processing is to be terminated (step S13: end), the classification unit 114 classifies the sequence data into one of a plurality of classes based on the index (step S14). If it is determined that the sequential processing is to be continued (step S13: continue), the process returns to step S11. That is, the information processing device 1 repeats the sequential processing until the determination unit 113 determines that the sequential processing is to be terminated. [1-3: Technical Effects of the Information Processing Device 1]

[0027] The information processing device 1 according to this disclosure can perform classification operations at an appropriate time by using a decision-making model constructed by reinforcement learning. The appropriate time means at least a time when appropriate classification (i.e., accurate classification) can be performed. [2: Second Embodiment]

[0028] A second embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the second embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 2 according to this disclosure. [2-1: Configuration of Information Processing Device 2]

[0029] The configuration of the information processing device 2 according to this disclosure will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2 according to this disclosure.

[0030] 3 , the information processing device 2 may further include an input device 14 and an output device 15 in addition to the calculation device 11, the storage device 12, and the communication device 13. However, the information processing device 2 does not necessarily have to include at least one of the input device 14 and the output device 15. The calculation device 11, the storage device 12, the communication device 13, the input device 14, and the output device 15 may be connected via a data bus 16.

[0031] 3, the calculation unit 212 may include an index calculation unit 2121 and an integrated index calculation unit 2122. The storage device 12 may also include an index holding unit 222.

[0032] The input device 14 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 14 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 14 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0033] The output device 15 is a device that outputs information to the outside of the information processing device 2. For example, the output device 15 may output information as an image. That is, the output device 15 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 15 may output information as sound. That is, the output device 15 may include an audio device (a so-called speaker) that can output sound. For example, the output device 15 may output information on paper. That is, the output device 15 may include a printing device (a so-called printer) that can print desired information on paper. [2-2: Information Processing Method Executed by Information Processing Device 2]

[0034] The information processing device 2 is configured as a device for classifying time-series data. For example, the information processing device 2 may be configured as a device that acquires images of time-series data and classifies the types of objects included in the images.

[0035] The information processing method executed by the information processing device 2 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the information processing method executed by the information processing device 2.

[0036] 4, when the information processing operation by the information processing device 2 starts, the current time t becomes 0 (step S20). In the information processing method described with reference to FIG. 4, the current time t is used as a counter for counting the number of times an element is acquired.

[0037] The acquisition unit 211 acquires an element included in the sequence data, and the current time t is incremented (step S21). The number of the current time t indicates the number of acquisitions that have occurred at the current time.

[0038] The sequence data includes an upper limit number T of elements. The upper limit number T is determined according to requirements. For example, the upper limit number T may be 10. The acquisition unit 211 sequentially acquires elements included in the sequence data. For example, the acquisition unit 211 acquires elements included in the sequence data one by one in order. An "element" may be considered a "unit" acquired by the acquisition unit 211 in a single acquisition. For example, if the sequence data is video data, an element may be one frame or multiple frames. What is considered an element may be determined arbitrarily according to the requirements to which the classification operation is applied.

[0039] The acquisition unit 211 may acquire data directly from any data acquisition device (for example, a camera, a microphone, etc.). Alternatively, the acquisition unit 211 may read data that has been acquired in advance by a data acquisition device and stored in storage, etc. When acquiring data from a camera, the acquisition unit 211 may be configured to acquire data from each of a plurality of cameras.

[0040] The index calculation unit 2121 calculates an index for the current time t based on the elements acquired by the acquisition unit 211 (step S22). The "index" here is a value indicating to which of multiple classes that are classification candidates the sequential data belongs. In this embodiment, the "index" used is a "likelihood ratio" that indicates the likelihood that the sequential data belongs to one of multiple classes. The calculation unit 212 may calculate a likelihood ratio corresponding to each of the multiple classes. The index calculation unit 2121 may calculate the likelihood ratio based on the elements acquired at the current time t and the elements acquired before the current time t. The index calculation unit 2121 stores the calculated likelihood ratio in the index storage unit 222.

[0041] The integrated index calculation unit 2122 calculates an integrated index for the current time t (step S23). The integrated index calculation unit 2122 calculates an integrated index for the current time t by integrating the integrated index calculated before the current time t and the index calculated in step S23. The integrated index calculation unit 2122 stores the calculated integrated index in the index storage unit 222.

[0042] The likelihood ratio may be a value that can be obtained as a result of likelihood ratio learning using a SPRT-based algorithm that treats sequence data as an Nth-order Markov process (SPRT-TANDEM). The calculation unit 212 may be configured, for example, by a trained neural network.

[0043] The calculation unit 212 may calculate the logarithm of the likelihood ratio (Log Likelihood Ratio (LLR)). In the following description, the log likelihood ratio may be simply referred to as the likelihood ratio.

[0044] The determination unit 213 determines whether the current time t indicates the upper limit number T (step S24). In other words, the determination unit 213 determines whether all of the elements of the upper limit number T have been acquired and the likelihood ratios have been calculated. If all of the elements of the upper limit number T have been acquired and the likelihood ratios have been calculated (step S24: Yes), the determination unit 213 determines to end the sequential processing.

[0045] The decision unit 213 uses the decision-making model to decide whether to terminate or continue the sequential processing (step S25). When the output of the calculation unit 212 is input, the decision-making model outputs information indicating at least one of the likelihood of terminating the sequential processing and the likelihood of continuing the sequential processing. An action in this disclosure consists of two elements: terminating the sequential processing and continuing the sequential processing. The decision-making model outputs information indicating which action to take, terminating the sequential processing or continuing the sequential processing. In addition to the likelihood ratio, an internal feature value of the calculation unit 212 may be input to the decision-making model. The internal feature value of the calculation unit 212 is a quantity indicating a feature of the internal state of the calculation unit 212 at the current time t.

[0046] The decision-making model is reinforced learning based on the reward for taking either the action of terminating the sequential process or the action of continuing the sequential process at the current state t. The current state t is represented by the likelihood ratio calculated at the current state t. The current state t is a point in time relative to the final point in time at which the upper limit number T is obtained.

[0047] The reward is expressed by the result of the action. The result of the action includes the classification result and the time when it is decided to end the sequential processing. In other words, the reward can be expressed using the classification result and the number of times an element is acquired when it is decided to end the sequential processing. The classification result is expressed by the correctness of the classification, that is, the classification accuracy.

[0048] If it is determined to continue the sequential processing (step S25: continue), the process returns to step S21. If it is determined to end the sequential processing (step S25: end and step S23: Yes), the process proceeds to step S26.

[0049] The classification unit 214 classifies the sequence data into one of a plurality of classes based on the likelihood ratio (step S26). The classification unit 214 may classify the sequence data into the class corresponding to the largest likelihood ratio among the plurality of classes. For example, when classifying the sequence data into two classes, class 0 and class 1, if the likelihood ratio corresponding to class 0 is larger than the likelihood ratio corresponding to class 1, the classification unit 214 classifies the sequence data into class 0. Similarly, if the likelihood ratio corresponding to class 1 is larger than the likelihood ratio corresponding to class 0, the classification unit 214 classifies the sequence data into class 1. Note that the plurality of classes may be three or more.

[0050] In this way, the information processing device 2 performs sequential operations to classify sequence data into appropriate classes. The calculation unit 212 is configured to calculate a larger likelihood ratio for the class to be classified as the more elements are acquired. [2-3: Technical Effects of the Information Processing Device 2]

[0051] The information processing device 2 according to this disclosure can perform classification operations at appropriate times because a decision-making model constructed by reinforcement learning makes decisions. [3: Third Embodiment]

[0052] A third embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the third embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 3 according to this disclosure. The information processing device 3 in the third embodiment prepares a decision-making model that receives as input a likelihood ratio calculated using a likelihood ratio estimation technique. [3-1: Configuration of the information processing device 3]

[0053] As shown in FIG. 5, the information processing device 3 further includes a learning unit 315 implemented in the arithmetic device 11. The learning unit 315 learns a decision-making model by reinforcement learning. The learning unit 315 includes a gradient calculation unit 3151 and a value calculation unit 3152. The storage device 12 may also include a learning process holding unit 323. Other features of the information processing device 3 may be the same as other features of the information processing device 2. [3-2: Information Processing Operations Performed by the Information Processing Device 3]

[0054] The learning unit 315 uses data obtained by combining sequence data with correct answer information indicating to which of a plurality of classes the sequence data belongs as learning data. The learning unit 315 uses the likelihood ratio calculated by the calculation unit 312 as a state in reinforcement learning. [a: Reward]

[0055] The learning unit 315 learns the decision-making model so as to maximize the reward. The learning unit 315 constructs the decision-making model using a reinforcement learning algorithm so as to maximize an arbitrary reward function. The reward is designed to increase as the decision is made more correctly and as the number of sequential processes is reduced. In other words, the reward is designed to increase as the classification accuracy e increases and as the end time is earlier. The reward may be expressed as a function f(e, t) of the classification accuracy e and the current time t. The reward function may be expressed using a likelihood ratio or a posterior probability instead of the classification accuracy e. The reward function may also be defined by adding variables other than the classification accuracy e and the current time t.

[0056] Alternatively, the learning unit 315 may construct a decision-making model using a reinforcement learning algorithm so as to minimize an objective function such as a cost function or an error function.

[0057] The information processing device 3 may give a reward that is a negative value of the risk associated with decision-making. The risk associated with decision-making may include the risk of incorrectly classifying sequential data and the risk of spending time classifying sequential data. [b: Policy]

[0058] The mechanism that associates states with actions is called a policy. A policy is a decision-making rule. A policy can be expressed as a probability distribution of possible actions depending on the state. A decision-making model operates according to the policy. A decision-making model associates states with actions according to the policy. When a state is input, the decision-making model outputs information indicating the likelihood of terminating the sequential processing and the likelihood of continuing the sequential processing according to the policy.

[0059] To calculate the expected reward, we need the probability of taking each action in addition to the reward that can be obtained by taking each action. The policy determines these probabilities.

[0060] The learning unit 315 obtains a policy that maximizes the reward resulting from the action according to the state. The learning unit 315 optimizes the policy. The learning unit 315 adjusts the policy based on the state, the action, and the reward. In other words, the learning unit 315 adjusts the parameters of the policy. The decision-making model makes decisions using the adjusted policy and the output of the calculation unit 212. [c: Reinforcement learning using the policy gradient method]

[0061] As described above, the learning unit 315 optimizes the policy so as to maximize the reward. The accumulation of reward R when an action is determined from a state based on the policy is called profit Gt. The profit Gt expected at the current time t may be determined, for example, as shown in the following formula 1. [Formula 1]

[0062] γ is a discount factor. The discount factor γ reduces the contribution of future values. In other words, by multiplying the reward R by the discount factor γ, the influence of the reward R is designed to become smaller as the time t moves further into the future.

[0063] The learning unit 315 optimizes the policy to maximize the profit Gt. The learning unit 315 may optimize the policy from the gradient of the profit Gt. The learning unit 315 may optimize the policy to minimize the gradient of the profit Gt. The learning unit 315 may determine an objective function that represents the gradient of the profit Gt, and optimize the policy to minimize the objective function.

[0064] The gradient of the profit Gt expressed by the above formula 1 can be expressed as the following formula 2. [Formula 2]

[0065] That is, the learning unit 315 minimizes a function such as the above-mentioned formula 2 to obtain the policy π θ can be optimized. ｔ indicates the state S at the current time t, and Aｔ indicates an action A that can be taken at the current time t. The learning unit 315 θ Based on the gradient of θ The learning unit 315 optimizes the policy π to maximize the expected profit Gt. θ In other words, the learning unit 315 uses the gradient of the profit Gt and the policy π according to the above formula 2. θ A decision-making model is constructed based on the gradient of

[0066] The gradient calculation unit 3151 calculates the policy π θ The gradient calculation unit 3151 calculates the gradient of the policy π θ The gradient calculation unit 3151 may calculate the gradient of the logarithm of the policy π at the current time t. θ The gradient of is stored in the learning process storage unit 323.

[0067] The value calculation unit 3152 calculates a value. The value calculated by the value calculation unit 3152 is an accumulated profit of a reward for the current time t and a reward for the future. The value calculation unit 3152 stores the value for the current time t in the learning process storage unit 323.

[0068] The learning unit 315 constructs a decision-making model based on the information stored in the learning process storage unit 323. The learning unit 315 θ A decision-making model is constructed by adjusting the parameter θ in the above equation. The decision-making model learns to associate states, actions, and rewards.

[0069] The policy gradient method is achieved by maximizing the expected value of an arbitrary reward R. The policy gradient method can learn a decision-making model using a technique such as backpropagation. When the policy gradient method is used, the reward is determined when a decision is made by SPRT. [3-3: Technical Effects of the Information Processing Device 3]

[0070] The information processing device 3 according to this disclosure can construct a decision-making model that can make correct decisions with less sequential processing by performing reinforcement learning using the policy gradient method.

[0071] Reinforcement learning increases the degree of freedom in reward design. For example, it is possible to design a reward function of the type f(e, t) described above. Furthermore, particularly when there is sufficient training data, reinforcement learning has a high possibility of obtaining a decision-making model that can make correct decisions with fewer sequential processes. [4: Fourth Embodiment]

[0072] Fourth Embodiment of Information Processing Apparatus, Information Processing Method, and Recording Medium Will Be Described Hereinafter, a fourth embodiment of an information processing apparatus, an information processing method, and a recording medium will be described using an information processing apparatus 4 according to this disclosure.

[0073] Similar to the information processing device 3, the information processing device 4 is configured as a device for preparing a decision-making model that receives as input likelihood ratios calculated using likelihood ratio estimation technology. The information processing device 4 differs from the information processing device 3 in the operation of the learning unit 415. The learning unit 415 performs reinforcement learning using an actor-critic method. The actor-critic method predicts the current reward simultaneously with an action. The learning unit 415 has a gradient calculation unit 4151, a state value function prediction unit 4153, and a reward prediction unit 4154. [4-1: Information Processing Method Executed by the Information Processing Device 4]

[0074] The learning unit 415 learns the policy π θ At the same time, the state value function V ω The state value function V ω is the policy π θ Under this condition, the state S ｔ The state value function V ω is the policy π θ The state value function V ω is predicted by the neural network during the training process.

[0075] The learning unit 415 calculates the state value function V ω The learning unit 415 performs learning so as to maximize the expected value of the cumulative reward represented by the state value function V ω The error of the expected value of the cumulative reward represented by policy π θSpecifically, the learning unit 415 may minimize a function such as that expressed in the following formula 3. [Formula 3]

[0076] The gradient calculation unit 4151 calculates the policy π θ The gradient calculation unit 4151 calculates the gradient of the policy π θ The gradient calculation unit 4151 may calculate the gradient of the logarithm of the policy π at the current time t. θ The gradient of is stored in the learning process storage unit 423.

[0077] The state value function prediction unit 4153 includes a neural network and predicts the state value function V ω The state value function prediction unit 4153 predicts the state value function V ω The learning process storage unit 423 stores information representing the state value function V ω The parameter ω may be stored in the learning process storage unit 423.

[0078] The reward prediction unit 4154 predicts the reward. The reward prediction unit 4154 predicts the reward using the predicted state value function V ω The state-value function predicting unit 4153 stores information about the predicted reward in the learning process storage unit 423.

[0079] The learning unit 415 constructs a decision-making model based on the information stored in the learning process storage unit 423. The learning unit 415 calculates the state value function V ω parameter ω and policy π θ A decision-making model is constructed by adjusting the parameter θ. [4-2: Technical Effects of Information Processing Device 4]

[0080] The information processing device 4 according to this disclosure can construct a decision-making model that can make correct decisions with fewer sequential processes by performing reinforcement learning using the actor-critic method. [5: Fifth Embodiment]

[0081] A fifth embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, a fifth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 5 according to this disclosure.

[0082] Similar to the information processing device 3 and the information processing device 4, the information processing device 5 is configured as a device for preparing a decision-making model that receives as input a likelihood ratio calculated using a likelihood ratio estimation technique. The information processing device 5 differs from the information processing device 3 and the information processing device 4 in the operation of the learning unit 515. The learning unit 515 performs reinforcement learning using Q-learning. The learning unit 515 has an action value function prediction unit 5155 and a reward calculation unit 5156. [5-1: Information Processing Method Executed by Information Processing Device 5]

[0083] The learning unit 515 learns the action value function Q simultaneously with the policy π. The action value function Q represents the expected value of the cumulative reward when an action A is taken in a state S under the policy π. The action value function Q is predicted in the learning process by a neural network.

[0084] The learning unit 515 performs learning so as to maximize the expected value of the cumulative reward represented by the action value function Q. The learning unit 515 may also perform learning so as to minimize the error in the expected value of the cumulative reward represented by the action value function Q. Specifically, the learning unit 515 may minimize a function such as that expressed in the following formula 4. [Formula 4]

[0085] The action value function prediction unit 5155 includes a neural network and predicts the action value function Q. The action value function prediction unit 5155 causes the learning process storage unit 523 to store information representing the action value function Q.

[0086] The reward calculation unit 5156 calculates the reward. The reward calculation unit 5156 calculates the reward based on the predicted action value function Q. The reward calculation unit 5156 may calculate the reward based on the predicted action value function Q and determine the error of the reward. The reward calculation unit 5156 stores information about the calculated reward in the learning process storage unit 523.

[0087] The learning unit 515 constructs a decision-making model based on the information stored in the learning process storage unit 523. [5-2: Technical Effects of the Information Processing Device 5]

[0088] The information processing device 5 according to this disclosure can construct a decision-making model that can make correct decisions with less sequential processing by performing reinforcement learning using Q-learning.

[0089] [6: Supplementary Note] The above-described embodiment can be further described as, but not limited to, the following supplementary note: [Supplementary Note 1] An information processing device comprising: acquisition means for sequentially acquiring elements from sequence data including a finite number of elements; calculation means for calculating, for each acquired element, an index indicating to which of multiple classes the sequence data belongs; decision means for using a decision-making model to decide whether to terminate or continue sequential processing including at least the acquisition process by the acquisition means and the calculation operation by the calculation means; and classification means for classifying the sequence data into one of the multiple classes based on the index when the decision means decides to terminate the sequential processing, wherein the decision-making model has undergone reinforcement learning based on rewards represented by results of actions taken in a current state, either to terminate the sequential processing or to continue the sequential processing. [Supplementary Note 2] The information processing device according to Supplementary Note 1, wherein, when the index is input, the decision-making model outputs information indicating at least one of the likelihood of terminating the sequential processing and the likelihood of continuing the sequential processing. [Supplementary Note 3] The information processing device according to Supplementary Note 1, wherein the current state is represented by an index calculated at the current time. [Supplementary Note 4] The information processing device according to Supplementary Note 1, wherein the result of the action includes a classification result and a time point at which it is decided to end the sequential processing. [Supplementary Note 5] The information processing device according to Supplementary Note 4, wherein the classification result is represented by the correctness of the classification. [Supplementary Note 6] The information processing device according to Supplementary Note 1, wherein the decision means decides to end the sequential processing when all of the finite number of elements have been obtained and the index has been calculated. [Supplementary Note 7] The information processing device according to Supplementary Note 1, further comprising: learning means that learns the decision-making model by reinforcement learning so as to maximize the reward. [Supplementary Note 8] The information processing device according to Supplementary Note 7, wherein the learning means learns the decision-making model by reinforcement learning so as to optimize a policy that associates the state with the action.[Supplementary Note 9] The information processing device according to Supplementary Note 8, wherein the policy is represented by a probability distribution of the actions that can be taken depending on the state, and the decision-making model operates according to the policy. [Supplementary Note 10] The information processing device according to Supplementary Note 8, wherein the learning means optimizes the policy based on a gradient of the policy. [Supplementary Note 11] The information processing device according to Supplementary Note 8, wherein the learning means optimizes the policy so as to minimize the gradient of the policy. [Supplementary Note 12] The information processing device according to Supplementary Note 7, wherein the learning means learns the decision-making model by reinforcement learning so as to maximize the reward represented by a state value function that is a function of the state. [Supplementary Note 13] The information processing device according to Supplementary Note 12, wherein the learning means learns the decision-making model by reinforcement learning so as to minimize an error in the reward represented by the state value function. [Supplementary Note 14] The information processing device according to Supplementary Note 7, wherein the learning means learns the decision-making model by reinforcement learning so as to maximize the reward represented by an action value function that is a function of the state and the action. [Supplementary Note 15] The information processing device according to Supplementary Note 14, wherein the learning means learns the decision-making model by reinforcement learning so as to minimize an error in the reward represented by the action value function. [Supplementary Note 16] The information processing device according to Supplementary Note 1, wherein the classification means classifies the series of data into a class corresponding to the largest index among the plurality of classes. [Supplementary Note 17] The information processing device according to Supplementary Note 1, wherein the calculation means includes: first calculation means for calculating a first index based on the acquired elements; and second calculation means for calculating a second index by integrating the index calculated by the calculation operation before the current time and the first index, and outputs the second index as the index. [Supplementary Note 18] The information processing device according to Supplementary Note 1, wherein the index is a likelihood ratio indicating the likelihood that the series of data belongs to a certain class of the plurality of classes.[Supplementary Note 19] An information processing method executed by an information processing device comprising: acquisition means for sequentially acquiring elements from sequence data including a finite number of elements; calculation means for calculating, each time the element is acquired, an index indicating to which of a plurality of classes the sequence data belongs; and classification means for classifying the sequence data into one of the plurality of classes based on the index when it is determined to terminate the sequential processing including at least the acquisition process by the acquisition means and the calculation operation by the calculation means, wherein the information processing method determines whether to terminate or continue the sequential processing using a decision-making model that has been reinforced learned based on a reward represented by a result of taking either of the actions of terminating the sequential processing or continuing the sequential processing in a current state. [Supplementary Note 20] An information processing method executed by an information processing device comprising: acquisition means for sequentially acquiring elements from sequence data including a finite number of elements; calculation means for calculating, each time the element is acquired, an index indicating to which of a plurality of classes the sequence data belongs; and classification means for classifying the sequence data into one of the plurality of classes based on the index when it is determined to terminate the sequential processing including at least the acquisition process by the acquisition means and the calculation operation by the calculation means, wherein the information processing method determines whether to terminate or continue the sequential processing using a decision-making model that has been reinforced learned based on a reward represented by a result of either of the actions of terminating the sequential processing or continuing the sequential processing in a current state.

[0090] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0091] 1, 2, 3, 4, 5 Information processing device 111, 211 Acquisition unit 112, 212 Calculation unit 113, 213 Determination unit 114, 214 Classification unit 2121 Index calculation unit 2122 Integrated index calculation unit 222 Index storage unit 315, 415, 515 Learning unit 323, 423, 523 Learning process storage unit 3151, 4151 Gradient calculation unit 3152 Value calculation unit 4153 State value function prediction unit 4154 Reward prediction unit 5155 Action value function prediction unit 5156 Reward calculation unit

Claims

1. An information processing device comprising: acquisition means for sequentially acquiring elements from sequence data containing a finite number of elements; calculation means for calculating, each time an element is acquired, an index indicating to which of multiple classes the sequence data belongs; decision means for using a decision-making model to decide whether to terminate or continue sequential processing including at least the acquisition process by the acquisition means and the calculation operation by the calculation means; and classification means for classifying the sequence data into one of the multiple classes based on the index when the decision means decides to terminate the sequential processing, wherein the decision-making model has undergone reinforcement learning based on a reward represented by the result of the action taken in the current state, either to terminate the sequential processing or to continue the sequential processing.

2. The information processing device according to claim 1, wherein the decision-making model, when inputted with the index, outputs information indicating at least one of the likelihood that the sequential processing should be terminated and the likelihood that the sequential processing should be continued.

3. The information processing device according to claim 1, wherein the current state is represented by an index calculated at the current time.

4. The information processing device according to claim 1, wherein the results of the behavior include a classification result and a time point at which it is determined to terminate the sequential processing.

5. The information processing device according to claim 4, wherein the classification result is expressed as the correctness of the classification.

6. The information processing device according to claim 1, wherein the determining means determines to end the sequential processing when all of the finite number of elements have been acquired and the index has been calculated.

7. The information processing device according to claim 1, further comprising: learning means for learning the decision-making model by reinforcement learning so as to maximize the reward.

8. The information processing device according to claim 7, wherein the learning means learns the decision-making model by the reinforcement learning so as to optimize a policy that associates the state with the action.

9. The information processing device according to claim 8, wherein the policy is expressed as a probability distribution of the actions that can be taken depending on the state, and the decision-making model operates in accordance with the policy.

10. The information processing device according to claim 8, wherein the learning means optimizes the policy based on a gradient of the policy.

11. The information processing device according to claim 8, wherein the learning means optimizes the policy so as to minimize the gradient of the policy.

12. The information processing device according to claim 7, wherein the learning means learns the decision-making model by the reinforcement learning so as to maximize the reward represented by a state value function that is a function of the state.

13. The information processing device according to claim 12, wherein the learning means learns the decision-making model by the reinforcement learning so as to minimize an error in the reward represented by the state value function.

14. The information processing device according to claim 7, wherein the learning means learns the decision-making model by the reinforcement learning so as to maximize the reward represented by an action value function that is a function of the state and the action.

15. The information processing device according to claim 14, wherein the learning means learns the decision-making model by the reinforcement learning so as to minimize an error in the reward represented by the action value function.

16. The information processing device according to claim 1, wherein the classification means classifies the sequence data into a class corresponding to the largest index among the plurality of classes.

17. The information processing device according to claim 1, wherein the calculation means includes: a first calculation means for calculating a first index based on the acquired element; and a second calculation means for calculating a second index by integrating the index calculated by the calculation operation before the current time with the first index; and outputs the second index as the index.

18. The information processing device according to claim 1, wherein the index is a likelihood ratio indicating the likelihood that the sequence data belongs to a certain class of the plurality of classes.

19. An information processing method executed by an information processing device comprising: acquisition means for sequentially acquiring elements from sequence data containing a finite number of elements; calculation means for calculating, each time an element is acquired, an index indicating to which of multiple classes the sequence data belongs; and classification means for classifying the sequence data into one of the multiple classes based on the index when it is determined to terminate the sequential processing, which includes at least the acquisition process by the acquisition means and the calculation operation by the calculation means; wherein the information processing method determines whether to terminate or continue the sequential processing using a decision-making model that has been reinforced learned based on a reward represented by the result of taking either of the actions of terminating the sequential processing or continuing the sequential processing in the current state.

20. An information processing method executed by an information processing device comprising: acquisition means for sequentially acquiring elements from sequence data containing a finite number of elements; calculation means for calculating, for each acquired element, an index indicating to which of multiple classes the sequence data belongs; and classification means for classifying the sequence data into one of the multiple classes based on the index when it is determined to terminate the sequential processing, which includes at least the acquisition process by the acquisition means and the calculation operation by the calculation means; wherein a recording medium having recorded thereon a computer program for causing a computer to execute the information processing method, the information processing method determining whether to terminate or continue the sequential processing using a decision-making model that has been reinforced learned based on a reward represented by the result of taking either of the actions of terminating or continuing the sequential processing in the current state.

Images