Information processing system, method, and program

Abstract

The present invention relates to an information processing system, method and program which enable determining performing learning independent of the input of external instructions. This information processing system determines actions on the basis of environmental information and a learning model obtained by learning based on an evaluation function for evaluating actions. This information processing system is provided with: an error detection unit which calculates the size of the difference between newly inputted environmental information or evaluation function, and a known environmental information or evaluation function; and a learning unit which, depending on the size of the difference, updates the learning model on the basis of the newly inputted environmental information or evaluation function and the reward quantity obtained by evaluation depending on the action. The present technology can be applied to information processing systems.

Description

[Technical Field]

[0001] The present technology relates to an information processing system, method, and program, and more particularly to an information processing system, method, and program that can determine whether to execute learning without relying on an external instruction input. [Background technology]

[0002] Reinforcement learning is a well-known technique that uses environmental information indicating the surrounding environment as input and learns appropriate behavior in response to the input.

[0003] As a technology related to reinforcement learning, a technology has been proposed that realizes efficient reinforcement learning by using sub-reward setting information based on annotations input by the user in addition to the state, behavior, and reward of an agent (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 150654 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, however, there has been a demand for agents to automatically switch the learning target themselves, i.e., to autonomously decide whether or not to perform reinforcement learning on a learning model, without relying on external instruction input.

[0006] However, with the above-mentioned technology, it is necessary to prepare data and evaluation functions for learning each time, and the agent itself cannot voluntarily switch the learning target.

[0007] The present technology has been made in view of such circumstances, and makes it possible to determine whether to execute learning without relying on an external instruction input. [Means for solving the problem]

[0008] An information processing system according to one aspect of the present technology is an information processing system that determines behavior based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating behavior, and includes an error detection unit that determines the magnitude of the difference between newly input environmental information or the evaluation function and existing environmental information or the evaluation function, and a learning unit that updates the learning model based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation in accordance with the behavior, depending on the magnitude of the difference.

[0009] An information processing method or program according to one aspect of the present technology is an information processing method or program for an information processing system that determines behavior based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating behavior, and includes a step of calculating the magnitude of the difference between newly input environmental information or the evaluation function and existing environmental information or the evaluation function, and updating the learning model based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation in accordance with the behavior, depending on the magnitude of the difference.

[0010] In one aspect of the present technology, in an information processing system that determines behavior based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating behavior, the magnitude of the difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function is calculated, and the learning model is updated based on the magnitude of the difference and the amount of reward obtained by the evaluation in accordance with the behavior. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a learning model. [Figure 2] FIG. 1 is a diagram illustrating the present technology. [Figure 3] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 4] 10 is a flowchart illustrating a behavior decision process. [Figure 5] FIG. 10 is a diagram illustrating an example of behavior depending on the magnitude of an error. [Figure 6] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0013] First Embodiment About the learning model This technology updates the learning model based on the magnitude of the difference between newly input environmental information or reward information and existing environmental information or reward information, thereby enabling the decision to execute learning to be made without relying on external instruction input, i.e., automatically switching the learning target.

[0014] First, a model (hereinafter referred to as a learning model) that is the target of reinforcement learning performed in this technology will be described.

[0015] In this technology, a learning model such as LSTM (Long Short Term Memory), whose inputs and outputs are environmental information, actions, rewards, and states, is generated by reinforcement learning, as shown in Figure 1.

[0016] In this example, environmental information, which is information about the surrounding environment at a given time t, behavior (information indicating behavior) at time t-1 immediately before time t, and reward (information indicating the amount of reward) for the behavior at time t-1 are input into the learning model.

[0017] The learning model performs a predetermined calculation based on the input environmental information, actions, and rewards, determines the action to be taken at time t, and outputs the determined action at time t (information indicating the action) and the state at time t that changes due to that action (information indicating the state).

[0018] The state that is the output of the learning model is the state of the agent (information processing system) that performs the action, or the changes in the surrounding environment that occur as a result of that action.

[0019] In this technology, the amount of reward given to a behavior changes depending on the behavior, which is the output of the learning model, that is, depending on the state of the environment and other factors that correspond to the behavior.

[0020] The learning model is associated with reward information consisting of an evaluation function or the like for evaluating the behavior determined by the learning model.

[0021] This reward information is used to evaluate the behavior determined by the learning model and determine the amount of reward that indicates the evaluation result, that is, to determine how much reward to give for the behavior.

[0022] The reward information also indicates the purpose (goal) of the behavior determined by the learning model, that is, the task that is the target of reinforcement learning.

[0023] The reward amount for the behavior determined by the learning model is determined by an evaluation function included in the reward information. For example, the evaluation function can be a function that takes the behavior as input and the reward amount as output. Alternatively, for example, a reward amount table that associates the behavior with the reward amount to be given for that behavior may be included in the reward information, and the reward amount for the behavior may be determined based on the reward amount table.

[0024] In the learning model, the next (future) action is determined using the past (immediately preceding) action and the amount of reward determined for that action based on reward information, so it can be said that reward information is also used to determine the action.

[0025] About Reinforcement Learning Next, with reference to FIG. 2, reinforcement learning performed in an information processing system to which the present technology is applied will be described.

[0026] An information processing system to which the present technology is applied performs, for example, reinforcement learning of the above-mentioned learning model, and functions as an agent that determines actions based on the learning model.

[0027] For example, in an information processing system, existing information is stored in past memories X as shown by arrow Q11. t-1 is held as.

[0028] The existing information includes, for example, a learning model, environmental information for each past situation for that learning model, reward information, selected behavior information indicating behaviors decided (selected) in the past, and the amount of reward given for the behavior indicated by the selected behavior information, i.e., the evaluation result of the behavior.

[0029] The environmental information included in the existing information is information about the environment around the information processing system, etc. Specifically, for example, the environmental information is map information showing a map of a specific city, information showing sensing results such as surrounding images obtained by sensing in a specific city, and information showing the relationship between the positions of surrounding objects, etc.

[0030] In the following, we will use the reward information included in the existing information, i.e., the existing reward information as R t-1 In the following, the action determined (selected) by the learning model will also be referred to as the selected action.

[0031] In the information processing system, new input information X t When the input is provided (input), the existing information is accessed, and as shown by arrow Q12, new input information X t and existing information, that is, past memory X t-1 A comparison is made with the

[0032] Newly entered information Xt The latest (new) reward information R t and environmental information.

[0033] Newly entered information X t The reward information and environmental information contained in the may be the same as the existing reward information and environmental information as the existing information, or may be updated reward information and environmental information that is different from the existing information.

[0034] Newly entered information X t When this is entered, the new input information X is selected from the existing information. t The past reward information and environmental information that are closest to, i.e., most similar to, the reward information and environmental information included in the previous reward information and environmental information are read out.

[0035] Then, the past reward information (evaluation function) and environmental information that have been read out are compared with the new input information X t For example, during matching, differences between past (existing) reward information and environmental information and new reward information and environmental information are detected.

[0036] In this case, the new input information X t And, Past Memories X t-1 An example in which the above is performed will be described.

[0037] However, this is not the only option. New input information X t and Past Memories X t-1 (Existing information) and estimate the current situation, and the estimated result, i.e., the expected value C t and new input information X t In this case, for example, the expected value C t Environmental information, reward information, behavior, etc. are estimated as the

[0038] Newly entered information X t And, Past Memories X t-1Then, as shown by arrow Q13, the difference between the environmental information and the reward information (evaluation function), more specifically the magnitude of the difference, is detected based on the result of the comparison, and the prediction error e t is generated.

[0039] In detecting the difference, at least one of the context-based error caused by environmental information (hereinafter also referred to as the context-based prediction error) and the cognition-based error caused by the evaluation function (reward information) (hereinafter also referred to as the cognition-based prediction error) is detected.

[0040] Context-based prediction error is an error caused by contextual deviations that depend on the environment, such as an unfamiliar location or context, or a sudden change in a known context. It is used to detect new environmental information, i.e., new environmental variables or changes in known environmental variables, and reflect (incorporate) it into learning models, etc.

[0041] Specifically, for example, the context-based prediction error is information indicating the magnitude of the difference between new environmental information and existing environmental information, and the new input information X t New environmental information and past memory X t-1 It is calculated based on the difference (difference) between the existing environmental information and the actual environmental information.

[0042] Cognitive-based prediction error is an error caused by cognitive conflict, such as a gap (information discrepancy) from what is known or predictable. This cognitive-based prediction error is used to suppress the use of known evaluation functions in situations where an error (conflict) occurs that cannot be resolved using existing methods (learning models), and to detect new evaluation functions and reflect (incorporate) them into the learning model. In other words, when a cognitive-based prediction error is detected, reinforcement learning (updating) is performed to obtain a learning model in which a new evaluation function is used and the use of the existing evaluation function is suppressed.

[0043] Specifically, for example, the cognition-based prediction error is information indicating the magnitude of the difference between the new evaluation function and the existing evaluation function, and the new input information Xt The new evaluation function as a function of past memory X t-1 It is calculated based on the difference (difference) between the existing evaluation function as

[0044] The information processing system calculates a final prediction error e based on at least one of the context-based prediction error and the cognition-based prediction error. t is required.

[0045] Prediction error e t is new input information X t It shows the magnitude of the difference between the newly input environmental information or reward information (evaluation function) and the existing environmental information or reward information (evaluation function) as the existing information. In other words, the prediction error e t is the new input information X based on the existing information. t It can also be said that this is the magnitude of the uncertainty when deciding on actions to take.

[0046] Specifically, for example, if only one of the context-based prediction error and the cognition-based prediction error is non-zero, that is, if only one of the context-based prediction error and the cognition-based prediction error is detected, the detected value is the prediction error e t It is said that.

[0047] In addition, the total prediction error obtained by performing some calculation based on the context-based prediction error and the cognition-based prediction error is called the prediction error e t It may also be possible to use the following.

[0048] Furthermore, for example, if both a context-based prediction error and a cognition-based prediction error are detected, the value of a predetermined one (the one with a higher priority) of the prediction errors is set as the prediction error e t It may be configured as follows.

[0049] In addition, context-based prediction error, cognition-based prediction error, and prediction error e tcan be a scalar value, a vector value, or an error distribution. For simplicity, we will use the context-based prediction error, the cognition-based prediction error, and the prediction error e t is assumed to be a scalar value.

[0050] Prediction error e t is obtained, the information processing system calculates the prediction error e t is compared with a predetermined threshold ±SD to obtain the prediction error e t In this example, the magnitude of the prediction error e t The magnitude of the error (the magnitude of the error k) is classified as either "small," "medium," or "large."

[0051] That is, the prediction error e t If is less than -SD, the error magnitude k is t The magnitude of the error is "small" when the prediction error e t The magnitude of this indicates that the new problem can be solved (determined action) without any problems by applying an existing learning model.

[0052] Also, the prediction error e t If is greater than or equal to -SD and less than or equal to SD, the magnitude of the error k is the prediction error e t The error magnitude is "medium" indicating that the prediction error e t However, when solving a new problem, the output obtained by applying an existing learning model is large enough to cause problems, but it is large enough that reinforcement learning of the learning model is possible. t If is larger than the SD, the magnitude of the error k is t The magnitude of the error is "large." When solving a new problem, even if learning is performed based on new input (new input information), learning does not take place. In other words, the prediction error e is so large that it is difficult for learning to converge. t This indicates that is expected to be large.

[0053] In the information processing system, the new input information X is calculated according to the magnitude k of the error obtained as a result of such discrimination. t It is determined whether to update the existing learning model, that is, whether to perform reinforcement learning on the learning model, using the above.

[0054] That is, the information processing system (agent) autonomously decides to execute reinforcement learning of the learning model based on the magnitude of the error k, without relying on external instruction input. In other words, the information processing system (agent) automatically switches the learning target.

[0055] Specifically, when the error magnitude k is "small", reinforcement learning of the learning model is not performed, and the existing information is used as is to execute the action, and then the next new input information X t This requires input, i.e., the search for new learning (new tasks).

[0056] The magnitude of the error k is "small" when, for example, the newly input information X t and Past Memories X t-1 This is the case when the difference between the existing reward information and the existing environmental information is small, that is, when the new reward information and the new environmental information are exactly the same or almost the same as the existing reward information and the new environmental information.

[0057] Therefore, in such a case, for example, new input information X t As the action determined for the new input information X, the selection action indicated by the selection action information stored as existing information can be selected as it is. t Based on the environmental information and the reward information, new input information X t An action to be taken may be determined.

[0058] Also, when the magnitude of the error k is "large," the information processing system does not perform reinforcement learning of the learning model, but performs avoidance behavior, as shown by arrow Q15. Then, after that, the next new input information X t This requires input, i.e., the search for new learning (new tasks).

[0059] For example, if the error magnitude k is "large", the prediction error e t , that is, the uncertainty is too large, and even if the learning model is reinforced learning, it may not be possible to select an appropriate action. In other words, in an information processing system, new input information X t It can be difficult to solve the problem presented by

[0060] Therefore, in the information processing system, reinforcement learning of the learning model is not performed, that is, the execution of reinforcement learning is suppressed, and as a process corresponding to the avoidance behavior, for example, new input information X t A process is performed to request another system to select an action for the target.

[0061] In this case, after the avoidance action, the next new input information X t This requires input, i.e., the exploration of new learning (new tasks), and shifts to reinforcement learning, a new learning model.

[0062] Others, such as existing learning models and new input information X t Based on the environmental information and the reward information, new input information X t The process of determining an action to be taken in response to the avoidance action and presenting the determined action to the user may be performed as a process corresponding to the avoidance action. In such a case, the user may select whether or not to actually perform the determined action.

[0063] Furthermore, for example, if the magnitude of the error k is "medium," the information processing system induces an approach (preference) to the execution of reinforcement learning of the learning model, and as shown by arrow Q16, a match is made with the reward (reward information) to determine the pleasure Rd (degree of pleasure).

[0064] In addition, the prediction error e tThe calculation method and threshold SD may be set so that cognition-based prediction errors are more difficult than context-based prediction errors, i.e., so that the approach to reinforcement learning is more likely to be induced. Such settings may also be achieved by adjusting the distribution of errors as context-based prediction errors and cognition-based prediction errors.

[0065] In the portion indicated by arrow Q16, the remuneration (remuneration information) is verified.

[0066] That is, new input information X t Remuneration information as R t and the existing reward information R t-1 and are read out, and their reward information R t and reward information R t-1 The comfort level Rd is calculated based on this.

[0067] The degree of pleasure Rd is the reward information R t and reward information R t-1 In more detail, the pleasure level Rd is calculated by the error (difference) of the amount of reward obtained for the action. t Newly entered environmental information or compensation information R t The amount of reward predicted based on (evaluation function) and the existing reward information R t-1 This shows the difference (error) between the amount of reward predicted based on existing information such as the above.

[0068] For example, the larger the error in the amount of reward, the larger the pleasure Rd, which is said to be positive for the implementation of reinforcement learning.

[0069] In other words, when the pleasure level Rd is large, the new input information X t It can be said that a positive reward is obtained for solving the corresponding task (reinforcement learning of the learning model), and when the pleasure Rd is small, a negative reward is obtained for solving the task.

[0070] Such a pleasure level Rd mimics human psychology (curiosity), in which pleasure increases when a large reward is obtained, leading to more proactive (positive) behavior.

[0071] For example, the pleasure level Rd is the reward information R t and reward information R t-1 Regarding the new input information X t The reward amount may be calculated by estimating the amount of reward that can be obtained for approximately the same conditions or behavior as the target, and finding the difference between the estimated amounts of reward, or by using other methods.

[0072] For example, the pleasure level Rd can be calculated by using the evaluation results (reward amount) for past selection behaviors included in the existing information as they are, or by using the evaluation results to calculate the pleasure level Rd using the newly input information X. t The behavior and reward amount for the target object may be estimated, and the estimated results may be used to calculate the pleasure level Rd.

[0073] In addition, the calculation of the pleasure Rd is based not only on the amount of reward based on the reward information, but also on the positive reward (positive reward) and the new input information X t Negative rewards (i.e., risk magnitudes) predicted based on existing information may also be taken into account. In this case, negative rewards may be calculated from reward information, or negative rewards may be predicted based on other information.

[0074] Once the pleasure level Rd is calculated, the information processing system compares the pleasure level Rd with a predetermined threshold value th, as shown by arrow Q17, to determine the magnitude of the pleasure level Rd. In this example, the magnitude of the pleasure level Rd (pleasure level V) is classified as either "low" or "high."

[0075] That is, when the pleasure level Rd is less than the threshold value th, the magnitude V of the pleasure level is set to "low," which indicates that the pleasure level Rd is low (small), that is, the reward obtained is negative.

[0076] On the other hand, when the pleasure level Rd is equal to or greater than the threshold value th, the magnitude V of the pleasure level is set to "high," which indicates that the pleasure level Rd is high (large), that is, the reward obtained is positive.

[0077] When the magnitude of pleasure V is "low," the reward obtained for solving the problem is negative, so just as when the magnitude of error k is "large," reinforcement learning of the learning model is not performed and avoidance behavior shown by arrow Q15 is taken.

[0078] On the other hand, when the magnitude of pleasure V is "high," the reward obtained for solving the problem is positive, and therefore, the behavior proximate to solving the problem is induced. That is, as shown by arrow Q18, the newly input information X t Based on this, reinforcement learning of the learning model included in the existing information is performed. At this time, new environmental information and the like are appropriately acquired as data for reinforcement learning.

[0079] In reinforcement learning of learning models, environmental information, the current action, and the amount of reward for the current action are input, and the gradients (coefficients) of the network nodes that make up the learning model, which outputs the next action and the environmental changes (state) that result from that action, are updated.

[0080] At this time, the weighting of the reinforcement learning may be changed according to the magnitude of the pleasure V, that is, the magnitude of the curiosity.

[0081] It is known that humans promote memory for objects that interest them, and that memory is consolidated. Since reinforcement learning occurs in a state of high curiosity, changing the learning weight according to the magnitude of pleasure V will result in behavior that mimics this relationship between curiosity and memory, and a learning model that makes behavioral choices that are closer to humans can be obtained.

[0082] In an information processing system, memory is updated when reinforcement learning of a learning model is performed.

[0083] In other words, the learning model obtained by reinforcement learning, that is, the updated learning model, and the new input information X t The existing information is updated so that the environmental information and reward information are included in the existing information as new memories. At this time, the learning model before the update included in the existing information is replaced with the updated learning model.

[0084] During reinforcement learning, the current situation such as the selected behavior and environmental changes (state) is checked sequentially, and the prediction error e t It is also possible to perform self-monitoring, in which learning is performed while updating the above.

[0085] Furthermore, the information processing system may also be configured to maintain a counter that counts how many times an action determined based on a learning model has been performed.

[0086] In this case, the smaller the counter value, the less bored the behavior is and the more curious the information processing system (agent) is about reinforcement learning (problem solving). Conversely, when the counter value is large, the behavior has been repeated too many times and bored has developed, meaning that the agent has adapted to the stimulus.

[0087] Therefore, if the counter value is less than a predetermined threshold, reinforcement learning of the learning model may continue, and if the counter value is greater than or equal to the threshold, reinforcement learning may be terminated and avoidance action shown by arrow Q15 may be taken.

[0088] Even without providing such a counter, when reinforcement learning of the learning model is repeated, new input information X t Since the magnitude of the error k and the magnitude of the pleasantness V change each time an input is made, a process that mimics adaptation (boredom) to a stimulus is realized. Specifically, for example, if the magnitude of the error k becomes "small" after repeated reinforcement learning, reinforcement learning is not performed, resulting in the same behavior as when boredom has occurred.

[0089] As described above, choosing avoidance actions or deciding to implement reinforcement learning depending on the magnitude of the error k, i.e., the magnitude of the uncertainty, and the magnitude of the pleasure V, can be said to be close to actual human behavior.

[0090] It has been found that the human brain promotes learning by correcting the prediction error between the actual sensory feedback corresponding to the action performed in response to a motor command and the sensory feedback predicted from the motor command, and that it particularly prefers a medium prediction error. This corresponds to a situation in which the information processing system approaches reinforcement learning when the error size k is "medium."

[0091] Furthermore, it has been found that the pleasantness of reward prediction error in the human brain correlates with the avoidance network (ventral prefrontal cortex, posterior cingulate cortex), and that high pleasantness promotes approach. This corresponds to the determination of reinforcement learning when the magnitude of pleasantness V is "high."

[0092] Furthermore, it has been found that prediction errors in sensory feedback can be classified into prediction errors due to context misalignment and prediction errors due to cognitive conflict (information misalignment), and that prediction errors induce two responses: curiosity and anxiety. In this case, memory is promoted for objects that evoke curiosity, while behavior is suppressed for objects that evoke anxiety.

[0093] This is the difference between the context-based prediction error and the cognition-based prediction error. t This corresponds to determining whether or not to perform reinforcement learning depending on the magnitude of the error k and the magnitude of the pleasure V.

[0094] Therefore, the behavior of the information processing system described with reference to FIG. 2 can be said to be close to human behavior, and according to the present technology, it is possible to realize an agent (information processing system) that behaves more like a human.

[0095] In other words, this technology makes it possible to realize an information processing system that is curious about reinforcement learning and can autonomously decide whether or not to perform reinforcement learning, i.e., to start or end reinforcement learning, and autonomously decide to transition (switch) the target of reinforcement learning.

[0096] We now further explain context-based prediction error and perception-based prediction error.

[0097] The context-based prediction error indicates the discrepancy between existing environmental information (past experience) and new environmental information. In other words, the context-based prediction error is an error caused by the discrepancy in environmental information.

[0098] Specifically, a context deviation occurs when a map is displayed in an unfamiliar location or when objects on the map change, and the magnitude of such a context deviation is the context-based prediction error.

[0099] When calculating the context-based prediction error, new contexts or sudden changes in context are detected by comparing new environmental information with existing environmental information, and the context-based prediction error is calculated based on the detection results.

[0100] Furthermore, conventional curiosity models, for example in route planning, emphasize exploring new learning targets and do not treat areas that have already been explored as new exploration targets (learning targets). As a result, the behavior of such curiosity models may be at odds with human curiosity-based behavior.

[0101] In contrast, in the information processing system of the present technology, which performs reinforcement learning according to the context-based prediction error, as described above, the robot may stop exploring due to boredom (terminating reinforcement learning), or its behavior may change depending on the magnitude k of the error based on the context-based prediction error.

[0102] The change in behavior here refers to the decision on whether or not to execute reinforcement learning, in other words, the start or end of reinforcement learning, the selection of avoidance behavior, etc.

[0103] For example, if the error magnitude k is "small," reinforcement learning is not performed; that is, exploration (reinforcement learning) is stopped (terminated) due to adaptation to the exploratory behavior itself. Also, if the error magnitude k is "medium," exploration by the curiosity module, i.e., reinforcement learning of the learning model, is performed, and if the error magnitude k is "large," avoidance behavior is performed due to behavioral inhibition.

[0104] The information processing system of this technology can be said to be a model that behaves more like a human being compared to general curiosity models.

[0105] In an information processing system, by using a context-based prediction error to determine whether to perform reinforcement learning, reinforcement learning that incorporates new changes in external information, i.e., changes in environmental information, can be realized. In other words, when a context-based prediction error is detected, reinforcement learning (updating) is performed to obtain a learning model that incorporates changes in environmental information.

[0106] The cognitive-based prediction error indicates the discrepancy between existing reward information (past experience) and new reward information, particularly the discrepancy between the existing evaluation function and the new evaluation function. In other words, the cognitive-based prediction error is an error caused by the discrepancy between the evaluation functions.

[0107] Specifically, the cognitive-based prediction error indicates how new the reward information is compared to the evaluation function used to evaluate past choice behavior, or the purpose or task of the behavior indicated by the reward information.

[0108] When calculating the cognition-based prediction error, the cognition-based prediction error is obtained based on a comparison of the gap between the known evaluation function and the new evaluation function, and past known information (existing information) is suppressed and the evaluation function is updated.

[0109] In the information processing system of the present technology that performs reinforcement learning in response to such cognition-based prediction errors, new reward information is recorded by updating the memory as described above. Therefore, the meaning of the existing behavioral goal (existing reward information) is lost due to the goal setting corresponding to the recorded new reward information, i.e., the behavioral goal indicated by the new reward information, and as a result, the use of the existing evaluation function (reward information) is suppressed.

[0110] Furthermore, by utilizing cognition-based prediction error in the information processing system of this technology, exploration can be stopped due to boredom (reinforcement learning can end), and behavior can change depending on the magnitude k of the error based on cognition-based prediction error.

[0111] For example, when the error magnitude k is "small," the cognitive-based prediction error is zero or small, so reinforcement learning is not performed and new learning (new tasks) are explored. In other words, the learning target is switched.

[0112] Furthermore, when the magnitude of the error k is "medium," exploration by the curiosity module, i.e., reinforcement learning of the learning model, is performed, and when the magnitude of the error k is "large," avoidance behavior is performed by behavioral inhibition.

[0113] In this way, information processing systems that utilize cognition-based prediction errors can spontaneously perform reinforcement learning and switch learning targets, thereby increasing the number of existing evaluation functions (reward information) and broadening the purpose of behavior.

[0114] <Example of information processing system configuration> Next, a configuration example of the information processing system of the present technology described above will be described.

[0115] The information processing system 11 shown in FIG. 3 includes an information processing device that functions as an agent that determines an action based on a reinforcement learning model and input environmental information and reward information, and executes the determined action.

[0116] The information processing system 11 may be configured to include one information processing device, or may be configured to include multiple information processing devices.

[0117] The information processing system 11 includes a behavior unit 21, a recording unit 22, a matching unit 23, a prediction error detection unit 24, an error determination unit 25, a reward matching unit 26, a pleasure level determination unit 27, and a learning unit .

[0118] The behavior unit 21 acquires new input information supplied from the outside, supplies the acquired new input information to the matching unit 23 and the recording unit 22, determines an action based on the learning model read out from the recording unit 22 and the acquired new input information, and actually executes the action.

[0119] The recording unit 22 records existing information and updates the existing information by recording environmental information and reward information as new input information supplied from the behavior unit 21 and the learning unit 28, as well as a learning model that has undergone reinforcement learning. The recording unit 22 also supplies the recorded existing information to the behavior unit 21, the matching unit 23, the reward matching unit 26, and the learning unit 28 as appropriate.

[0120] The existing information recorded in the recording unit 22 includes, as described above, a learning model, environmental information for each past situation for the learning model, reward information, past selection behavior information, and the amount of reward given to the behavior indicated by the selection behavior information (behavior evaluation result). In other words, the learning model included in the existing information is obtained by reinforcement learning based on the existing environmental information and reward information included in the existing information. Furthermore, the environmental information may be any information related to the environment surrounding the information processing system 11.

[0121] The matching unit 23 matches the new input information supplied from the behavior unit 21 with the existing information supplied from the recording unit 22, more specifically, with existing environmental information and reward information, i.e., matches the new input information with past memories, and supplies the matching result to the prediction error detection unit 24.

[0122] The prediction error detector 24 calculates the prediction error. The prediction error calculated by the prediction error detector 24 is the above-mentioned prediction error e t is.

[0123] The prediction error detector 24 includes a context-based prediction error detector 31 and a perception-based prediction error detector 32 .

[0124] The context-based prediction error detector 31 calculates a context-based prediction error based on the collation result from the collation unit 23, that is, the new environmental information as the newly input information and the environmental information included in the existing information.

[0125] The cognition-based prediction error detection unit 32 calculates a cognition-based prediction error based on the collation result from the collation unit 23, that is, the new reward information as the newly input information and the reward information included in the existing information.

[0126] The prediction error detection unit 24 calculates a final prediction error based on the context-based prediction error calculated by the context-based prediction error detection unit 31 and the perception-based prediction error calculated by the perception-based prediction error detection unit 32, and supplies the final prediction error to the error determination unit 25.

[0127] The error determination unit 25 determines the magnitude of the prediction error (error magnitude k) based on the prediction error supplied from the prediction error detection unit 24. That is, the error determination unit 25 determines whether the magnitude of the prediction error (error magnitude k) is "large," "medium," or "small."

[0128] In addition, depending on the result of determining the magnitude of the prediction error (error magnitude k), the error determination unit 25 instructs the reward matching unit 26 to match the reward (reward information) or instructs the behavior unit 21 to perform behavior other than reinforcement learning.

[0129] In response to instructions from the error determination unit 25, the reward matching unit 26 acquires reward information, etc. from the behavior unit 21 and the recording unit 22 and compares the reward (reward information) to calculate the pleasure Rd and supplies it to the pleasure degree determination unit 27.

[0130] The pleasantness determination unit 27 determines the magnitude of the pleasantness Rd (pleasantness magnitude V) supplied from the reward matching unit 26, and instructs the behavior unit 21 to take avoidance action or the learning unit 28 to perform reinforcement learning depending on the determination result.

[0131] The learning unit 28 acquires new input information and existing information from the behavior unit 21 and the recording unit 22 in response to an instruction from the pleasantness level determination unit 27, and performs reinforcement learning of the learning model.

[0132] In other words, the learning unit 28 updates the existing learning model based on the environmental information and reward information (evaluation function) newly input as new input information, and the amount of reward obtained by evaluation using the reward information according to the behavior, depending on the magnitude of error k and the magnitude of pleasantness V.

[0133] The learning unit 28 includes a curiosity module 33 and a memory module 34 .

[0134] The curiosity module 33 updates the learning model included in the existing information by performing reinforcement learning based on the learning weights during reinforcement learning, i.e., the parameters for reinforcement learning, determined by the memory module 34. The memory module 34 determines the learning weights (parameters) during reinforcement learning based on the magnitude V of the pleasure.

[0135] <Explanation of Action Decision Processing> Next, a description will be given of the operation of the information processing system 11. That is, the behavior decision processing by the information processing system 11 will be described below with reference to the flowchart of FIG.

[0136] In step S11, the behavior unit 21 acquires new input information containing at least either new environmental information or reward information from the outside and supplies it to the matching unit 23 and the recording unit 22, and also instructs the recording unit 22 to output existing information corresponding to the new input information.

[0137] Then, in response to instructions from the behavior unit 21, the recording unit 22 selects from the existing recorded information the environmental information and reward information that are most similar (have the highest degree of similarity) to the environmental information and reward information as new input information supplied from the behavior unit 21 and supplies them to the matching unit 23 as past memory.

[0138] In step S12, the matching unit 23 matches the new input information supplied from the behavior unit 21 with the past memory supplied from the recording unit 22, and supplies the matching result to the prediction error detection unit 24.

[0139] In step S12, for example, a comparison is made to determine whether there is a difference between the environmental information as newly input information and the existing environmental information as past memories, and a comparison is made to determine whether there is a difference between the reward information as newly input information and the existing reward information as past memories.

[0140] In step S13, the context-based prediction error detection unit 31 calculates a context-based prediction error based on the collation result from the collation unit 23, that is, new environmental information as newly input information and environmental information as past memory.

[0141] In step S14, the cognition-based prediction error detection unit 32 calculates a cognition-based prediction error based on the collation result from the collation unit 23, that is, the new reward information as the newly input information and the reward information as the past memory.

[0142] Furthermore, the prediction error detector 24 calculates a final prediction error e based on the context-based prediction error calculated by the context-based prediction error detector 31 and the perception-based prediction error calculated by the perception-based prediction error detector 32. t and supplies it to the error determination unit 25.

[0143] Furthermore, the error determination unit 25 receives the prediction error e t The magnitude k of the error is classified into "small," "medium," or "large" by comparing it with a predetermined threshold ±SD.

[0144] Here, as mentioned above, the prediction error e t If is less than -SD, the error magnitude k is considered "small" and the prediction error e t If is greater than or equal to -SD and less than or equal to SD, the error magnitude k is considered "medium" and the prediction error e t If is greater than SD, the error magnitude k is considered "large."

[0145] In step S15, the error determination unit 25 determines whether the magnitude k of the error is "small" or not.

[0146] If it is determined in step S15 that the magnitude of the error k is "small," the error determination unit 25 instructs the behavior unit 21 to select an action using an existing learning model, etc., and then the process proceeds to step S16. In this case, reinforcement learning (updating) of the learning model is not performed.

[0147] In step S16, the behavior unit 21, in response to instructions from the error determination unit 25, determines (selects) the next action to be taken based on the new input information acquired in step S11 and the existing learning model and reward information recorded in the recording unit 22.

[0148] For example, the behavior unit 21 inputs environmental information as new input information and a reward amount calculated from reward information (evaluation function) included in existing information into an existing learning model, performs calculations, and determines the behavior obtained as an output as the behavior to be performed. Then, the behavior unit 21 executes the determined behavior, and the behavior determination process ends. Note that, as described above, the behavior indicated by the selected behavior information included in the existing information may be determined as the behavior to be performed.

[0149] If it is determined in step S15 that the magnitude k of the error is not "small", the error determination unit 25 determines in step S17 whether the magnitude k of the error is "medium".

[0150] If it is determined in step S17 that the magnitude k of the error is not "medium," that is, that the magnitude k of the error is "large," the error determination unit 25 instructs the behavior unit 21 to perform an avoidance behavior, and then the process proceeds to step S18. In this case, reinforcement learning (updating) of the learning model is not performed.

[0151] In step S18, the behavior unit 21 performs avoidance behavior in accordance with the instruction from the error determination unit 25, and the behavior determination process ends.

[0152] For example, the behavior unit 21 supplies the new input information acquired in step S11 to an external system, and performs a process of requesting the determination (selection) of an appropriate behavior corresponding to the new input information as a process corresponding to the avoidance behavior. Then, when the behavior unit 21 receives information indicating the determined behavior from the external system, it executes the behavior indicated by the information.

[0153] Furthermore, for example, the behavior unit 21 may present to the user on a display unit (not shown) alternative solutions for solving the problem corresponding to the newly input information, such as by making an inquiry to an external system, and may perform processing corresponding to the avoidance behavior by executing an action in accordance with the instruction input by the user in response to the presented solution.

[0154] Furthermore, the behavior unit 21 may present to the user the behavior determined by the same processing as in step S16, and perform processing to execute the behavior in accordance with the instruction input by the user in response to the presentation, as processing corresponding to the avoidance behavior.

[0155] Alternatively, the avoidance behavior may be controlled so that the behavior unit 21 does not determine (select) and execute a behavior based on an existing learning model.

[0156] When avoidance behavior such as the above is performed, reinforcement learning of the learning model is not performed, and after the avoidance behavior is performed, the system moves on to new learning (new task), that is, the search for reinforcement learning of a new learning model.

[0157] Furthermore, if it is determined in step S17 that the magnitude k of the error is "medium", the error determination unit 25 instructs the remuneration matching unit 26 to perform matching of the remuneration (remuneration information), and then the process proceeds to step S19.

[0158] In step S19, the reward collating unit 26 collates the reward (reward information) in accordance with an instruction from the error determining unit 25 to calculate the pleasure level Rd and supplies it to the pleasure level determining unit 27.

[0159] That is, the reward matching unit 26 acquires the new input information acquired in step S11 from the behavior unit 21, and reads out from the recording unit 22 the existing environmental information, reward information, selection behavior information, and evaluation results (reward amounts) for past selection behaviors contained in the existing information.

[0160] The reward matching unit 26 then calculates the pleasure Rd based on the environmental information and reward information as newly input information, the existing environmental information and reward information included in the existing information, the selection behavior information, and the evaluation results of past selection behaviors. At this time, the reward matching unit 26 also calculates the pleasure Rd using the negative reward (risk) calculated from the reward information, etc.

[0161] In addition, the pleasure level determination unit 27 compares the pleasure level Rd supplied from the reward matching unit 26 with a predetermined threshold value th to classify the magnitude of the pleasure level Rd (pleasure level V) as either "high" or "low."

[0162] Here, as described above, if the pleasantness Rd is less than the threshold value th, the magnitude V of the pleasantness is set to "low," and if the pleasantness Rd is equal to or greater than the threshold value th, the magnitude V of the pleasantness is set to "high."

[0163] In step S20, the pleasure level determination unit 27 determines whether the pleasure level V is "high" or not.

[0164] If it is determined in step S20 that the magnitude V of the pleasure is not "high", that is, is "low", then an avoidance action is taken in step S18, and the action decision process ends.

[0165] In this case, reinforcement learning (updating) of the learning model is not performed, and the pleasantness determination unit 27 instructs the behavior unit 21 to take avoidance action, and the behavior unit 21 takes avoidance action in accordance with the instruction.

[0166] On the other hand, if it is determined in step S20 that the pleasure level V is "high," the pleasure level determination unit 27 supplies the pleasure level V to the learning unit 28 and instructs the learning unit 28 to execute reinforcement learning, and then the process proceeds to step S21. In this case, the pleasure level determination unit 27 has decided (selected) to execute reinforcement learning.

[0167] In step S21, the learning unit 28 performs reinforcement learning of the learning model in response to an instruction from the pleasantness level determination unit 27.

[0168] That is, the learning unit 28 acquires the new input information acquired in step S11 from the behavior unit 21, and reads out from the recording unit 22 the existing learning model, environmental information, reward information, selection behavior information, and evaluation results (reward amount) for past selection behavior contained in the existing information.

[0169] Furthermore, the memory module 34 of the learning unit 28 determines weighting (parameters) for learning during reinforcement learning based on the magnitude V of the pleasure level supplied from the pleasure level determination unit 27.

[0170] Furthermore, the curiosity module 33 of the learning unit 28 performs reinforcement learning of the learning model using the learning weights at the time of reinforcement learning determined by the memory module 34, based on the environmental information and reward information as new input information, and the existing learning model, selection behavior information, etc. included in the existing information. That is, the curiosity module 33 updates the existing learning model by performing calculation processing based on the learning weights (parameters).

[0171] In the reinforcement learning of the learning model, new data such as environmental information required for reinforcement learning is collected as needed. This data is acquired by the behavior unit 21 from a sensor (not shown) and supplied to the learning unit 28, and the curiosity module 33 of the learning unit 28 also uses the data supplied from the behavior unit 21 to perform reinforcement learning.

[0172] Through reinforcement learning, an updated learning model is obtained that takes as input environmental information, actions, and the reward (reward amount) for an action determined from reward information as new input information, and outputs the next action and state.

[0173] In step S22, the learning unit 28 updates information. That is, the learning unit 28 supplies the updated learning model obtained by the reinforcement learning in step S21, and the environmental information and reward information as new input information to the recording unit 22, and causes them to be recorded.

[0174] Once the learning model, environmental information, and reward information have been recorded and the existing information has been updated, the behavior decision-making process is completed.

[0175] In this way, when new input information is supplied, the information processing system 11 calculates the magnitude of error k and the magnitude of pleasantness V, and depending on these magnitudes, it spontaneously selects an action based on existing information, performs reinforcement learning, or takes avoidance action.

[0176] In this way, the information processing system 11 can autonomously decide to execute reinforcement learning without relying on external instruction input. In other words, it can automatically switch the learning target, and realize an agent that behaves more like a human.

[0177] <Specific examples> Here, a specific example of reinforcement learning in the learning model described above will be described.

[0178] Here, as a specific example, we will explain a learning model that performs route search (path planning) and outputs the most appropriate route from a specified starting point, such as the current location, to a destination that meets the conditions (purpose of the action) indicated by newly input information (reward information).

[0179] In particular, with regard to such a learning model, we will explain, with reference to Figure 5, the cases where only context-based prediction errors indicating contextual misalignment are detected and the cases where only cognitive-based prediction errors indicating cognitive misalignment (cognitive conflict) are detected.

[0180] First, the case where only a context-based prediction error is detected will be described.

[0181] In this example, environmental information includes, for example, location information of a destination such as a hospital, map information (map data) of the area around the destination, basic information related to the map information such as direction and one-way streets, the driving time normally required for each route on the map, and information related to the vehicles that will be driving as part of the activity.

[0182] Then, for example, it is assumed that the map information (map data) has been updated as a result of comparing (collating) the environmental information as the newly input information with the environmental information included in the existing information.

[0183] In this case, the context-based prediction error can be calculated based on factors such as the increase (change) in detour distance and driving time to the destination due to the map information update, the number of roads that require route changes, and differences in cities, regions, countries, and traffic rules between the new and existing map information.

[0184] When only the context-based prediction error is detected, the prediction error detection unit 24 calculates, for example, the context-based prediction error, i.e., the difference between the environmental information as the newly input information and the environmental information included in the existing information, as the prediction error e t and the prediction error e t Let the magnitude of the error be k.

[0185] If the error determination unit 25 determines that the magnitude k of the error is "small," reinforcement learning is not performed in the information processing system 11, and an existing learning model is used to select an action. That is, processing is performed using the existing learning model, and the results are output.

[0186] For example, a case in which the magnitude of the error k is "small" may occur when the new map information and the existing map information are both map information for the same city, but the maps shown by the map information, i.e., the roads, buildings, etc., are slightly different.

[0187] In such cases, the difference in environmental information is sufficiently small that the output of the learning model is likely not to change significantly.

[0188] Therefore, the behavior unit 21 searches for a route to the destination using the learning model and reward information included in the existing information and the environmental information as newly input information, and presents the route that is the search result to the user. Then, when the user instructs the vehicle to travel to the destination, the behavior unit 21 performs control in accordance with the instruction so that the vehicle actually travels along the route obtained as a result of the route search.

[0189] Furthermore, for example, when the error determination unit 25 determines that the magnitude k of the error is "medium," reinforcement learning of the learning model is performed in the information processing system 11. That is, the learning model is updated.

[0190] For example, the following cases are possible when the magnitude of the error k is "medium."

[0191] That is, there are many experiences in which map information about a city has been read as new environmental information in the information processing system 11, and such environmental information has been recorded as existing information. Then, there may be a case in which map information about a new city is read as new environmental information (newly input information) in the information processing system 11, and a route search is requested in the new city.

[0192] In such a case, the difference in environmental information, i.e., the magnitude of the context-based prediction error (error magnitude k) is medium ("medium"), so reinforcement learning of the learning model (execution of new learning) is performed.

[0193] During reinforcement learning, the learning unit 28 uses new environmental information, existing learning models, and reward information to determine as a hypothesis the route from the starting position to the destination position that is considered to be optimal and that matches the purpose indicated by the reward information.

[0194] Then, the learning unit 28 appropriately collects data such as environmental information required for reinforcement learning when taking action based on the obtained hypothesis, that is, when traveling along the hypothesised route, via the behavior unit 21 and the like.

[0195] When collecting data, for example, environmental information required for reinforcement learning is acquired (sensed) by sensors installed inside or outside the information processing system 11, or the vehicle is controlled to travel slowly or at different speeds to obtain data under various conditions.

[0196] Furthermore, for example, the learning unit 28 acquires the actual driving results (trial results), that is, the reward (reward amount) for the hypothesis from an input by the user or the like, or finds it from reward information.

[0197] Once information necessary for reinforcement learning, such as environmental information, actions (hypothesis), and the amount of reward for the actions (hypothesis), is obtained in this way, the learning unit 28 performs reinforcement learning of the learning model based on that information, the existing learning model, the new input information, the existing information, and the magnitude of pleasure V.

[0198] Furthermore, for example, if the error determination unit 25 determines that the magnitude k of the error is "large," the information processing system 11 determines that reinforcement learning cannot be performed to obtain a learning model that determines an appropriate behavior for newly input information, and therefore performs avoidance behavior. In other words, if the magnitude k of the error is determined to be "large," reinforcement learning is not performed and avoidance behavior is performed.

[0199] For example, the following cases are possible when the magnitude k of the error is "large."

[0200] That is, the information processing system 11 has many experiences of reading map information about large cities as new environmental information, and such environmental information is recorded as existing information. In such a situation, the information processing system 11 may read map information about a small regional city or a foreign city as new environmental information (newly input information), and a route search for the new city may be requested.

[0201] In such cases, for example, the map in the new map information may include narrow roads such as mountain paths, but the city in the existing map information may not have narrow roads such as mountain paths, making it difficult to search for an appropriate route using existing learning model methods.

[0202] Furthermore, for example, if the city in the new map information and the city in the existing map information are in different countries and have different traffic rules, it becomes difficult to search for an appropriate route using existing learning model methods.

[0203] Therefore, if the magnitude of the error k is "large", avoidance action is taken.

[0204] A specific example of the avoidance action is, as described above, a process of presenting the user with alternative solutions such as making an inquiry to an external system and prompting the user to make an appropriate selection.

[0205] In addition, for example, a process corresponding to avoidance behavior may be performed in which an action is determined (a route is searched) based on an existing learning model and reward information, and environmental information as newly input information, and the resulting route is presented to the user.

[0206] In this case, the decision of whether to actually drive along the presented route, i.e., whether to actually execute the action, is left to the user. Furthermore, for example, when a drive (trial) along the presented route is actually performed, the user may also be allowed to decide whether to use the information obtained from the actual trial and the selected action (route) for subsequent reinforcement learning of the learning model.

[0207] Next, we consider the case where only perception-based prediction errors are detected.

[0208] In this example, the same information as in the example where only a context-based prediction error is detected, that is, the location information and map information of a destination such as a hospital, is assumed to be the environmental information.

[0209] For example, suppose that as a result of comparing (collating) the reward information as newly input information with the reward information included in the existing information, the objective as an evaluation function, that is, the objective of the behavior indicated by the reward information, has been changed.

[0210] Specifically, a change in objective may occur, for example, when the objective of the action indicated by the reward information is changed from reaching the destination in the shortest time to heading to the destination with as little shaking as possible because there is a sick person.

[0211] In this example, the objective as an evaluation function (the objective of the behavior indicated by the reward information) is not one but a set of multiple conditions, that is, KPIs (Key Performance Indicators).

[0212] Specifically, for example, assume that the KPIs indicated by the existing evaluation function are A, B, and C, and the KPIs indicated by the new evaluation function are B, C, D, and E.

[0213] In such a case, for example, the perception-based prediction error detection unit 32 calculates the perception-based prediction error by dividing the number of KPIs that differ between the existing evaluation function and the new evaluation function by the larger number of KPIs in either the existing evaluation function or the new evaluation function.

[0214] In addition, when only a perception-based prediction error is detected, the prediction error detection unit 24 directly calculates the perception-based prediction error, that is, the difference between the evaluation function as the newly input information and the evaluation function included in the existing information, as the prediction error e t and the prediction error e t Let the magnitude of the error be k.

[0215] If the error determination unit 25 determines that the magnitude of the error k is "small," the same process as in the case where only a context-based prediction error is detected is performed. That is, reinforcement learning is not performed, and an existing learning model is used to select an action.

[0216] Furthermore, for example, when the error determination unit 25 determines that the magnitude k of the error is "medium," reinforcement learning of the learning model is performed in the information processing system 11. That is, the learning model is updated.

[0217] Even when the error magnitude k is "medium," the process is essentially the same as in the case where only a context-based prediction error is detected: data necessary for reinforcement learning is collected as appropriate and reinforcement learning is performed.

[0218] However, during reinforcement learning, the collected data such as environmental information and the reward amount calculated from the new evaluation function are used in accordance with the new evaluation function. At this time, if necessary, the user may be inquired about the reward amount calculated from the new evaluation function, such as whether the reward amount is appropriate, or whether the behavior (correct answer data) corresponding to the output of the learning model is correct.

[0219] During reinforcement learning, an evaluation is also made as to whether the behavior (searched path) that is the output of the learning model can be evaluated using a new evaluation function.

[0220] As described above, when a cognitive-based prediction error is detected and the learning model is updated, reinforcement learning (updating the learning model) results in a learning model that evaluates behavior based on a new (novel) evaluation function.

[0221] Furthermore, for example, if the error determination unit 25 determines that the magnitude of the error k is "large," the same processing as when only a context-based prediction error is detected is performed. That is, reinforcement learning is not performed, and avoidance behavior is selected.

[0222] As mentioned above, the prediction error e t This involves contextual deviation (context-based prediction error) and cognitive deviation (cognitive-based prediction error).

[0223] In the learning model obtained by reinforcement learning, the prediction error e t The number and content of behaviors that can be output from the learning model, i.e., the population of candidate behaviors, changes depending on whether the behavior is due to contextual misalignment or cognitive misalignment. This is because the objective function (evaluation function), i.e., KPIs, that must be satisfied change depending on contextual misalignment and cognitive misalignment.

[0224] Furthermore, for example, if there is a cognitive discrepancy, the options (candidate actions), i.e., the output of the learning model, will change depending on the magnitude of the cognitive discrepancy (cognitive-based prediction error).

[0225] For example, when the cognition-based prediction error is small, options (candidate actions) that satisfy the existing evaluation function appear. On the other hand, when the cognition-based prediction error is medium, new conditions (KPIs) are added to the existing conditions (KPIs), so the number of candidate actions is smaller than when the cognition-based prediction error is small.

[0226] <Application example> The present technology described above can be applied to various technologies.

[0227] Specifically, this technology can be applied to general control based on online reinforcement learning, factory picking, robot operation, autonomous driving, drone control, conversation, and recognition systems.

[0228] For example, as an example of control based on online reinforcement learning, this technology can be applied to autofocus motor control in digital cameras, control of the operation of robots, and other various control systems.

[0229] Furthermore, when it comes to picking items in factories, for example, this technology can be used to increase the number of items that a picking machine can grasp through reinforcement learning, even if the properties of the object to be picked, such as shape, softness, or slipperiness, change.

[0230] In addition, by using this technology, it will become possible to perform a wide range of tasks, from simple to complex, including the objectives (goals) of actions, such as picking items without breaking them, moving them without spilling, or moving them quickly.

[0231] Furthermore, when applying this technology to autonomous driving, driving control can also be performed using other variables such as data obtained through a CAN (Controller Area Network), the behavior of other vehicles obtained through sensing, the state of the driver (user), and information obtained from the infrastructure.

[0232] Here, data obtained through CAN includes, for example, data on the accelerator, brake, steering wheel, vehicle body inclination, fuel consumption, etc., and the user's condition includes, for example, stress, drowsiness, fatigue, motion sickness, and comfort level, which are obtained from in-vehicle cameras and biosensors. Information obtained from the infrastructure includes, for example, traffic congestion information and information on the provision of in-vehicle related services.

[0233] If this technology is applied to autonomous driving, it will become possible to improve accuracy in terms of, for example, "avoiding collisions with people" and "avoiding accidents," as well as control specific and complex micro / macro conditions such as "ride comfort" and "optimization across the entire urban transportation network."

[0234] Furthermore, if this technology is applied to drone control, it will be possible to achieve control based on external disturbances such as attitude and wind, topographical data, GPS (Global Positioning System) information, and regional weather conditions, as well as improved accuracy for specific purposes, diversification of purposes, and swarm control of drones.

[0235] Furthermore, this technology can also be applied to conversational guide robots, call center automation, chat robots, and chat robots.

[0236] In such cases, it is possible to improve the appropriateness of the conversation depending on the situation, for example, whether it is an appropriate response or an interesting conversation, as well as to realize more diverse and flexible responses to users and situations, and to changes in the situation.

[0237] This technology can also be applied to recognition systems that monitor the status of the environment, people, etc. In such cases, it will not only improve the accuracy of recognition, but also enable the system to respond more flexibly to a wider variety of users and situations, as well as to changes in the situation.

[0238] This technology can also be applied to robot control in general, making it possible to realize, for example, human-like or animal-like robots.

[0239] More specifically, this technology can realize, for example, a robot that learns spontaneously without having to set learning content, a robot that starts and stops learning according to its interests, a robot that remembers what it is interested in and the content it remembers is also influenced by its interests, etc. Furthermore, this technology can also realize, for example, a robot that is curious but also gets bored, a robot that self-monitors and tries hard or gives up, and an animal robot such as a pet cat.

[0240] In addition, this technology can be applied to assisting humans in overcoming boredom in learning, and to autism models by setting thresholds in attention networks.

[0241] <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0242] FIG. 6 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

[0243] In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0244] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0245] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a nonvolatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0246] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0247] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0248] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0249] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0250] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0251] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0252] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0253] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0254] Furthermore, the present technology can also be configured as follows.

[0255] (1) An information processing system that determines an action based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating the action, an error detection unit that determines the magnitude of a difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function; a learning unit that updates the learning model based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation in accordance with the behavior, in accordance with the magnitude of the difference; An information processing system comprising: (2) a determination unit that determines whether the magnitude of the difference is large, medium, or small; The learning unit updates the learning model when the magnitude of the difference is medium. The information processing system according to (1). (3) When the magnitude of the difference is medium, the learning unit updates the learning model according to the magnitude of a pleasure determined by the difference between the reward amount based on the newly input environmental information or the evaluation function and the reward amount based on the existing evaluation function. (2) An information processing system according to the present invention. (4) The learning unit updates the learning model when the magnitude of the pleasantness is equal to or greater than a predetermined threshold. (3) An information processing system according to the present invention. (5) The learning unit updates the learning model with a weighting according to the magnitude of the pleasantness. (4) An information processing system according to (4). (6) The learning unit does not update the learning model when the magnitude of the pleasantness is less than the threshold. An information processing system according to (4) or (5). (7) The learning unit does not update the learning model when the magnitude of the difference is small. An information processing system according to any one of (2) to (6). (8) If the magnitude of the difference is small, a behavior unit determines an action based on the newly input environmental information or the evaluation function and the learning model. (7) An information processing system according to (7). (9) The learning unit does not update the learning model when the magnitude of the difference is large. An information processing system according to any one of (2) to (8). (10) If the magnitude of the difference is large, the behavior is not determined based on the learning model. (9) An information processing system according to (9). (11) The error detection unit determines, as the magnitude of the difference, the magnitude of a context-based error caused by a deviation in the environmental information or the magnitude of a perception-based error caused by a deviation in the evaluation function. An information processing system according to any one of (1) to (10). (12) When an error in the recognition base is detected, the learning unit performs the update so as to obtain the learning model based on the newly input evaluation function. (11) An information processing system according to (11). (13) When an error in the perception base is detected, the learning unit updates the learning model so as to suppress use of the existing evaluation function. The information processing system according to (11) or (12). (14) When the context-based error is detected, the learning unit performs the update so as to obtain the learning model that incorporates the change in the environmental information. An information processing system according to any one of (11) to (13). (15) When the cognitive-based error is detected, the learning model is more likely to be updated than when the context-based error is detected. An information processing system according to any one of (11) to (14). (16) an information processing system that determines an action based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating the action, determining the magnitude of the difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function; The learning model is updated based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation according to the behavior, depending on the magnitude of the difference. Information processing methods. (17) a computer that controls an information processing system that determines an action based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating an action; determining the magnitude of the difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function; The learning model is updated based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation according to the behavior, depending on the magnitude of the difference. A program that executes a process. [Explanation of symbols]

[0256] 11 Information processing system, 21 Behavior module, 22 Recording module, 23 Matching module, 24 Prediction error detection module, 25 Error judgment module, 26 Reward matching module, 27 Pleasure judgment module, 28 Learning module, 31 Context-based prediction error detection module, 32 Cognition-based prediction error detection module, 33 Curiosity module, 34 Memory module

Claims

1. An information processing system that determines an action based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating the action, an error detection unit that determines the magnitude of a difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function; a determination unit that determines whether the magnitude of the difference is large, medium, or small; a learning unit that updates the learning model based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation in accordance with the action, when the magnitude of the difference is medium; An information processing system comprising:

2. When the magnitude of the difference is medium, the learning unit updates the learning model according to the magnitude of a pleasure determined by the difference between the reward amount based on the newly input environmental information or the evaluation function and the reward amount based on the existing evaluation function. The information processing system according to claim 1 .

3. The learning unit updates the learning model when the magnitude of the pleasantness is equal to or greater than a predetermined threshold. The information processing system according to claim 2 .

4. The learning unit updates the learning model with a weighting according to the magnitude of the pleasantness. The information processing system according to claim 3 .

5. The learning unit does not update the learning model when the magnitude of the pleasantness is less than the threshold. The information processing system according to claim 3 .

6. The learning unit does not update the learning model when the magnitude of the difference is small. The information processing system according to claim 1 .

7. If the magnitude of the difference is small, a behavior unit determines an action based on the newly input environmental information or the evaluation function and the learning model. The information processing system according to claim 6.

8. The learning unit does not update the learning model when the magnitude of the difference is large. The information processing system according to claim 1 .

9. If the magnitude of the difference is large, the behavior is not determined based on the learning model. The information processing system according to claim 8 .

10. The error detection unit determines, as the magnitude of the difference, the magnitude of a context-based error caused by a deviation in the environmental information or the magnitude of a perception-based error caused by a deviation in the evaluation function. The information processing system according to claim 1 .

11. When an error in the recognition base is detected, the learning unit performs the update so as to obtain the learning model based on the newly input evaluation function. The information processing system according to claim 10.

12. When an error in the perception base is detected, the learning unit updates the learning model so as to suppress use of the existing evaluation function. The information processing system according to claim 10.

13. When the context-based error is detected, the learning unit performs the update so as to obtain the learning model that incorporates the change in the environmental information. The information processing system according to claim 10.

14. an information processing system that determines an action based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating the action, determining the magnitude of the difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function; determining whether the magnitude of the difference is large, medium, or small; If the magnitude of the difference is medium, the learning model is updated based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation according to the action. Information processing methods.

15. a computer that controls an information processing system that determines an action based on environmental information and a learning model obtained by learning based on an evaluation function for evaluating an action; determining the magnitude of the difference between the newly input environmental information or the evaluation function and the existing environmental information or the evaluation function; determining whether the magnitude of the difference is large, medium, or small; If the magnitude of the difference is medium, the learning model is updated based on the newly input environmental information or the evaluation function and the amount of reward obtained by the evaluation according to the action. A program that executes a process.

Images