Agent program, server, and learning method for performing reinforcement learning while estimating part of state information

Abstract

To provide a program, a server and a reinforcement learning method that estimate action information even when an agent entity of reinforcement learning can only receive a part of state information and a compensation value from an environment entity.SOLUTION: Provided is an agent entity 1 having a reinforcement learning engine 10 which is realized by executing a program that causes a computer mounted in a device serving as an agent to function, a supervised learning engine 11, a determination unit 12, and a second state information request unit 13. The reinforcement learning engine 10 trains a reinforcement learning model by the "first state information," "second state information," and "compensation value" included in a first dataset, confirms "action information" with which the compensation value becomes maximum, and replies to an environment entity 2. The supervised learning engine 11 performs training as a training stage taking the "first state information" and "compensation value" included in the first dataset as explanatory variables and taking the "second state information" as an objective variable.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to agent technology in reinforcement learning, and is particularly suited to applications in which a server acting as an agent entity transmits intervention information that influences behavioral change to a user operating a terminal acting as an environmental entity. [Background technology]

[0002] Reinforcement learning is a technique in which an agent entity repeatedly trains and estimates based on the experience collected by the environmental entity. Specifically, it is possible to maximize the reward value of a task in the environmental entity through repeated interactions between the environmental entity and the agent entity. Unlike supervised learning and unsupervised learning, this technique does not require the preparation of data samples in advance. In other words, there is no need to collect, preprocess, or label data samples in advance. Reinforcement learning can continue learning on its own without human intervention by providing appropriate incentives.

[0003] Figure 1 shows a general reinforcement learning framework.

[0004] In reinforcement learning, an environment entity generally transmits detected "state information" and "reward value" to an agent entity. In response, the agent entity estimates "action information" that maximizes the reward value and returns it to the environment entity. After executing the "action information," the environment entity sends the newly detected "state information" and "reward value" to the agent entity. In response, the agent entity trains a reinforcement learning model using the "state information" and "reward value" that result from the "action information," while estimating new "action information" that maximizes the reward value and returning it to the environment entity. This process is repeated.

[0005] Conventionally, there is a technology that outputs the next recommended health state target value based on the target health state value and the measured value of the user's health state (see, for example, Patent Document 1). According to this technology, the next health state target value is determined based on the current health state and a preset ideal future health state. Appropriate intervention content is presented to the user while shifting the target health state value to be more effective.

[0006] There is also technology for a system that supports behavioral change in subjects (see, for example, Patent Document 2). According to this technology, for each user, multiple messages linked to behavioral change techniques are generated for each behavioral change factor. These messages are output to the user once or multiple times. For users who respond to the message, a behavioral change technique that acts on the behavioral change factor is selected, and a message linked to it is generated. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-035365 [Patent Document 2] Patent No. 7062327 [Non-patent literature]

[0008] [Non-Patent Document 1] "Reinforcement learning", [online], [Retrieved February 25, 2023], Internet<URL:https: / / ja.wikipedia.org / wiki / %E5%BC%B7%E5%8C%96%E5%AD%A6%E7%BF%92> [Non-patent document 2] "Supervised learning", [online], [searched February 25, 2023], Internet <URL:https: / / ja.wikipedia.org / wiki / %E6%95%99%E5%B8%AB%E3%81%82%E3%82%8A%E5%AD%A6%E7%BF%92> Summary of the Invention [Problem to be solved by the invention]

[0009] According to the aforementioned Patent Document 1, the estimation device receives the current health condition of the user and returns the next target value. Similarly, the aforementioned Patent Document 2 also receives response information from the subject and returns a message according to the behavioral change factor.

[0010] In response to this, the inventors of the present application have considered that in reinforcement learning, some state information transmitted by environmental entities may not always be transmitted. For example, consider a case where the environmental entity is a terminal that can be operated by a user. In this case, state information such as the user's physical state can be automatically acquired at all times, for example, by a sensor, and can be transmitted to the agent entity. On the other hand, state information such as the user's psychological state must be asked to the user, for example, through a questionnaire, and cannot be acquired at all times. This is because questionnaires place a heavy burden on the user and take time to answer. The inventors of the present application considered that in the case of a system that encourages users to change their behavior, the psychological state of the user will have a large influence even if the user's physical state is the same.

[0011] Furthermore, according to a reinforcement learning framework, an agent entity typically trains a reinforcement learning model for each environment entity. In response to this, the inventors of the present application have considered that state information and behavioral information corresponding to reward values ​​may be common to multiple environmental entities.

[0012] Therefore, the present invention aims to provide a program, a server, and a reinforcement learning method that can train a reinforcement learning model common to one or more environmental entities as an agent entity for reinforcement learning, and estimate behavioral information even when only some state information and reward values ​​can be received from the environmental entities. [Means for solving the problem]

[0013] According to the present invention, there is provided a program for causing a computer to function as an agent entity that communicates with an environmental entity in reinforcement learning, the program comprising: upon receiving a first data set from an environment entity, the first data set including first state information, second state information, and a reward value; a reinforcement learning engine that trains a reinforcement learning model using the first state information, the second state information, and the reward value included in the first data set, estimates action information that maximizes the reward value, and returns the action information to the environmental entity; a supervised learning engine that trains a supervised learning model using the first state information and the reward value included in the first dataset as explanatory variables and the second state information as a target variable; and make it work, upon receiving a second data set from the environment entity, the second data set including the first state information and the reward value; the supervised learning engine receives the first state information and the reward value included in the second dataset and estimates the second state information; The reinforcement learning engine trains a reinforcement learning model using the first state information and reward value included in the second dataset and the second state information estimated by the supervised learning engine, estimates action information that maximizes the reward value, and returns the action information to the environmental entity. The present invention is characterized in that the computer functions as follows.

[0014] According to another embodiment of the program of the present invention, a determining means for determining whether a difference between the second state information estimated by the supervised learning engine and the second state information included in the first data set received before the second data set is equal to or less than a predetermined threshold; and make it work, When the determination means determines that the answer is true, the reinforcement learning engine trains a reinforcement learning model using the first state information and reward value included in the second data set and the second state information estimated by the supervised learning engine, and estimates action information that maximizes the reward value. It is also preferable to make the computer function in this way.

[0015] According to another embodiment of the program of the present invention, a second state information requesting means for requesting the environmental entity to transmit a first data set including second state information when the determination means determines that the result is false; It is also preferable to make the computer function as described above.

[0016] According to another embodiment of the program of the present invention, An agent entity communicates with one or more environmental entities, The reinforcement learning engine trains a common reinforcement learning model for one or more environment entities, A supervised learning engine trains a common supervised learning model for one or more environmental entities. It is also preferable to make the computer function in this way.

[0017] According to another embodiment of the program of the present invention, As a distributed reinforcement learning method, the method further includes a master agent entity that communicates with the plurality of agent entities, The agent entity communicates with one or more environment entities and transmits a reinforcement learning model trained with the environment entities to a master agent entity; The master agent entity generates a master reinforcement learning model that integrates the reinforcement learning models received from the multiple agent entities, and distributes the master reinforcement learning model to each agent entity. It is also preferable to make the computer function in this way.

[0018] According to another embodiment of the program of the present invention, The environmental entity is a terminal, The agent entity is a server It is also preferable to make the computer function in this way.

[0019] According to another embodiment of the program of the present invention, The terminal is operated by the user, The first state information is physical state information of the user, The second state information is psychological state information of the user, The reward value is the user's behavioral state information, Behavioral information is intervention information that influences user behavior change. It is also preferable to make the computer function in this way.

[0020] According to another embodiment of the program of the present invention, The physical state information is the distance traveled measured by the device's sensor, The psychological state information is a questionnaire result answered to questions displayed on the terminal display, The behavioral status information is the number of steps measured by the device sensor. It is also preferable to make the computer function in this way.

[0021] According to another embodiment of the program of the present invention, The physical condition information is first biological information measured by a sensor of the terminal, The psychological state information is a questionnaire result answered to questions displayed on the terminal display, The behavioral state information is the second biological information measured by the device sensor. It is also preferable to make the computer function in this way.

[0022] According to the present invention, in a server serving as an agent entity that communicates with a terminal serving as an environment entity in reinforcement learning, Upon receiving a first data set from the terminal, the first data set includes first state information, second state information, and a reward value; a reinforcement learning engine that trains a reinforcement learning model using the first state information, the second state information, and the reward value included in the first data set, estimates action information that maximizes the reward value, and returns the action information to the terminal; a supervised learning engine that trains a supervised learning model using the first state information and the reward value included in the first dataset as explanatory variables and the second state information as a target variable; and Upon receiving a second data set from the terminal, the second data set includes the first state information and the reward value. the supervised learning engine receives the first state information and the reward value included in the second dataset and estimates the second state information; The reinforcement learning engine trains a reinforcement learning model using the first state information and reward value included in the second data set and the second state information estimated by the supervised learning engine, estimates action information that maximizes the reward value, and returns the action information to the terminal. It is characterized by:

[0023] According to the present invention, in a reinforcement learning method for a server serving as an agent entity that communicates with a terminal serving as an environmental entity, The server Upon receiving a first data set from the terminal, the first data set includes first state information, second state information, and a reward value; Using a reinforcement learning engine, train a reinforcement learning model using the first state information, the second state information, and the reward value included in the first dataset, estimate action information that maximizes the reward value, and return the action information to the environmental entity; Using a supervised learning engine, train a supervised learning model using the first state information and the reward value included in the first dataset as explanatory variables and the second state information as a response variable; Upon receiving a second data set from the terminal, the second data set includes the first state information and the reward value. using a supervised learning engine, inputting the first state information and the reward value included in the second dataset, and estimating second state information; Using a reinforcement learning engine, a reinforcement learning model is trained using the first state information and reward value included in the second data set and the second state information estimated by the supervised learning engine, and action information that maximizes the reward value is estimated and returned to the terminal. A server characterized by: [Effects of the Invention]

[0024] According to the program, server, and reinforcement learning method of the present invention, a reinforcement learning model common to one or more environmental entities is trained as an agent entity for reinforcement learning, and behavioral information can be estimated even when only some state information and reward values ​​can be received from the environmental entities. [Brief explanation of the drawings]

[0025] [Figure 1] A general reinforcement learning framework. [Figure 2] FIG. 1 is a diagram showing the configuration of a first entity in reinforcement learning according to the present invention. [Figure 3] FIG. 10 is a diagram showing a second entity configuration of reinforcement learning in the present invention. [Figure 4] FIG. 3 is a first explanatory diagram of the walking assistant in FIG. 2. [Figure 5] FIG. 4 is a second explanatory diagram of the walking assistant in FIG. 3. [Figure 6] This is a framework in which the present invention is applied to a distributed reinforcement learning method. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0027] FIG. 2 is a diagram showing the configuration of a first entity of reinforcement learning in the present invention.

[0028] FIG. 2 shows the configuration of a reinforcement learning system, which consists of an agent entity 1 and one or more environment entities 2. The agent entity 1 of the present invention comprises a reinforcement learning engine 10, a supervised learning engine 11, a determination unit 12, and a second state information request unit 13. These functional components are realized by executing a program that causes a computer installed in the device that serves as the agent to function. The processing flow of these functional components can also be understood as a learning method in the agent device.

[0029] The agent entity 1 receives the following two patterns of data sets from the environment entity 2: First dataset: First state information, Second state information, Reward value Second dataset: First state information, reward value

[0030] 2, this represents a case where the agent entity 1 receives a "first data set" from the environment entity 2. It is assumed that the second state information is not always receivable from the environment entity 2.

[0031] [Reinforcement Learning Engine 10] The reinforcement learning engine 10 trains a reinforcement learning model using the "first state information", "second state information" and "reward value" contained in the first data set, estimates the "action information" that maximizes the reward value, and returns it to the environment entity 2. This function is based on general reinforcement learning (see, for example, Non-Patent Document 1). Furthermore, the reinforcement learning engine 10 does not train a reinforcement learning model for each environment entity 2, but trains a common reinforcement learning model for one or more environment entities 2. In other words, the reinforcement learning engine 10 does not identify the environment entity 2 that is the source of the first state information and the reward value.

[0032] [Supervised Learning Engine 11] In the training stage, the supervised learning engine 11 performs training using the "first state information" and the "reward value" included in the first dataset as explanatory variables and the "second state information" as a target variable. This function is based on general supervised learning (see, for example, Non-Patent Document 2). The supervised learning engine 11 also trains a common supervised learning model for one or more environment entities 2, i.e., it does not identify the environment entity 2 from which the first state information and reward value are sent.

[0033] [Judgment section 12] 2, the determination unit 12 receives the "second state information" for each environment entity 2 and stores the last received second state information. This will be described later with reference to FIG.

[0034] FIG. 3 is a diagram showing the second entity configuration of reinforcement learning in the present invention. FIG. 3 shows a case where the agent entity 1 receives a "second data set" including first state information and a reward value from the environment entity 2.

[0035] [Supervised Learning Engine 11] In the estimation stage, the supervised learning engine 11 inputs the "first state information" and the "reward value" contained in the second data set, and estimates the "second state information."

[0036] Here, there are the following two embodiments. (First embodiment) Supervised learning engine 11 outputs the estimated "second state information" to reinforcement learning engine 10 as is. Then, the reinforcement learning engine 10 trains a reinforcement learning model using the "first state information" and "reward value" contained in the second dataset and the "second state information" estimated by the supervised learning engine 11, estimates the "action information" that maximizes the reward value, and returns it to the environmental entity 2.

[0037] Second Embodiment: Supervised learning engine 11 outputs the estimated “second state information” to determination unit 12. [Judgment section 12] The determination unit 12 determines whether the difference between the "second state information" estimated by the supervised learning engine 11 and the "second state information" included in the first data set received before the second data set is less than or equal to a predetermined threshold.

[0038] If the determination unit 12 determines that this is true (the difference between the second state information is small), it outputs the "second state information" estimated by the supervised learning engine 11 to the reinforcement learning engine 10. This means that the second state information estimated from the second data set (the first state information and the reward value) is compared with and similar to the second state information of the first data set received before the second data set. The determination unit 12 stores the previously received second state information in advance, as shown in FIG. 2. Then, the reinforcement learning engine 10 trains a reinforcement learning model using the "first state information" and "reward value" contained in the second data set and the "second state information" estimated by the supervised learning engine 11, and estimates the "behavioral information" that maximizes the reward value.

[0039] If the determination unit 12 determines that the result is false (the difference between the second state information is large), it outputs a message to that effect to the second state information request unit 13. This means that the second state information estimated by the second data set (the first state information and the reward value) is different from the second state information of the first data set received before the second data set. [Second state information request unit 13] The second state information requesting unit 13 requests the environment entity 2 to transmit a first data set including the second state information, thereby prompting the environment entity 2 to transmit the first data set (first state information, second state information, reward value).

[0040] FIG. 4 is a first explanatory diagram of the walking assistant in FIG.

[0041] As shown in Figure 4, the reinforcement learning framework of the present invention is applied to user behavior change. The agent entity 1 is configured as a server, and the environment entity 2 is configured as a terminal operated by the user.

[0042] Here, assuming application to user behavioral changes, the information contained in the data set exchanged between the server 1 and the terminal 2 is defined as follows. First state information: User's physical state information Second state information: user's psychological state information Reward value: User behavioral state information Behavioral information: Intervention information that influences user behavior change

[0043] The first state information (physical state information of the user) and the reward value (behavioral state information of the user) are assumed to be data that can be detected at any time by, for example, a sensor or a device. Therefore, the terminal 2 can transmit the first state information and the reward value to the agent entity 1 at any time. In contrast, the second state information (user's mental state information) is data on the user's inner psychology that cannot be acquired by a sensor, such as a questionnaire obtained by asking the user questions. Considering the burden on the user to respond, the second state information cannot be transmitted to the agent entity at any time.

[0044] Furthermore, if the state information of an environmental entity is assumed to be the physical and psychological information of a user, it is likely that there is some correlation between the physical and psychological information common to many users. For this reason, the supervised learning engine 11 trains a supervised learning model common to one or more users.

[0045] Here, the "intervention information" as behavioral information may be a message and its frequency according to the strength to influence the user's behavioral change. The reinforcement learning engine 10 determines the strength of the intervention information as behavioral information so as to maximize the user's "behavioral state information" as a reward value.

[0046] As a first specific example, referring to FIG. 4, assuming a service of LX (Life Transformation) or healthcare, the following is defined to encourage users to walk. Physical status information: distance traveled measured by device sensors Psychological state information: Answers to questions displayed on the device display Survey results Activity status information: number of steps measured by the device sensor Intervention information: Walking encouragement message According to this specific example, the server 1 uses the reinforcement learning engine 10 to train a reinforcement learning model using the "distance traveled by the user," "results of a questionnaire from the user," and "number of steps taken by the user" received from the terminal 2. The server 1 then estimates a "message encouraging walking" that will maximize the number of steps taken by the user, and returns this to the terminal 2. Note that while the distance traveled and the number of steps can be detected at any time by a sensor, the results of the questionnaire to the user cannot be obtained at any time.

[0047] As shown in Figure 4, the reinforcement learning engine 10 inputs the travel distance, questionnaire results, and number of steps for each user for each date and time, and estimates intervention information. The travel distance, questionnaire results, number of steps, and intervention information are each expressed in normalized form rather than in quantitative form. Normalizing to a predetermined range enables relative evaluation. The intervention information may change the assertion strength of the message depending on the strength of the intervention information. The intervention information may also change the number of times the message is sent depending on the strength of the intervention information.

[0048] A second specific example is not limited to that shown in FIG. 4, and may be as follows. Physical condition information: Primary biological information measured by the device's sensors Psychological state information: Answers to questions displayed on the device display Survey results Behavioral state information: Secondary biological information measured by the device's sensors The biological information may be information that can be acquired by a sensor, such as heart rate, blood oxygen concentration, skin temperature, blood pressure, sweat rate, activity level, stress level, brain waves, etc. Terminal 2 may acquire such biological information from a device worn by the user, such as a smart watch or a headset.

[0049] In the first and second specific examples, if the psychological state information is a questionnaire result, the burden of answering the questionnaire is placed on the user. Therefore, it is not possible to constantly ask the user to answer the questionnaire. According to the present invention, even if the number of times the user answers the questionnaire is small, the reinforcement learning engine 10 can return intervention information (behavioral information) to the terminal 2, which is an environmental entity.

[0050] FIG. 5 is a second explanatory diagram of the walking assistant in FIG.

[0051] 5, it is assumed that the determination unit 12 is set to a "predetermined threshold value 2." It is also assumed that the values ​​of the travel distance, questionnaire, number of steps, and intervention information are normalized within the range of 0-15.

[0052] As of 15:00, user 001 has received the following: 12:00: Distance traveled: 10, Survey: 2, Steps: 8 15:00: Distance traveled: 13, , steps: 7 At this time, it is assumed that supervised learning engine 11 estimates questionnaire 3 as the second state information for user 001 at 15:00. 12:00: Distance traveled: 10, Survey: 2, Steps: 8 15:00: Distance traveled: 13, estimated (questionnaire: 3), steps: 7 The determination unit 12 determines whether the difference between the estimated questionnaire 3 and questionnaire 2 included in the first data set received before the second data set is equal to or less than a predetermined threshold 2. Here, the difference 1 (= 3 - 2) is equal to or less than the predetermined threshold 2, so the estimated questionnaire 3 is input to the reinforcement learning engine 10 as second state information. As a result, the reinforcement learning engine 10 estimates intervention information from the travel distance 13, questionnaire 3, and number of steps 7.

[0053] Also, at 15:00, the following message has been received for user 002: 12:00: Distance traveled 5, Survey 5, Steps 5 15:00: Distance traveled: 7, , steps: 3 At this time, it is assumed that for user 002, supervised learning engine 11 infers questionnaire 8 as the second state information at 15:00. 12:00: Distance traveled 5, Survey 5, Steps 5 15:00: Distance traveled: 7, Estimated (questionnaire: 8), Steps: 3 The determination unit 12 determines whether the difference between the estimated questionnaire 8 and the questionnaire 5 included in the first dataset received before the second dataset is equal to or less than a predetermined threshold 2. Here, the difference 3 (= 8 - 5) is greater than the predetermined threshold 2, so the second state information request unit 13 sends a questionnaire response request to the terminal 2 of the user 002. At this time, there is a high possibility that the psychological state (second state information) deviates from the value estimated by the supervised learning engine 11. The terminal 2 of the user 002 displays the questionnaire and asks the user to respond. The server 1 then waits for the first dataset (travel distance, questionnaire, number of steps) from the terminal 2 and attempts to train using the reinforcement learning engine 10 with the accurate questionnaire value.

[0054] Furthermore, at 15:00, user 003 received the following: 12:00: Distance traveled 4, Survey 4, Steps 5 15:00: Distance traveled: 5, , steps: 6 At this time, it is assumed that for user 003, supervised learning engine 11 infers questionnaire 5 as the second state information at 15:00. 12:00: Distance traveled 4, Survey 4, Steps 5 15:00: Distance traveled 5, Estimated (survey 5), Steps 6 The determination unit 12 determines whether the difference between the estimated questionnaire 5 and the questionnaire 4 included in the first data set received before the second data set is equal to or less than a predetermined threshold 2. Here, since the difference 1 (= 5 - 4) is equal to or less than the predetermined threshold 2, the estimated questionnaire 5 is input to the reinforcement learning engine 10 as second state information. As a result, the reinforcement learning engine 10 estimates intervention information from the travel distance 5, the questionnaire 5, and the number of steps 6.

[0055] FIG. 6 shows a framework in which the present invention is applied to a distributed reinforcement learning method.

[0056] An agent entity 1 communicates with one or more environment entities 2, forming a framework for a reinforcement learning engine. For example, the agent entity 1 and the environment entities 2 are running on a device (such as a smartphone) owned by a user. In this case, the device always executes synchronously with the user through information exchange.

[0057] According to FIG. 6, the distributed reinforcement learning system further includes a master agent entity 3 that communicates with the multiple agent entities 1. The agent entity 1 sends the reinforcement learning model (e.g., parameters, gradients, etc.) that it has trained with one or more environment entities 2 to the master agent entity 3. The master agent entity 3 generates a master reinforcement learning model that integrates the reinforcement learning models received from multiple agent entities 2. Then, the master reinforcement learning model is distributed to each agent entity 1, and used as a reinforcement learning model. When the agent entity 1 and the environment entity 2 are running on a user's device, the master agent entity running on the server generates a master reinforcement learning model asynchronously with the device. This makes it possible to build a master reinforcement learning model based on broad knowledge, unlike individual reinforcement learning frameworks.

[0058] As described above in detail, the program, server, and reinforcement learning method of the present invention allow a reinforcement learning model common to one or more environmental entities to be trained as a reinforcement learning agent entity, and allows behavioral information to be estimated even when only some state information and reward values ​​can be received from the environmental entities.

[0059] The present invention is applicable to services that encourage users to change their behavior, and is particularly suited to LX (Life Transformation) and healthcare services. Specifically, when the server uses a reinforcement learning framework to estimate intervention information (behavioral information) to encourage a user to change their behavior, it can take into account not only the user's physical state information (first state information) but also psychological state information (second state information) in response to a questionnaire given to the user. Generally, collecting psychological state information requires conducting a questionnaire that places a burden on the user to answer. In contrast, according to the present invention, it is possible to estimate psychological state information (second state information) from physical state information (first state information) and behavioral state information (reward value) and incorporate it into the reinforcement learning framework without constantly acquiring the psychological state.

[0060] Furthermore, according to the present invention, the reinforcement learning engine trains a common reinforcement learning model without identifying each environmental entity. This allows a common reinforcement learning model to be trained for multiple users in a relatively shorter training period, and the accuracy of the model can be improved. In other words, behavioral information can be estimated with relatively high accuracy even for environmental entities that have just started interacting with the agent entity.

[0061] Furthermore, this will enable the company to, for example, "encourage users to change their behavior," thereby contributing to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Ensure healthy lives and promote well-being for all at all ages."

[0062] With respect to the various embodiments of the present invention described above, various changes, modifications, and omissions that fall within the scope of the technical spirit and aspects of the present invention may be easily made by those skilled in the art. The above description is merely illustrative and is not intended to be limiting in any way. The present invention is limited only by the claims and their equivalents. [Explanation of symbols]

[0063] 1 Agent entity, server 10 Reinforcement Learning Engine 11 Supervised Learning Engine 12 Judgment section 13 Second status information request section 2 Environmental entities, terminals 3 Master Agent Entity

Claims

1. In a program for causing a computer to function as an agent entity that communicates with an environmental entity in reinforcement learning, upon receiving a first data set from an environment entity, the first data set including first state information, second state information, and a reward value; a reinforcement learning engine that trains a reinforcement learning model using the first state information, the second state information, and the reward value included in the first data set, estimates action information that maximizes the reward value, and returns the action information to the environmental entity; a supervised learning engine that trains a supervised learning model using the first state information and the reward value included in the first dataset as explanatory variables and the second state information as a target variable; and make it work, upon receiving a second data set from the environment entity, the second data set including the first state information and the reward value; the supervised learning engine receives the first state information and the reward value included in the second dataset and estimates the second state information; The reinforcement learning engine trains a reinforcement learning model using the first state information and reward value included in the second dataset and the second state information estimated by the supervised learning engine, estimates action information that maximizes the reward value, and returns the action information to the environmental entity. A program that causes a computer to function in such a way.

2. a determining means for determining whether a difference between the second state information estimated by the supervised learning engine and the second state information included in the first data set received before the second data set is equal to or less than a predetermined threshold; and make it work, When the determination means determines that the answer is true, the reinforcement learning engine trains a reinforcement learning model using the first state information and reward value included in the second data set and the second state information estimated by the supervised learning engine, and estimates action information that maximizes the reward value.

2. The program according to claim 1, which causes a computer to function as follows:

3. second state information request means for requesting the environmental entity to transmit a first data set including second state information when the determination means determines that the result is false; 3. The program according to claim 2, further causing a computer to function.

4. an agent entity that communicates with one or more environmental entities; a reinforcement learning engine for training a common reinforcement learning model for one or more environmental entities; The supervised learning engine trains a common supervised learning model for one or more environmental entities.

3. The program according to claim 1, wherein the program causes a computer to function as described above.

5. As a distributed reinforcement learning method, the method further includes a master agent entity that communicates with the plurality of agent entities, The agent entity communicates with one or more environment entities and transmits a reinforcement learning model trained with the environment entities to a master agent entity; The master agent entity generates a master reinforcement learning model that integrates the reinforcement learning models received from the multiple agent entities, and distributes the master reinforcement learning model to each agent entity.

3. The program according to claim 1, wherein the program causes a computer to function as described above.

6. The environmental entity is a terminal, The agent entity is a server 3. The program according to claim 1, wherein the program causes a computer to function as described above.

7. The terminal is operated by the user, the first state information is physical state information of the user; The second state information is psychological state information of the user, The reward value is the user's behavioral state information, Behavioral information is intervention information that influences user behavior change.

7. The program according to claim 6, which causes a computer to function as follows.

8. The physical state information is the distance traveled measured by the device's sensor, The psychological state information is a questionnaire result answered to questions displayed on the terminal display, The behavioral status information is the number of steps measured by the device sensor.

8. The program according to claim 7, which causes a computer to function as follows.

9. The physical condition information is first biological information measured by a sensor of the terminal, The psychological state information is a questionnaire result answered to questions displayed on the terminal display, The behavioral state information is second biological information measured by a sensor of the terminal.

8. The program according to claim 7, which causes a computer to function as follows.

10. In a server that serves as an agent entity and communicates with a terminal that serves as an environment entity in reinforcement learning, Upon receiving a first data set from the terminal, the first data set includes first and second state information and a reward value; a reinforcement learning engine that trains a reinforcement learning model using the first state information, the second state information, and the reward value included in the first data set, estimates action information that maximizes the reward value, and returns the action information to the terminal; a supervised learning engine that trains a supervised learning model using the first state information and the reward value included in the first dataset as explanatory variables and the second state information as a target variable; and upon receiving a second data set from the terminal, the second data set including the first state information and the reward value; the supervised learning engine receives the first state information and the reward value included in the second dataset and estimates the second state information; The reinforcement learning engine trains a reinforcement learning model using the first state information and reward value included in the second data set and the second state information estimated by the supervised learning engine, estimates action information that maximizes the reward value, and returns the action information to the terminal. A server characterized by:

11. In a reinforcement learning method for a server that serves as an agent entity and communicates with a terminal that serves as an environmental entity, The server Upon receiving a first data set from the terminal, the first data set includes first and second state information and a reward value; Using a reinforcement learning engine, train a reinforcement learning model using the first state information, the second state information, and the reward value included in the first dataset, estimate action information that maximizes the reward value, and return the action information to the environmental entity; Using a supervised learning engine, train a supervised learning model using the first state information and the reward value included in the first dataset as explanatory variables and the second state information as a response variable; upon receiving a second data set from the terminal, the second data set including the first state information and the reward value; using a supervised learning engine to input the first state information and the reward value included in the second dataset and estimate the second state information; Using a reinforcement learning engine, a reinforcement learning model is trained using the first state information and reward value included in the second data set and the second state information estimated by the supervised learning engine, and action information that maximizes the reward value is estimated and returned to the terminal. A reinforcement learning method for a server, characterized in that

Images