Data analysis device, data analysis method, and program

Abstract

This data analysis device is provided with: a learning means that performs reinforcement learning in which a policy function is updated using an evaluation function that includes a value function, and the value function is updated using an evaluation function that includes the policy function; and an evaluation means that evaluates the degree of influence imposed by evaluation data on learning results using an evaluation of learning results obtained when the reinforcement learning is performed using a data set that includes the evaluation data, and an evaluation of learning results obtained when the reinforcement learning is performed using a data set that does not include the evaluation data.

Description

Technical Field

[0001] The present invention relates to a data analysis device, a display device, a data analysis method, and a recording medium.

Background Art

[0002] One of machine learning is reinforcement learning (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is preferable to be able to evaluate the degree of influence of each data used in reinforcement learning on the learning result.

[0005] An example of the object of this disclosure is to provide a data analysis device, a data analysis method, and a recording medium capable of solving the above-described problems.

Means for Solving the Problems

[0006] According to a first aspect of the present invention, a data analysis device includes a learning means for performing reinforcement learning to update a policy function using an evaluation function including a value function and to update the value function using an evaluation function including the policy function, an evaluation for a learning result when performing the reinforcement learning using a data set including the data to be evaluated, and an evaluation for a learning result when performing the reinforcement learning using a data set not including the data to be evaluated, and an evaluation means for evaluating the degree of influence of the data to be evaluated on the learning result.

[0007] According to a second aspect of the present invention, the display device includes a display means for displaying the degree to which the data included in the training dataset, which is a dataset used for reinforcement learning, has an influence on the results of the reinforcement learning.

[0008] According to a third aspect of the present invention, the data analysis method includes a computer updating a policy function using an evaluation function that includes a value function, performing reinforcement learning that updates the value function using an evaluation function that includes a policy function, and evaluating the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when the reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated.

[0009] According to a fourth aspect of the present invention, the program is a program that causes a computer to perform reinforcement learning, which updates a policy function using an evaluation function that includes a value function, and updates the value function using an evaluation function that includes a policy function; and evaluate the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when the reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated. [Effects of the Invention]

[0010] According to the present invention, it is possible to evaluate the degree to which individual data used in reinforcement learning have an influence on the learning results. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of a data analysis device according to the embodiment. [Figure 2] This figure shows an example of how the display unit according to the embodiment displays an evaluation of the results of reinforcement learning. [Figure 3]This figure shows an example of a procedure for evaluating the degree to which data has an influence on the results of reinforcement learning, using a data analysis device according to an embodiment. [Figure 4] This figure shows another example of the configuration of the data analysis device according to the embodiment. [Figure 5] This figure shows an example of the configuration of a display device according to the embodiment. [Figure 6] This figure shows an example of the configuration of the processing apparatus according to the embodiment. [Figure 7] This figure shows an example of the processing steps in the data analysis method according to the embodiment. [Figure 8] This is a schematic block diagram showing the configuration of a computer according to at least one embodiment. [Modes for carrying out the invention]

[0012] The following describes embodiments of the present invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0013] In the following, characters with a macron or overline will sometimes be represented by adding a ￣ after the character. For example, the character "φ" with a macron will also be written as "φ￣". Sometimes, a string with a macron or overline is represented by enclosing the string in parentheses and then adding a ￣ symbol after it. For example, "M j The string with an overline around it is "(M j It can also be written as )￣」

[0014] Figure 1 is a diagram showing an example of the configuration of a data analysis device according to an embodiment. In the configuration shown in Figure 1, the data analysis device 100 comprises a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a control unit 190. The control unit 190 comprises a learning unit 191, an evaluation unit 192, and a processing execution unit 193.

[0015] The data analysis device 100 evaluates the extent to which each individual data point included in the dataset used for reinforcement learning influences the reinforcement learning process. Specifically, the data analysis device 100 evaluates the extent to which the data to be evaluated influences the learning results by comparing the evaluation of the learning results when using a dataset that includes the data to be evaluated with the evaluation of the learning results when using a dataset that does not include the data to be evaluated. The dataset used in reinforcement learning is also called the training dataset. The dataset used to evaluate the learning results is also called the test dataset.

[0016] Reinforcement learning, in this context, involves defining a policy (a set of rules for an agent's actions in a given environment) and a state (a state in the environment) and a reward (a reward representing the evaluation of that state or action). a This is a machine learning method that learns based on (rd).

[0017] The "agent" in reinforcement learning targeted by the data analysis device 100 is not limited to a specific entity. For example, the "agent" may be a robot such as an autonomous mobile robot or an industrial robot, or a mobile device such as an automated guided vehicle or a drone. Alternatively, the "agent" may be a plant such as a factory or power plant, a part thereof, or a standalone device. The "agent" is also referred to as the "controlled object."

[0018] The evaluation results from the data analysis device 100 can be used, for example, to edit the training dataset, or to analyze and adjust the "agent" or "environment". For example, consider a case where the "agent" is a robot, and the robot's performance as a result of reinforcement learning differs depending on when the data included in the training dataset was obtained. Factors that could cause the results of reinforcement learning to differ depending on when the data was obtained include differences in the environment and differences in the data collection methods.

[0019] Furthermore, the data analysis device 100 evaluates that the robot's performance based on reinforcement learning is relatively better when the influence of new data is relatively stronger than when the influence of old data is relatively stronger. In this case, the data analysis device 100 may delete all or some of the old data from the training dataset. The data analysis device 100 may then perform reinforcement learning on the robot again using the training dataset from which the old data has been removed.

[0020] This makes it easier for new data to be reflected in reinforcement learning, and is expected to improve the robot's performance based on the results of reinforcement learning. Furthermore, by deleting old data from the training dataset, the data size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. Also, because the data size of the training dataset is reduced, it is expected that when the data analysis device 100 performs reinforcement learning on the robot using the reduced training dataset, the learning results will converge in a relatively short time, thus shortening the time required for learning.

[0021] Alternatively, when the data analysis device 100 performs reinforcement learning on the robot again, if it can select the same data from the training dataset multiple times, it may adjust the frequency with which each data is selected from the training dataset so that the frequency of selecting new data is relatively high.

[0022] This makes it easier for new data to be reflected in reinforcement learning, and is expected to improve the robot's performance based on the results of reinforcement learning. Furthermore, it is expected that the learning results will converge relatively quickly, resulting in a relatively shorter learning time.

[0023] Furthermore, the data analysis device 100 may display an evaluation of the robot's performance based on the reinforcement learning results, linked to data that was evaluated as having a strong influence on the learning results. For example, a user may refer to data linked to an evaluation of good robot performance and adjust the robot or its operating environment.

[0024] Furthermore, for example, if the data indicates that the robot performs best when the humidity in the room where it operates is within the range of 40% to 60%, the user may adjust the humidity in the room to be within that range using a humidifier or dehumidifier.

[0025] Alternatively, the data analysis device 100 may be configured to automatically adjust the robot or the robot's operating environment. For example, the data analysis device 100 may pre-store rules indicating the relationship between humidity levels shown in data associated with a good robot performance evaluation and the settings for the humidifier and dehumidifier, and configure the humidifier and dehumidifier settings according to these rules.

[0026] The communication unit 110 communicates with other devices. For example, the communication unit 110 may receive training datasets from a server device that stores training datasets. Alternatively, if the data analysis device 100 acquires training datasets using the actual device under control or a simulator outside the data analysis device 100, it may send commands to the actual device or simulator instructing it to "act". The data analysis device 100 may also receive data from the sensor device or simulator indicating the results of observations of the "environment".

[0027] The display unit 120 includes a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 may display the evaluation results of the impact of data on the learning results. The display unit 120 is an example of a display means.

[0028] The operation input unit 130 includes, for example, input devices such as a keyboard and a mouse, and accepts user input. For example, the operation input unit 130 may be configured to accept user input instructing the system to delete some data from the training dataset.

[0029] The memory unit 180 stores various types of data. For example, the memory unit 180 may store training datasets, test datasets, and various parameters that are updated during reinforcement learning. The memory unit 180 is configured using the memory devices provided by the data analysis device 100.

[0030] The control unit 190 controls various parts of the data analysis device 100 to perform various processes. The functions of the control unit 190 may also be performed by the CPU (Central Processing Unit) of the data analysis device 100 reading a program from the storage unit 180 and executing it.

[0031] The learning unit 191 performs reinforcement learning using the training dataset. In the following explanation, we will describe an example where the policy function and value function are constructed using neural networks, and the learning unit 191 performs reinforcement learning using a mask that specifies a subset network from which a part of the neural network has been excluded.

[0032] The value function referred to here is a function that outputs a predicted value of the reward. More specifically, the value function is a function that outputs a predicted value of the cumulative sum of discounted rewards that an agent would receive if it took a certain action under certain conditions. The following explanation uses the Q function as the value function as an example. However, the value function used by the data analysis device 100 is not limited to a specific one. Masks used in reinforcement learning are also called training masks.

[0033] In this context, a function being constructed using a neural network means that the neural network is used to calculate the value of that function. A subset of a neural network, which excludes a portion of the original neural network, may be a neural network composed of subsets obtained by excluding one or more intermediate nodes from the original neural network.

[0034] The learning unit 191 generates a unique learning mask for each data point included in the training dataset, for both the policy function and the Q function. Regarding the policy function, the process of (temporarily) constructing a subset network specified by a mask is also referred to as applying the mask to the policy function. Similarly, regarding the Q function, the process of (temporarily) constructing a subset network specified by a mask is also referred to as applying the mask to the Q function.

[0035] When multiple Q-functions are used in reinforcement learning, the learning unit 191 may generate a unique learning mask for each of the multiple Q-functions, based on the data included in the training dataset. Alternatively, when multiple Q-functions are constructed using neural networks with the same structure, the learning unit 191 may generate a common learning mask for these multiple Q-functions, uniquely based on the data included in the training dataset.

[0036] The learning unit 191 applies the learning mask associated with the sampled data to the neural network each time it samples data from the training dataset, updating the parameter values ​​of the policy function and the Q function.

[0037] Here, the parameters of the policy function and the Q function are the parameters of the neural network that constitutes the policy function and the parameters of the neural network that constitutes the Q function. The masked portion of the neural network parameters are excluded from updating the parameter values ​​in reinforcement learning for the data associated with the mask.

[0038] In reinforcement learning using masks, the learning unit 191 can be understood as performing reinforcement learning on the masked portion of the neural network using a dataset that does not contain data associated with the training mask. Alternatively, the learning unit 191 can be understood as performing reinforcement learning on the unmasked portion of the neural network using a dataset that contains data associated with the training mask.

[0039] In this way, the learning unit 191 performs reinforcement learning using a learning mask, enabling it to perform reinforcement learning on each individual data point in the training dataset, both using the dataset containing that data and using the dataset without that data, in a single reinforcement learning run. In this respect, the learning unit 191 can efficiently perform reinforcement learning to evaluate the influence of individual data points on reinforcement learning.

[0040] However, the method by which the learning unit 191 performs reinforcement learning based on a dataset containing certain data and reinforcement learning based on a dataset that does not contain that data is not limited to the method using a learning mask. For example, the learning unit 191 may perform reinforcement learning using all the data included in the training dataset, and also perform reinforcement learning using all the data other than the data included in the training dataset for each piece of data.

[0041] Furthermore, the machine learning models that constitute the policy function and the Q-function are not limited to neural networks. Various models capable of calculating function values ​​using submodels can be used as the machine learning models that constitute the policy function and the Q-function.

[0042] The learning unit 191 alternately updates the parameter values ​​of the policy function and the parameter values ​​of the Q function, using a policy function that includes the Q function and a Q function that includes the policy function. The following explanation will use the case where the learning unit 191 updates the values ​​of the Q function parameters φ1 and φ2 based on equation (1) as an example.

[0043]

number

[0044] In equation (1), i = 1 and 2, and two values ​​for the parameters of the Q function are calculated, such as parameters φ1 and φ2. "argumin" is a function that outputs the value of the variable shown below "argumin" such that the value of the function shown after "argumin" is minimized. Equation (1) shows the parameter φ such that the value of the function shown after "argumin" is minimized. i This indicates that we are looking for the value of [this].

[0045] E represents the expected value. "(s,a,r,s',j)~D" indicates that the quartet data (s,a,r,s') consisting of state s, action a, reward r, and next state s' follows a distribution in the Replay Buffer D. "j" represents the identification number of the quartet data (s,a,r,s'). A set of four data points (s, a, r, s') is also simply referred to as data.

[0046] The replay buffer D is a buffer for recording data used in reinforcement learning. The set of data recorded in the replay buffer D at the end of reinforcement learning corresponds to an example of a training dataset. γ is a constant coefficient indicating the discount rate. α is a constant coefficient indicating the step size.

[0047] "min" is a function that outputs the minimum value of the expression shown after "min" under the conditions shown under "min". "Q" represents the Q-function, and the subscript description of "Q" indicates the conditions applied to the Q-function. "Q φ￣i,Mj (s’,a’)」 represents calculating the value of the Q-function when the action a’ is taken under the next state s’ as a target parameter φ￣ of the Q-function i and applying the learning mask M to the Q-function j and applying it.

[0048] j represents the identification number of the data. The identification number of the data can also be used as the identification number of the mask. As described above, the Q-function corresponds to an example of a value function. The value of the Q-function is also referred to as the Q-value.

[0049] The target parameter φ￣ i is a parameter used to avoid sudden changes in the Q-value. The value of the target parameter φ￣ i is updated to a weighted average value between its own value and the value of the parameter φ of the Q-function i . The update of the value of the target parameter φ￣ i can be expressed as in Equation (2). i The update of the value of the target parameter φ￣

[0050]

Equation

[0051] ρ is the target parameter φ￣ iThis is a constant coefficient representing the weight for the value of .

[0052] "π" represents the policy function, and the subscripts following "π" indicate the conditions that apply to the policy function. θ,Mj (a'|s')」'θ,M j This involves applying the parameter θ as a parameter to the policy function π, and applying the learning mask M to the policy function π. j This indicates that it will be applied.

[0053] Here, we will explain using the example where the policy function π is constructed as a conditional probability density function that probabilistically selects an action depending on the state. "(a'|s')" represents that the policy function π selects action a' under state s'. a' is a learning mask M j The policy function π to which this is applied represents the action decided for the next state s'. The action a' can be expressed as shown in equation (3).

[0054]

number

[0055] "π θ,Mj (a'|s') represents the probability that the policy function π selects action a' under the next state s'. Equation (1) is calculated by minimizing the magnitude of the difference obtained by subtracting the estimated total future reward (calculated as the Q-value) from the sum of the estimated total future reward (calculated as the Q-value) obtained by subtracting the estimated total future reward (calculated as the Q-value) from the sum of the reward for taking action a in state s and the estimated total future reward (calculated for taking action a' in the next state s'). i This indicates that the value of [the variable] is determined.

[0056] Furthermore, the following explanation will use the case where the learning unit 191 updates the value of the policy function parameter θ based on equation (4) as an example.

[0057]

number

[0058] The "log" term in equation (4) represents the policy entropy. Equation (4) indicates that the value of the parameter θ is determined such that the estimated sum of future rewards calculated as the Q value is as large as possible, and the randomness of the behavior, as indicated by the value of the policy entropy term, is as large as possible.

[0059] However, the policy function and value function used by the data analysis device 100 are not limited to specific ones. For example, the policy function used by the data analysis device 100 is not limited to one that probabilistically selects actions, but may also be one that determins actions depending on the state, etc.

[0060] Furthermore, the reinforcement learning method used by the data analysis device 100 can be any of the various methods that alternately update the parameter values ​​of the policy function and the parameter values ​​of the value function. Furthermore, the ratio of masked to unmasked areas in a learning mask is not limited to a specific ratio.

[0061] The evaluation unit 192 evaluates the degree to which individual data points influence learning. Specifically, the evaluation unit 192 evaluates the degree to which the data to be evaluated influences learning by using the evaluation of the learning results when the learning unit 191 performs reinforcement learning using a dataset that includes the data to be evaluated, and the evaluation of the learning results when the learning unit 191 performs reinforcement learning using a dataset that does not include the data to be evaluated.

[0062] The following explanation uses an example where the evaluation unit 192 evaluates the degree of influence of individual data on learning using a mask obtained by inverting the training mask. The mask obtained by inverting the training mask is also called the inverted mask. Training Mask M j The inverted mask of (M j It can also be written as )￣.

[0063] In this context, mask inversion means reversing which parts of the neural network are included in a subset and which are excluded. More specifically, among the intermediate nodes of the neural network, those included in the subset in the training mask are excluded from the subset in the inverted mask. Among the intermediate nodes of the neural network, those excluded from the subset in the training mask are included in the subset in the inverted mask.

[0064] The evaluation unit 192 evaluates the data d of the degree of impact. j Regarding data d j Learning mask M linked to j Inverted mask for (M j )￣ is generated. The evaluation unit 192 inverts the mask (M) for each learning mask. j )￣ is generated. Therefore, the evaluation unit 192 generates an inversion mask (M) for each of the policy function and the Q function. j )￣ will be generated.

[0065] In reinforcement learning, multiple Q-functions are used, and the learning unit 191 applies a learning mask M to each of the multiple Q-functions. j If a Q function is being generated, the evaluation unit 192 generates an inversion mask (M j )￣ is generated. Alternatively, the learning unit 191 generates a learning mask M common to multiple Q functions. j If a common inversion mask (M) is being generated, the evaluation unit 192 generates a common inversion mask (M) for multiple Q functions. j )￣ will be generated.

[0066] Data to be evaluated d j If there are multiple, the evaluation unit 192 evaluates the data d j Invert mask for each (M j )￣ is generated. For example, all the data d contained in replay buffer D j If the item is to be evaluated, the evaluation unit 192 evaluates all the data d contained in the replay buffer D. jEach inverted mask (M j )￣ will be generated.

[0067] The evaluation unit 192 then calculates a performance evaluation index for the agent using the data included in the test dataset, without applying a mask to either the policy function or the Q function. The evaluation unit 192 uses the performance evaluation index obtained in this case as the evaluation value of the reinforcement learning result using all the data included in the training dataset. As an evaluation metric for the agent's performance, an index that correlates with the expected value of the reward obtained under the test dataset may be used.

[0068] Furthermore, the evaluation unit 192 evaluates one data point d j Inverted mask according to (M j The )￣ is applied to the policy function and the Q function, respectively, to calculate the agent's performance evaluation index using the data included in the test dataset. The evaluation unit 192 uses the performance evaluation index obtained in this case as an evaluation value of the reinforcement learning result using the data included in the training dataset, excluding the data associated with the inversion mask.

[0069] The evaluation unit 192 evaluates the degree of influence of the data associated with the inversion mask on the reinforcement learning results by comparing the evaluation value of the reinforcement learning results using all the data included in the training dataset with the evaluation value of the reinforcement learning results using the data included in the training dataset excluding the data associated with the inversion mask.

[0070] For example, the evaluation unit 192 calculates the difference between the evaluation value of the reinforcement learning result using all the data included in the training dataset and the evaluation value of the reinforcement learning result using all the data included in the training dataset, and uses this difference as an evaluation value of the degree of influence of the data included in the inverted mask on the reinforcement learning result.

[0071] The evaluation unit 192 calculates an evaluation value for each data set being evaluated, indicating the degree of influence of that data on the reinforcement learning results. The evaluation unit 192 may use equation (5) to calculate an evaluation value of the degree to which the data influences the reinforcement learning results.

[0072]

number

[0073] I is data d j This represents an evaluation index of the degree to which it influences the outcome of reinforcement learning. L represents a performance evaluation metric. Here, the performance evaluation metric is an indicator that shows the performance of the agent resulting from reinforcement learning. In equation (5), the Q function and the policy function are given an inversion mask (M j The difference obtained by subtracting the performance evaluation metrics when the mask is not applied to the Q function and policy function from the performance evaluation metrics when the mask is applied is calculated using data d. j This is used as an evaluation index to assess the degree to which it influences the results of reinforcement learning.

[0074] Inverted mask (M j When )￣ is applied, the Q function and policy function are given by the data d j It can be considered that the learning effect is not present when using . From this, in equation (5), data d j When using the training dataset excluding data d j The change in performance evaluation metrics compared to when using a training dataset that includes data d j This can be interpreted as evaluating the degree of influence it exerts.

[0075] The evaluation unit 192 may use the index shown in equation (6) as the performance evaluation index L.

[0076]

number

[0077] Q π s represents the discounted cumulative reward sum that would actually be obtained in the future if the policy function π were followed under a given state and action. s is the test dataset D test This represents the state sampled from the test dataset D. a￣ is the test dataset D test This represents the behavior sampled from the data.

[0078] The performance evaluation index L shown in equation (6) indicates the degree of error, specifically how much the value of the Q function deviates from the correct value. A value of L closer to 0 indicates better performance of the Q function, while a larger absolute value of L indicates poorer performance of the Q function.

[0079] When the learning unit 191 performs reinforcement learning using a dataset that does not include the data to be evaluated, using all data in the training dataset other than the data to be evaluated, the evaluation unit 192 calculates the "Q" in equation (6). φi,(Mj)￣ The inverted mask (M) is indicated by (s,a). j Alternatively, instead of using the Q-value obtained with ), the performance evaluation index L may be calculated using the Q-value based on the results of reinforcement learning using all data in the training dataset except for the data being evaluated. However, the method for calculating the performance evaluation index L in equation (5) is not limited to a specific method.

[0080] Also, test dataset D test The program may be structured in the form of episodes. The evaluation unit 192 may use the total rewards obtained in one episode as the true value of that episode to calculate the performance evaluation index L.

[0081] Furthermore, the mask used by the evaluation unit 192 is not limited to an inverted mask. The evaluation unit 192 may invert the parts not masked by the learning mask so that they are masked, and use an inverted mask so that only a part of the parts masked by the learning mask is not masked. In other words, a part of the parts masked by the learning mask may also be masked by the mask used by the evaluation unit 192. The mask used by the evaluation unit 192 is also called an evaluation mask.

[0082] Test dataset D test This is not limited to a specific dataset, but can be any various datasets on which the agent can run. Test dataset D test The test dataset may be the same as the training dataset, or it may be a different dataset from the training dataset. The test dataset and the training dataset do not need to contain common data.

[0083] The processing execution unit 193 performs processing based on the evaluation results from the evaluation unit 192. The processing execution unit 193 is an example of a processing execution means. The data analysis device 100 is an example of a processing device. The processing execution unit 193 may, based on the evaluation results from the evaluation unit 192, delete data from the training dataset that it determines has a stronger impact than a predetermined standard when the agent's performance is worse than a predetermined standard.

[0084] For example, as described above with respect to the data analysis device 100, if it determines that the robot's performance based on the reinforcement learning results is relatively better when the influence of new data is relatively stronger than when the influence of old data is relatively stronger, the processing execution unit 193 may delete all or some of the old data from the training dataset. The processing execution unit 193 may be configured to delete data automatically, or it may be configured to delete data according to user instructions.

[0085] The processing execution unit 193 may adjust the frequency with which each data point included in the training dataset is selected when the learning unit 191 performs reinforcement learning again, based on the evaluation results from the evaluation unit 192. For example, as described above with respect to the data analysis device 100, the processing execution unit 193 may adjust the frequency with which each data item in the training dataset is selected so that the learning unit 191 selects new data relatively frequently when performing reinforcement learning on the robot again.

[0086] The processing execution unit 193 may adjust the controlled object or the operating environment of the controlled object based on the evaluation results from the evaluation unit 192. Here, the controlled object corresponds to an agent in reinforcement learning. The operating environment of the controlled object corresponds to the environment in reinforcement learning.

[0087] For example, as described above with respect to the data analysis device 100, the memory unit 180 may pre-store rules indicating the relationship between humidity levels shown in data associated with a good robot performance evaluation and the settings for the humidifier and dehumidifier. The processing execution unit 193 may then set the humidifier and dehumidifier according to these rules so that the humidity level is such that the evaluation unit 192 determines that the robot's performance is good. The processing execution unit 193 may automatically adjust the controlled object or the operating environment of the controlled object, or it may do so according to user instructions.

[0088] Figure 2 shows an example of how the display unit 120 displays an evaluation of the reinforcement learning results. Figure 2 shows an example of a display screen showing the evaluation of the reinforcement learning results for five quadruped robots. The five robots are named REDQ09 seed0, REDQ09 seed1, REDQ09 seed2, REDQ09 seed3, REDQ09 seed4, and REDQ09 seed5.

[0089] The display unit 120 displays an evaluation screen for the reinforcement learning results in accordance with the control unit 190. For example, the display unit 120 may display an evaluation screen for the reinforcement learning results in accordance with the control unit 192 and / or the processing execution unit 193.

[0090] Area A11 is the region where graphs showing performance evaluations for each robot after a certain number of operations are performed are displayed. The horizontal axis of the graph in area A11 represents the number of interactions between the robot and the environment (i.e., the number of times an action was selected based on the policy), and is 10 5 The units are shown. In the example in Figure 2, reinforcement learning by the learning unit 191 shows that the interaction between the robot and the environment is 10 3 Each time the test is repeated, the evaluation unit 192 performs the operation test and records the evaluation value in the storage unit 180. The vertical axis shows the evaluation value of the robot's performance. The closer the average bias of the evaluation is to 0, the better the evaluation. In the example in Figure 2, the greater the degree of robot's forward movement per unit time, the better the evaluation, and the closer the average bias of the evaluation is to 0.

[0091] Line L110 shows the average evaluation for each iteration of the REDQ09 seed0 operation. Line L111 shows the average evaluation for each iteration of the REDQ09 seed1 operation. Line L112 shows the average evaluation for each iteration of the REDQ09 seed2 operation. Line L113 shows the average evaluation for each iteration of the REDQ09 seed3 operation. Line L114 shows the average evaluation for each iteration of the REDQ09 seed4 operation.

[0092] Area A12 is a region where graphs are displayed that show, using heatmaps, the degree to which the data used for reinforcement learning for each robot influenced the results of that robot's reinforcement learning. The display unit 120 displays graphs for each robot in windows from seed0 to seed4.

[0093] The horizontal axis of the graph in area A12 represents the number of times the robot's motion test was performed, 10 3 The values ​​are shown in units. The vertical axis shows the normalized values ​​of the data numbers, which are within the range of 0 to 1. For example, the evaluation unit 192 assigns a sequential identification number to each data point in the test dataset, in order of oldest data, and performs normalization. The value of the vertical axis, Normalized sample index, is closer to 0, indicating older data, and closer to 1, indicating newer data.

[0094] The heatmap exemplified in area A12 shows the degree of influence of each piece of data on the robot's motion test, indicated by the number of runs on the horizontal axis of the graph, and the influence of each piece of data, indicated by the number on the vertical axis of the graph. The stronger the influence of the data, the larger the Influence value and the lighter the hatching on the heatmap.

[0095] Area A13 is the area where a message from the data analysis device 100 to the user is displayed. In the graph in region A11, the average bias of REDQ09 seed4, indicated by line L114, is far from zero. The user, referring to the graph in region A11, determines that REDQ09 seed4 is performing poorly and selects the graph for REDQ09 seed4 from among the graphs of each robot shown in region A12. The display unit 120 brings the window for the selected REDQ09 seed4 graph to the foreground.

[0096] The display unit 120 then displays a message regarding the selected REDQ09 seed4 graph in area A13. In the example in Figure 2, the processing execution unit 193 determines, based on the evaluation by the evaluation unit 192, that the data corresponding to the area enclosed by the thick border in the REDQ09 seed4 graph gives a strong evaluation to the performance of REDQ09 seed4. The processing execution unit 193 then displays a message on the display unit 120 asking whether or not to delete the data corresponding to the area enclosed by the thick border.

[0097] Area A14 is the display area for buttons that accept user input. The display unit 120 displays a "Yes" button and a "No" button in area A14. The operation input unit 130 accepts user input, such as selecting either the "Yes" button or the "No" button by clicking the mouse or touching the screen.

[0098] If the operation input unit 130 detects a user operation in which the "Yes" button is selected, the processing execution unit 193 deletes the data corresponding to the area enclosed by the thick border in the EDQ09 seed4 graph from the training dataset. On the other hand, if the operation input unit 130 detects a user operation in which the "No" button is selected, the processing execution unit 193 does not delete the data. In this case, the display unit 120 may also remove the display of the message in area A13 and the button in area A14.

[0099] Figure 3 shows an example of the procedure by which the data analysis device 100 evaluates the degree to which the data influences the results of reinforcement learning.

[0100] (Step S1) The learning unit 191 performs the initial setup for reinforcement learning. Specifically, the learning unit 191 initializes the values ​​of the policy function parameter θ and the parameters φ1 and φ2 of the Q function, and prepares an empty replay buffer D. Furthermore, the learning unit 191 uses the target parameter φ￣ i The parameter φ of the Q function i Substitute the values ​​for i. Here, i = 1 and i = 2. After step S1, the process proceeds to step S2.

[0101] (Step S2) The learning unit 191 acquires data and adds it to the replay buffer D. Specifically, the learning unit 191 processes the policy function π θ and current state s tBased on this, action a t The decision was made, and the decided action a t The agent will execute this command. Here, time will be represented in terms of time steps, with the current time represented by "t" and the next time represented by "t+1".

[0102] One way to get an agent to perform an action is to simulate the agent performing that action, or to control the actual device that corresponds to the agent and have it perform that action. Policy function π θ and current state s t Based on this, action a t Determining this can be expressed as shown in equation (7).

[0103]

number

[0104] Then, the learning unit 191 receives the reward r t And then, the next state s t+1 We observe and the quadruple data (s t ,a t ,r t ,s t+1 ) is added to replay buffer D. A set of four data (s t ,a t ,r t ,s t+1 Adding ) to replay buffer D can be expressed as shown in equation (8).

[0105]

number

[0106] After step S2, the process proceeds to step S3.

[0107] (Step S3) The learning unit 191 samples minibatch B from replay buffer D. Here, minibatch B is a set of quads of data. Minibatch B can be expressed as shown in equation (9).

[0108]

number

[0109] d j This represents a set of four data points. j The "j" in this case represents an identification number that identifies the data in a set of four. As mentioned above, s represents a state, a represents an action, r represents a reward, and s' represents the next state after state s. The data d is a set of four. j Simply data d j It is also called by this name. Furthermore, the learning unit 191 processes data d j Each learning mask M j Generates. After step S3, the process proceeds to step S4.

[0110] (Step S4) The learning unit 191 calculates the Q target y. The Q target y is the target value of the Q function. The learning unit 191 may calculate the Q target y based on equation (10).

[0111]

number

[0112] As mentioned above, M j is data d j This represents the learning mask associated with [the device / item]. Q φ￣i,Mj The Q function has a target parameter φ i and learning mask M j This indicates that it will be applied.

[0113] π θ,Mj This involves the policy function having parameters θ and a learning mask M. jIt represents applying. The action a’ is the policy function π j applied with the learning mask M θ,Mj and determined based on the next state s’. After step S4, the process proceeds to step S5.

[0114] (Step S5) The learning unit 191 updates the values of the parameters φ1 and φ2 of the Q function. The learning unit 191 updates the values of the parameters φ1 and φ2 based on the above formula (1). The learning unit 191 may update the values of the parameters φ1 and φ2 using the Gradient Descent method based on the gradient shown in formula (11).

[0115] [Number]

[0116] |B| represents the size of the mini-batch B. Specifically, |B| represents the number of data d j contained in the mini-batch B. After step S5, the process proceeds to step S6.

[0117] (Step S6) The learning unit 191 updates the value of the target parameter φ￣ i The learning unit 191 updates the value of the target parameter φ￣ i based on the above formula (2). After step S6, the process proceeds to step S7.

[0118] (Step S7) The learning unit 191 updates the value of the parameter θ of the policy function. The learning unit 191 updates the value of the parameter θ based on the above formula (4). The learning unit 191 may update the value of the parameter θ using the Gradient Descent method based on the gradient shown in formula (12).

[0119]

number

[0120] After step S7, the process proceeds to step S8.

[0121] (Step S8) The learning unit 191 determines whether or not the termination condition for reinforcement learning has been met. The termination condition for reinforcement learning here is not limited to any specific condition. For example, the termination condition for reinforcement learning here may be that the loop from step S2 to S8 has been executed a predetermined number of times or more. Alternatively, the termination condition for reinforcement learning here may be that the expected value shown in equation (1) and / or the expected value shown in equation (4), or either of these, is less than a predetermined value.

[0122] If the learning unit 191 determines that the termination condition for reinforcement learning has not been met (step S8: NO), the process returns to step S2. On the other hand, if the learning unit 191 determines that the termination condition for reinforcement learning is met (step S8: YES), the process proceeds to step S11.

[0123] (Step S11) The evaluation unit 192 acquires the test dataset. After step S11, the process proceeds to step S12.

[0124] (Step S12) The evaluation unit 192 evaluates the degree to which the data influences the results of reinforcement learning. The evaluation unit 192 evaluates the data d based on the above formula (5). j It may also be possible to calculate an evaluation value for the degree of influence. After step S12, the process proceeds to step S13.

[0125] (Step S13) The evaluation unit 192 determines whether the termination condition for evaluating the degree of influence of the data on the reinforcement learning results has been met. For example, the termination condition here could be that the evaluation values ​​for the degree of influence of all the data being evaluated have been calculated.

[0126] If the evaluation unit 192 determines that the termination condition for evaluating the degree to which the data influences the reinforcement learning results is not met (step S13: NO), the process returns to step S12. On the other hand, if the evaluation unit 192 determines that the termination condition for evaluating the degree of influence of the data on the reinforcement learning results is met (step S13: YES), the data analysis device 100 terminates the process shown in Figure 3.

[0127] As described above, the learning unit 191 updates the policy function using an evaluation function that includes the value function, and performs reinforcement learning that updates the value function using an evaluation function that includes the policy function. The evaluation unit 192 evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when reinforcement learning is performed using a dataset that does not include the data to be evaluated.

[0128] According to the data analysis device 100, it is possible to evaluate the degree to which each piece of data used in reinforcement learning has an influence on the learning results.

[0129] Furthermore, the learning unit 191 acquires one data point from the data included in the training dataset and selects a learning mask corresponding to the acquired data. Then, the learning unit 191 uses the selected learning mask as the policy. function With the machine learning models constituting the policy function and the machine learning models constituting the value function applied, the parameter values ​​of the submodels of the machine learning models constituting the policy function and the parameter values ​​of the submodels of the machine learning models constituting the value function are updated using the acquired data.

[0130] The training dataset is the dataset used for reinforcement learning. The training mask is a mask that uniquely specifies, for each data point in the training dataset, a submodel that excludes parts of the machine learning model that constitutes the policy function, and a submodel that excludes parts of the machine learning model that constitutes the value function.

[0131] The evaluation unit 192 uses an evaluation mask to determine the policy function The evaluation value obtained by applying the machine learning model that constitutes the value function and the machine learning model that constitutes the value function to the reinforcement learning is used as the evaluation value of the learning result when the reinforcement learning is performed using a dataset that does not include the data to be evaluated, thereby evaluating the degree to which the data to be evaluated has an influence on the learning result.

[0132] An evaluation mask is a mask that excludes portions of a single data point selected from the training dataset that are to be evaluated, while simultaneously excluding all or part of those portions that were excluded by the training mask.

[0133] Now, let's consider the case where we remove each data point from the training dataset one by one, perform reinforcement learning, and evaluate the degree of influence of that data. In this case, it is necessary to initialize the policy function and Q-function models for each data point to be evaluated and perform reinforcement learning for each data point.

[0134] In contrast, the data analysis device 100 uses a mask to construct a unique submodel for each data point being evaluated, allowing it to continue learning on each data point without needing to initialize the model. According to the data analysis device 100, this enables efficient reinforcement learning to evaluate the impact of individual data points on reinforcement learning.

[0135] Furthermore, according to the data analysis device 100, by performing reinforcement learning using a learning mask, it is possible to perform reinforcement learning using the dataset containing each individual data point in the training dataset, and reinforcement learning using the dataset without that data point, all in a single reinforcement learning session. In this respect as well, the data analysis device 100 enables efficient reinforcement learning to evaluate the influence of individual data points on reinforcement learning.

[0136] Furthermore, the display unit 120 displays the degree to which the data included in the training dataset influences the results of reinforcement learning. The user can refer to the display on the display unit 120 to perform actions such as deleting data from the training dataset, adjusting the frequency of data selection during retraining, adjusting the controlled object or the operating environment of the controlled object, or a combination thereof.

[0137] For example, a user can view a display showing the degree to which the data in the training dataset influences the reinforcement learning results, as well as an evaluation of the reinforcement learning results. If the evaluation of the reinforcement learning results is poor, the user can remove data from the training dataset that has a strong influence on the reinforcement learning results.

[0138] As a result, when the learning unit 191 performs reinforcement learning again using the updated training dataset, the new data will be more easily reflected in the reinforcement learning, and it is expected that the results of the reinforcement learning will improve. Furthermore, by deleting some data from the training dataset, the size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. Also, because the size of the training dataset is reduced, it is expected that when the learning unit 191 performs reinforcement learning using the reduced training dataset, the learning results will converge in a relatively short time, and the time required for learning will be relatively short.

[0139] Furthermore, if the evaluation of the reinforcement learning results is poor, the user can change the settings for data selection during reinforcement learning so that when the learning unit 191 performs reinforcement learning again, the frequency with which data from the training dataset that have a strong influence on the reinforcement learning results is reduced.

[0140] This makes it less likely that data that is thought to have a negative impact on the reinforcement learning results will be reflected in the reinforcement learning process, and is expected to improve the performance of the controlled object based on the reinforcement learning results. Furthermore, it is expected that the learning results will converge relatively quickly, resulting in a relatively shorter learning time.

[0141] Furthermore, users can adjust the settings of the controlled object or the operating environment of the controlled object to reproduce the "state" when the reinforcement learning results are good, by referring to data that has a strong influence on the reinforcement learning results when the reinforcement learning results are good.

[0142] The display unit 120 also displays the degree to which the data included in the training dataset influences the reinforcement learning results, as well as an evaluation of the reinforcement learning results. The user can refer to the display on the display unit 120 to perform actions such as deleting data from the training dataset, adjusting the frequency of data selection during retraining, adjusting the controlled object or the operating environment of the controlled object, or a combination thereof.

[0143] For example, a user can view a display showing the degree to which the data in the training dataset influences the reinforcement learning results, as well as an evaluation of the reinforcement learning results. If the evaluation of the reinforcement learning results is poor, the user can remove data from the training dataset that has a strong influence on the reinforcement learning results.

[0144] As a result, when the learning unit 191 performs reinforcement learning again using the updated training dataset, the new data will be more easily reflected in the reinforcement learning, and it is expected that the results of the reinforcement learning will improve. Furthermore, by deleting some data from the training dataset, the size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. Also, because the size of the training dataset is reduced, it is expected that when the learning unit 191 performs reinforcement learning using the reduced training dataset, the learning results will converge in a relatively short time, and the time required for learning will be relatively short.

[0145] Furthermore, if the evaluation of the reinforcement learning results is poor, the user can change the settings for data selection during reinforcement learning so that when the learning unit 191 performs reinforcement learning again, the frequency with which data from the training dataset that have a strong influence on the reinforcement learning results is reduced.

[0146] This makes it less likely that data that is thought to have a negative impact on the reinforcement learning results will be reflected in the reinforcement learning process, and is expected to improve the performance of the controlled object based on the reinforcement learning results. Furthermore, it is expected that the learning results will converge relatively quickly, resulting in a relatively shorter learning time.

[0147] Furthermore, users can adjust the settings of the controlled object or the operating environment of the controlled object to reproduce the "state" when the reinforcement learning results are good, by referring to data that has a strong influence on the reinforcement learning results when the reinforcement learning results are good.

[0148] The display unit 120 also displays the degree to which the data included in the training dataset influences the reinforcement learning results, an evaluation of the reinforcement learning results, and a message regarding the deletion of some of the data included in the training dataset. The user can refer to the display on the display unit 120 to decide whether or not to delete some of the data included in the training dataset.

[0149] Furthermore, the display unit 120 displays an evaluation of the reinforcement learning results for each of the multiple controlled objects, and displays the degree to which the data included in the training dataset influences the reinforcement learning results for each controlled object, in a selectable format. The display unit 120 then displays a query asking whether or not to delete some of the data included in the training dataset, selected based on the degree to which it influences the reinforcement learning results for the selected controlled object.

[0150] The user can refer to evaluations of the reinforcement learning results for each of several control targets, select one of those control targets, and then refer to the degree to which the data included in the training dataset influences the reinforcement learning results for the selected control target. Based on the degree to which the data included in the training dataset influences the reinforcement learning results, the user can then decide whether or not to delete some of the data indicated in the query displayed by the display unit.

[0151] Furthermore, the display unit 120 displays the degree of influence on the reinforcement learning results for each data point and for each evaluation iteration in the form of a heatmap, on a coordinate space composed of a first axis showing numerical values ​​that identify the data included in the training dataset and a second axis showing the number of iterations of evaluating the degree of influence of each data point included in the training dataset on the reinforcement learning results.

[0152] The user can refer to the display on the display unit 120 to perform actions such as deleting data from the training dataset, adjusting the frequency of data selection during retraining, adjusting the controlled object or the operating environment of the controlled object, or a combination thereof.

[0153] For example, a user can view a display showing the degree to which the data in the training dataset influences the reinforcement learning results, as well as an evaluation of the reinforcement learning results. If the evaluation of the reinforcement learning results is poor, the user can remove data from the training dataset that has a strong influence on the reinforcement learning results.

[0154] As a result, when the learning unit 191 performs reinforcement learning again using the updated training dataset, the new data will be more easily reflected in the reinforcement learning, and it is expected that the results of the reinforcement learning will improve. Furthermore, by deleting some data from the training dataset, the size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. Also, because the size of the training dataset is reduced, it is expected that when the learning unit 191 performs reinforcement learning using the reduced training dataset, the learning results will converge in a relatively short time, and the time required for learning will be relatively short.

[0155] Furthermore, if the evaluation of the reinforcement learning results is poor, the user can change the settings for data selection during reinforcement learning so that when the learning unit 191 performs reinforcement learning again, the frequency with which data from the training dataset that have a strong influence on the reinforcement learning results is reduced.

[0156] This makes it less likely that data that is thought to have a negative impact on the reinforcement learning results will be reflected in the reinforcement learning process, and is expected to improve the performance of the controlled object based on the reinforcement learning results. Furthermore, it is expected that the learning results will converge relatively quickly, resulting in a relatively shorter learning time.

[0157] Furthermore, the display unit 120 displays a heatmap showing the degree of influence on the reinforcement learning results for each data point and for each evaluation iteration, in a coordinate space composed of a first axis showing numerical values ​​that identify the data included in the training dataset and a second axis showing the number of iterations of evaluating the degree of influence of each data point in the training dataset on the reinforcement learning results. The heatmap includes a display of selected portions of the heatmap according to the degree of influence. The display unit 120 also displays a query asking whether or not to delete the selected portion of the data from the training dataset.

[0158] The user can refer to the display on the display unit 120 and decide and instruct whether or not to delete the selected portion of the data from the training dataset. For example, if data is selected that has a poor evaluation of the reinforcement learning results and has a strong influence on the reinforcement learning results, the user may choose to delete the selected portion of the data from the training dataset.

[0159] Furthermore, the display unit 120 displays the evaluation of the reinforcement learning results for each of the multiple control targets, for each iteration of evaluating the degree of influence of each data in the training dataset on the reinforcement learning results. The display unit 120 also displays a heatmap showing the degree of influence on the reinforcement learning results for each data and each evaluation iteration in a coordinate space formed by a first axis showing numerical values ​​that identify the data in the training dataset and a second axis showing the number of iterations of the evaluation of the degree of influence, and the heatmap is selectable for each control target, including the display of a portion of the heatmap selected according to the degree of influence. Furthermore, the display unit 120 displays a query asking whether or not to delete the data in the selected portion of the selected heatmap from the training dataset.

[0160] The user can refer to the display on the display unit 120 and decide and instruct whether or not to delete the selected portion of the data from the training dataset. For example, if data is selected that has a poor evaluation of the reinforcement learning results and has a strong influence on the reinforcement learning results, the user may choose to delete the selected portion of the data from the training dataset.

[0161] Furthermore, the learning unit 191 updates the policy function using an evaluation function that includes a value function, and performs reinforcement learning that updates the value function using an evaluation function that includes the policy function. The evaluation unit 192 evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when reinforcement learning is performed using a dataset that does not include the data to be evaluated. The processing execution unit 193 selects some of the data included in the training dataset and deletes the selected data from the training dataset based on the evaluation of the learning results and the evaluation of the degree of influence.

[0162] The data analysis device 100 can delete data that has a strong influence on the reinforcement learning results if the evaluation of the reinforcement learning results is poor. As a result, when the learning unit 191 performs reinforcement learning again using the updated training dataset, the new data will be more easily reflected in the reinforcement learning, and it is expected that the results of the reinforcement learning will improve.

[0163] Furthermore, by deleting some data from the training dataset, the size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. Also, because the size of the training dataset is reduced, it is expected that when the learning unit 191 performs reinforcement learning using the reduced training dataset, the learning results will converge in a relatively short time, and the time required for learning will be relatively short.

[0164] Furthermore, the learning unit 191 updates the policy function using an evaluation function that includes a value function, and performs reinforcement learning that updates the value function using an evaluation function that includes the policy function. The evaluation unit 192 evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when reinforcement learning is performed using a dataset that does not include the data to be evaluated. The processing execution unit 193 selects one or more data points from the data points included in the training dataset based on the evaluation of the learning results and the evaluation of the degree of influence. Then, the processing execution unit 193 makes adjustments to at least one of the controlled object that is the target of reinforcement learning, or the operating environment of that controlled object, based on the selected data.

[0165] This allows the data analysis device 100 to automatically make adjustments to at least one of the controlled object or the operating environment of the controlled object. For example, the data analysis device 100 can refer to data that has a strong influence on the reinforcement learning results when the reinforcement learning results are good, and adjust the settings of the controlled object or the operating environment of the controlled object to reproduce the "state" when the reinforcement learning results are good.

[0166] The data units used by the evaluation unit 192 to assess the impact of the data are not limited to quadruple data units. For example, the training dataset may be composed of episodes. The learning unit 191 may generate a learning mask for each episode and perform reinforcement learning, while the evaluation unit 192 may evaluate the degree to which each episode has an influence on the results of reinforcement learning. In this case, the processing unit 193 may manage the training dataset on an episode-by-episode basis, such as by deleting episodes from the training dataset.

[0167] Figure 4 shows another example of the configuration of the data analysis device according to the embodiment. In the configuration shown in Figure 4, the data analysis device 610 comprises a learning unit 611 and an evaluation unit 612. In this configuration, the learning unit 611 updates the policy function using an evaluation function that includes the value function, and updates the value function using an evaluation function that includes the policy function. The evaluation unit 612 evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when reinforcement learning is performed using a dataset that does not include the data to be evaluated. The learning unit 611 is an example of a learning means. The evaluation unit 612 is an example of an evaluation means.

[0168] According to the data analysis device 610, it is possible to evaluate the degree to which each piece of data used in reinforcement learning has an influence on the learning results.

[0169] The learning unit 611 can be implemented using functions such as the learning unit 191 shown in Figure 1. The evaluation unit 612 can be implemented using functions such as the evaluation unit 192 shown in Figure 1.

[0170] Figure 5 shows an example of the configuration of a display device according to the embodiment. In the configuration shown in Figure 5, the display device 620 includes a display unit 621. In this configuration, the display unit 621 includes a display unit that shows the degree to which the data included in the training dataset, which is a dataset used for reinforcement learning, has an influence on the results of the reinforcement learning. The display unit 621 is an example of a display means.

[0171] According to the display device 620, the user can refer to the display unit 621 to delete data from the training dataset, adjust the frequency of data selection during retraining, adjust the controlled object or the operating environment of the controlled object, or a combination of these.

[0172] For example, a user can view the degree to which the data in the training dataset influences the reinforcement learning results, and refer to an evaluation of the reinforcement learning results. If the evaluation of the reinforcement learning results is poor, the user can remove data from the training dataset that has a strong influence on the reinforcement learning results.

[0173] This means that when a reinforcement learning device performs reinforcement learning again using the updated training dataset, the new data will be more easily reflected in the reinforcement learning process, and it is expected that the results of the reinforcement learning will improve. Furthermore, by removing some data from the training dataset, the size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. In addition, because the size of the training dataset is reduced, it is expected that when a reinforcement learning device performs reinforcement learning using the reduced training dataset, the learning results will converge in a relatively short time, and the time required for learning will be relatively short.

[0174] Furthermore, if the evaluation of the reinforcement learning results is poor, the user can change the settings for data selection during reinforcement learning so that when the reinforcement learning device performs reinforcement learning again, the frequency with which data from the training dataset that have a strong influence on the reinforcement learning results is reduced.

[0175] This makes it less likely that data that is thought to have a negative impact on the reinforcement learning results will be reflected in the reinforcement learning process, and is expected to improve the performance of the controlled object based on the reinforcement learning results. Furthermore, it is expected that the learning results will converge relatively quickly, resulting in a relatively shorter learning time.

[0176] Furthermore, users can adjust the settings of the controlled object or the operating environment of the controlled object to reproduce the "state" when the reinforcement learning results are good, by referring to data that has a strong influence on the reinforcement learning results when the reinforcement learning results are good. The display unit 621 can be implemented using, for example, the functions of the display unit 120 shown in Figure 1.

[0177] Figure 6 shows an example of the configuration of a processing apparatus according to the embodiment. In the configuration shown in Figure 6, the processing apparatus 630 includes a learning unit 631, an evaluation unit 632, and a processing execution unit 633.

[0178] In this configuration, the learning unit 631 updates the policy function using an evaluation function that includes the value function, and performs reinforcement learning that updates the value function using an evaluation function that includes the policy function. The evaluation unit 632 evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when reinforcement learning is performed using a dataset that does not include the data to be evaluated.

[0179] The processing execution unit 633 selects some of the data included in the training dataset, which is the dataset used for reinforcement learning, based on an evaluation of the learning results and an evaluation of the degree of influence, and deletes the selected data from the training dataset. The learning unit 631 is an example of a learning means. The evaluation unit 632 is an example of an evaluation means. The processing execution unit 633 is an example of a processing execution means.

[0180] The processing unit 633 can delete data that has a strong influence on the reinforcement learning results if the evaluation of the reinforcement learning results is poor. As a result, when the learning unit 631 performs reinforcement learning again using the updated training dataset, the new data will be more easily reflected in the reinforcement learning, and it is expected that the results of the reinforcement learning will improve.

[0181] Furthermore, by deleting some data from the training dataset, the size of the training dataset is reduced, making it possible to relatively reduce the memory capacity required to store the training dataset. Also, because the size of the training dataset is reduced, it is expected that when the learning unit 631 performs reinforcement learning using the reduced training dataset, the learning results will converge in a relatively short time, and the time required for learning will be relatively short.

[0182] The learning unit 631 can be implemented, for example, using the functions of the learning unit 191 in Figure 1. The evaluation unit 632 can be implemented, for example, using the functions of the evaluation unit 192 in Figure 1. The processing execution unit 633 can be implemented, for example, using the functions of the processing execution unit 193 in Figure 1.

[0183] The processing execution unit 633 may select one or more data points from the training dataset, which is a dataset used for reinforcement learning, based on an evaluation of the learning results and an evaluation of the degree of influence, and make adjustments to at least one of the controlled object that is the target of reinforcement learning, or the operating environment of that controlled object, based on the selected data.

[0184] This allows the processing unit 630 to automatically make adjustments to at least one of the controlled object or the operating environment of the controlled object. For example, the processing unit 630 can refer to data that has a strong influence on the reinforcement learning results when the reinforcement learning results are good, and adjust the settings of the controlled object or the operating environment of the controlled object to reproduce the "state" when the reinforcement learning results are good.

[0185] Figure 7 shows an example of the processing steps in the data analysis method according to the embodiment. The data analysis method shown in Figure 7 includes performing learning (step S611) and performing evaluation (step S612). In the learning process (step S611), the computer performs reinforcement learning, updating the policy function using an evaluation function that includes the value function, and then updating the value function using an evaluation function that includes the policy function.

[0186] In the evaluation step (S612), the computer evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when reinforcement learning is performed using a dataset that does not include the data to be evaluated.

[0187] The data analysis method shown in Figure 7 allows us to evaluate the degree to which each data point used in reinforcement learning influences the learning outcome.

[0188] Figure 8 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. In the configuration shown in Figure 8, the computer 700 comprises a CPU 710, a main memory 720, an auxiliary memory 730, an interface 740, and a non-volatile recording medium 750.

[0189] One or more of the above-mentioned data analysis device 100, data analysis device 610, display device 620, and processing device 630, or a part thereof, may be implemented in the computer 700. In that case, the operation of each processing unit described above is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it into the main memory 720, and executes the above processing according to the program. The CPU 710 also reserves memory areas in the main memory 720 corresponding to each of the above-mentioned storage units according to the program. Communication between each device and other devices is performed by the interface 740 having a communication function and communicating according to the control of the CPU 710.

[0190] When the data analysis device 100 is implemented in a computer 700, the operation of the control unit 190 and its various components is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from auxiliary storage device 730, loads it into main memory 720, and executes the above processing according to the program.

[0191] Furthermore, the CPU 710 allocates the storage area of ​​the storage unit 180 in the main memory 720 according to the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and accepting user operations under the control of the CPU 710.

[0192] When the data analysis device 610 is implemented in the computer 700, the operations of the learning unit 611 and the evaluation unit 612 are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above processes according to the program.

[0193] Furthermore, the CPU 710 reserves memory in the main memory 720 for processing by the data analysis device 610 according to the program. Communication between the data analysis device 610 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the data analysis device 610 and the user is performed by the interface 740 equipped with a display device and an input device, displaying various images under the control of the CPU 710 and accepting user operations.

[0194] When the display device 620 is implemented in the computer 700, the operation of the display device 620 is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from auxiliary storage device 730, loads it into main memory device 720, and executes the above process according to the program.

[0195] Furthermore, the CPU 710 reserves memory in the main memory 720 for the display device 620 to process, according to the program. Communication between the display device 620 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the display device 620 and the user is performed by the interface 740, which is equipped with a display device and an input device, displays various images under the control of the CPU 710, and accepts user operations.

[0196] When the processing unit 630 is implemented in the computer 700, the operations of the learning unit 631, the evaluation unit 632, and the processing execution unit 633 are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above processing according to the program.

[0197] Furthermore, the CPU 710 reserves memory in the main memory 720 for the processing unit 630 to perform processing according to the program. Communication between the processing unit 630 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the processing unit 630 and the user is performed by the interface 740 being equipped with a display device and an input device, displaying various images under the control of the CPU 710, and accepting user operations.

[0198] One or more of the above-mentioned programs may be recorded on the non-volatile recording medium 750. In this case, the interface 740 may read the program from the non-volatile recording medium 750. The CPU 710 may then either directly execute the program read by the interface 740, or temporarily save it in the main memory 720 or auxiliary memory 730 before executing it.

[0199] Alternatively, a program for executing all or part of the processing performed by the data analysis device 100 and the data analysis device 610 may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform the processing of each part. The term "computer system" here includes hardware such as the operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs (Read Only Memory), CD-ROMs (Compact Disc Read Only Memory), and storage devices such as hard disks built into computer systems. The above-mentioned program may be intended to implement only a part of the functions described above, and may also be able to implement the above-mentioned functions in combination with programs already recorded in the computer system.

[0200] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0201] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0202] (Note 1) A learning method that performs reinforcement learning by updating the policy function using an evaluation function that includes a value function, and updating the value function using an evaluation function that includes a policy function, Evaluation means for evaluating the degree of influence of the data to be evaluated on the learning result, using the evaluation of the learning result when performing the reinforcement learning using a data set including the data to be evaluated and the evaluation of the learning result when performing the reinforcement learning using a data set not including the data to be evaluated. A data analysis device comprising the same.

[0203] (Appendix 2) The learning means acquires any one data among the data included in the training data set which is a data set used for reinforcement learning, and for each data included in the training data set, a partial model obtained by excluding a part of the machine learning model constituting the policy function and a partial model obtained by excluding a part of the machine learning model constituting the value function are specified uniquely by a learning mask. Among the learning masks, a learning mask corresponding to the acquired data is selected, and with the selected learning mask applied to the machine learning model constituting the policy and the machine learning model constituting the value function, the acquired data is used to update the values of the parameters of the partial model of the machine learning model constituting the policy function and the values of the parameters of the partial model of the machine learning model constituting the value function. function With the selected learning mask applied to the machine learning model constituting the policy and the machine learning model constituting the value function, the acquired data is used to update the values of the parameters of the partial model of the machine learning model constituting the policy function and the values of the parameters of the partial model of the machine learning model constituting the value function. The evaluation means excludes from the target of exclusion the part that is targeted for exclusion by the learning mask corresponding to one piece of evaluation target data selected from the data included in the training data set, and uses an evaluation mask that targets for exclusion all or part of the part that is not targeted for exclusion by that learning mask. With the evaluation mask applied to the machine learning model constituting the policy and the machine learning model constituting the value function, the evaluation value obtained by evaluating the learning result of the reinforcement learning is used as the evaluation value of the learning result when performing the reinforcement learning using a data set not including the data to be evaluated, to evaluate the degree of influence of the data to be evaluated on the learning result. function With the evaluation mask applied to the machine learning model constituting the policy and the machine learning model constituting the value function, the evaluation value obtained by evaluating the learning result of the reinforcement learning is used as the evaluation value of the learning result when performing the reinforcement learning using a data set not including the data to be evaluated, to evaluate the degree of influence of the data to be evaluated on the learning result. The data analysis device according to Appendix 1.

[0204] (Appendix 3) A computer Perform reinforcement learning to update a policy function using an evaluation function including a value function, and update the value function using an evaluation function including the policy function. Using the evaluation of the learning result when performing the reinforcement learning using a data set including the data to be evaluated and the evaluation of the learning result when performing the reinforcement learning using a data set not including the data to be evaluated, evaluate the degree of influence of the data to be evaluated on the learning result. A data analysis method including this.

[0205] (Appendix 4) On a computer, Perform reinforcement learning to update a policy function using an evaluation function including a value function, and update the value function using an evaluation function including the policy function. Using the evaluation of the learning result when performing the reinforcement learning using a data set including the data to be evaluated and the evaluation of the learning result when performing the reinforcement learning using a data set not including the data to be evaluated, evaluate the degree of influence of the data to be evaluated on the learning result. A data analysis method including this.

[0206] (Appendix 5) Learning means for performing reinforcement learning to update a policy function using an evaluation function including a value function, and update the value function using an evaluation function including the policy function. Evaluation means for using the evaluation of the learning result when performing the reinforcement learning using a data set including the data to be evaluated and the evaluation of the learning result when performing the reinforcement learning using a data set not including the data to be evaluated to evaluate the degree of influence of the data to be evaluated on the learning result. Based on the evaluation of the learning result and the evaluation of the degree of influence, select some data from the data included in the training data set, which is the data set used for the reinforcement learning, and delete the selected data from the training data set. Processing execution means. A processing device comprising this.

[0207] (Note 6) A learning method that performs reinforcement learning by updating the policy function using an evaluation function that includes a value function, and updating the value function using an evaluation function that includes a policy function, An evaluation means for evaluating the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when the reinforcement learning is performed using a dataset containing the data to be evaluated, and an evaluation of the learning results when the reinforcement learning is performed using a dataset that does not contain the data to be evaluated. Processing execution means that, based on the evaluation of the learning results and the evaluation of the degree of influence, select one or more data from the data included in the training dataset, which is the dataset used for reinforcement learning, and adjust at least one of the controlled object that is the target of reinforcement learning, or the operating environment of the controlled object, based on the selected data. A processing device equipped with the following features.

[0208] (Note 7) A display means for displaying the degree to which the data included in the training dataset, which is a dataset used for reinforcement learning, influences the results of the reinforcement learning. A display device equipped with the following features.

[0209] (Note 8) Display means for displaying the degree of influence of the data included in the training dataset, which is a dataset used for reinforcement learning, on the results of the reinforcement learning, and an evaluation of the results of the reinforcement learning. A display device equipped with the following features.

[0210] (Note 9) Display means for displaying: the degree of influence of the data included in the training dataset, which is a dataset used for reinforcement learning, on the results of the reinforcement learning; an evaluation of the results of the reinforcement learning; and a message regarding the deletion of some of the data included in the training dataset. A display device equipped with the following features.

[0211] (Note 10) Display means for each of multiple controlled objects to display an evaluation of the reinforcement learning results for that controlled object, to display, in a selectable manner for each controlled object, the degree to which the data included in the training dataset, which is the dataset used for reinforcement learning, has an influence on the reinforcement learning results, and to display a query asking whether or not to delete some of the data included in the training dataset that was selected based on the degree to which it had an influence on the reinforcement learning results for the selected controlled object. A display device equipped with the following features.

[0212] (Note 11) A display means for displaying the degree of influence on the reinforcement learning results for each data point and for each evaluation iteration in the form of a heatmap in a coordinate space comprising: a first axis representing numerical values ​​that identify data points included in a training dataset, which is a dataset used for reinforcement learning; and a second axis representing the number of iterations of evaluating the degree of influence of each data point included in the training dataset on the reinforcement learning results. A display device equipped with the following features.

[0213] (Note 12) A coordinate space comprising a first axis representing numerical values ​​that identify data included in a training dataset used for reinforcement learning, and a second axis representing the number of iterations of evaluating the degree of influence of each data in the training dataset on the results of the reinforcement learning, displays a heatmap showing the degree of influence of each data and each iteration of the evaluation on the results of the reinforcement learning, including the display of a portion of the heatmap selected according to the degree of influence, and displays a query asking whether or not to delete the data in the selected portion from the training dataset. A display device equipped with the following features.

[0214] (Note 13) For each repetition of the evaluation of the degree of influence of each data included in the training data set, which is a data set used for reinforcement learning, on the result of the reinforcement learning, display the evaluation of the result of the reinforcement learning, and in the coordinate space constituted by the first axis indicating the numerical value for identifying the data included in the training data set and the second axis indicating the number of repetitions of the evaluation of the degree of influence, display a heat map showing the degree of influence of each data and for each repetition of the evaluation on the result of the reinforcement learning, including display of a selected portion in the heat map according to the degree of influence, and display means for displaying an inquiry as to whether or not to delete the data of the selected portion from the training data set A display device comprising the same.

[0215] (Appendix 14) For each of a plurality of control targets, for each repetition of the evaluation of the degree of influence of each data included in the training data set, which is a data set used for reinforcement learning for that control target, on the result of the reinforcement learning, display the evaluation of the result of the reinforcement learning, and in the coordinate space constituted by the first axis indicating the numerical value for identifying the data included in the training data set and the second axis indicating the number of repetitions of the evaluation of the degree of influence, display a heat map showing the degree of influence of each data and for each repetition of the evaluation on the result of the reinforcement learning, including display of a selected portion in the heat map according to the degree of influence, and display means for selectively displaying for each control target and displaying an inquiry as to whether or not to delete the data of the selected portion in the selected heat map from the training data set A display device comprising the same.

Industrial Applicability

[0216] The present invention may be applied to a data analysis device, a data analysis method, and a recording medium.

Explanation of Signs

[0217] 100, 610 Data analysis device 110 Communication unit 120 Display section 130 Operation Input Section 180 Storage section 190 Control Unit 191, 611 Learning Department 192, 612 Evaluation Department 193 Processing Execution Unit 620 Display device 630 Processing Unit

Claims

1. A learning method that performs reinforcement learning by updating the policy function using an evaluation function that includes a value function, and updating the value function using an evaluation function that includes a policy function, An evaluation means for evaluating the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when the reinforcement learning is performed using a dataset containing the data to be evaluated, and an evaluation of the learning results when the reinforcement learning is performed using a dataset that does not contain the data to be evaluated. Equipped with, The learning means acquires one data point from the training dataset, which is a dataset used for reinforcement learning, and selects a learning mask from a set of learning masks that uniquely specify for each data point in the training dataset, which excludes a submodel obtained by excluding a part of the machine learning model that constitutes the policy function and a submodel obtained by excluding a part of the machine learning model that constitutes the value function. The learning means then applies the selected learning mask to the machine learning model that constitutes the policy function and the machine learning model that constitutes the value function, and updates the parameter values ​​of the submodels of the machine learning model that constitutes the policy function and the submodels of the machine learning model that constitutes the value function using the acquired data. The evaluation means evaluates the degree to which the data to be evaluated has an influence on the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated, by applying an evaluation mask that excludes the portion of the data to be evaluated that is excluded by the learning mask corresponding to one data to be evaluated selected from the data included in the training dataset, and excludes all or part of the portion that is excluded by the learning mask, to the machine learning model that constitutes the policy function and the machine learning model that constitutes the value function, and using the evaluation value obtained in this state, the evaluation of the learning results is used as the evaluation value of the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated, thereby evaluating the degree to which the data to be evaluated has an influence on the learning results. Data analysis device.

2. Computers A reinforcement learning process that updates the policy function using an evaluation function that includes the value function, and then updates the value function using an evaluation function that includes the policy function, An evaluation process is performed to evaluate the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when the reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated. The reinforcement learning process includes: acquiring one data point from the training dataset, which is a dataset used for reinforcement learning; selecting a learning mask from a set of learning masks that uniquely designate a partial model obtained by excluding a part of the machine learning model that constitutes the policy function, and a partial model obtained by excluding a part of the machine learning model that constitutes the value function, for each data point in the training dataset; applying the selected learning mask to the machine learning model that constitutes the policy function and the machine learning model that constitutes the value function; and updating the parameter values ​​of the partial model of the machine learning model that constitutes the policy function and the partial model of the machine learning model that constitutes the value function using the acquired data. The evaluation process includes evaluating the degree to which the data to be evaluated has an influence on the learning results, by applying an evaluation mask to the machine learning model constituting the policy function and the machine learning model constituting the value function, which excludes the portion of the data to be evaluated that is excluded by the learning mask corresponding to one data to be evaluated selected from the data included in the training dataset, and excludes all or part of the portion that is excluded by the learning mask, and then using the evaluation value obtained by evaluating the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated, Data analysis methods.

3. On the computer, A reinforcement learning process that updates the policy function using an evaluation function that includes the value function, and then updates the value function using an evaluation function that includes the policy function, An evaluation process that evaluates the degree to which the data to be evaluated has an influence on the learning results, using an evaluation of the learning results when the reinforcement learning is performed using a dataset that includes the data to be evaluated, and an evaluation of the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated. Make it run, The reinforcement learning process includes: acquiring one data point from the training dataset, which is a dataset used for reinforcement learning; selecting a learning mask from a set of learning masks that uniquely designate a partial model obtained by excluding a part of the machine learning model that constitutes the policy function, and a partial model obtained by excluding a part of the machine learning model that constitutes the value function, for each data point in the training dataset; applying the selected learning mask to the machine learning model that constitutes the policy function and the machine learning model that constitutes the value function; and updating the parameter values ​​of the partial model of the machine learning model that constitutes the policy function and the partial model of the machine learning model that constitutes the value function using the acquired data. The evaluation process includes evaluating the degree to which the data to be evaluated has an influence on the learning results, by applying an evaluation mask to the machine learning model constituting the policy function and the machine learning model constituting the value function, which excludes the portion of the data to be evaluated that is excluded by the learning mask corresponding to one data to be evaluated selected from the data included in the training dataset, and excludes all or part of the portion that is excluded by the learning mask, and then using the evaluation value obtained by evaluating the learning results when the reinforcement learning is performed using a dataset that does not include the data to be evaluated, program.

Images