Reinforcement learning assistance device, reinforcement learning assistance system, and reinforcement learning assistance method

Abstract

To provide a reinforcement learning aid apparatus, a reinforcement learning aid system, and a reinforcement learning aid method capable of further efficiently performing at least one of generation of learning scenarios and evaluation of the learning scenarios.SOLUTION: According to an embodiment, the reinforcement learning aid apparatus comprises a generation unit and a prediction unit. The generation unit generates a learning scenario, which is a unit of learning data for a learning model. The prediction unit predicts the convergence of the learning model with respect to a test scenario that corresponds to the learning scenario and differs from the learning scenario in the learning model learned using the learning scenario.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a reinforcement learning assistance device, a reinforcement learning assistance system, and a reinforcement learning assistance method. [Background technology]

[0002] In recent years, reinforcement learning, a type of artificial intelligence technology, has been attracting attention as a breakthrough technology in fields where models are complex and advanced control is required. In a learning model that uses reinforcement learning to learn the control of motors and other devices, an operation amount is given to the controlled object, a reward value is calculated from the resulting control value, and the operation amount for each state is learned so that a high reward value can be obtained.

[0003] It is also known that when controlling motors, such as air conditioners and pumps, whose operation patterns are determined in advance, a learning model is trained using anticipated operation patterns as learning scenarios. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-34994 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in reinforcement learning using a learning scenario, the accuracy after learning and the number of learning rounds required for convergence vary depending on the learning scenario. Therefore, it is necessary to try various learning scenarios to find one that can control the vehicle according to the expected driving pattern. As there are countless learning scenarios, it is desirable to quickly evaluate each learning scenario and find learning data that can control the vehicle according to the movement pattern in a short time.

[0006] However, learning scenarios are typically created using mathematical formulas in a program. This makes it difficult for users to understand, and because it is necessary to set target values ​​corresponding to continuous time, changing the learning data takes time. Furthermore, reinforcement learning requires learning until convergence, and if the number of learning cycles until convergence is large, the learning time becomes long, which can make it difficult to evaluate the learning scenario.

[0007] In order to solve such problems, the object of this embodiment is to provide a reinforcement learning assistance device, a reinforcement learning assistance system, and a reinforcement learning assistance method that can more efficiently generate learning scenarios and / or evaluate learning scenarios. [Means for solving the problem]

[0008] According to this embodiment, the reinforcement learning assistance device includes a generation unit and a prediction unit. The generation unit generates a learning scenario that serves as a unit of learning data for the learning model. The prediction unit predicts the convergence of the learning model for a test scenario that corresponds to the learning scenario and is different from the learning scenario, for the learning model trained using the learning scenario. [Effects of the Invention]

[0009] At least one of generating a learning scenario and evaluating the learning scenario can be performed more efficiently. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example configuration of a reinforcement learning assistance system. [Figure 2] FIG. 10 is a diagram showing an example of a learning scenario. [Figure 3] FIG. 10 is a diagram showing an example of a test scenario. [Figure 4] FIG. 4 is a diagram showing the output results of the inference unit for the test scenario in FIG. 3. [Figure 5] FIG. 10 is a diagram showing the calculation process of the convergence prediction unit. [Figure 6]FIG. 10 is a diagram showing the inference process of a convergence prediction unit. [Figure 7] An example of a display image for creating a learning scenario. [Figure 8] FIG. 8 is a diagram showing an example of an evaluation result of the prediction unit when the learning scenario of FIG. 7 is used. [Figure 9] In the figure showing example results for the learning scenario in Figure 7 . [Figure 10] FIG. 10 is a diagram showing an example of a display image for creating a test scenario. [Figure 11] FIG. 10 is a diagram showing the output results of the inference unit for the test scenario of FIG. 9. [Figure 12] FIG. 10 is a diagram showing an example of a process for generating a learning scenario by modifying a test scenario. [Figure 13] FIG. 10 is a diagram showing an example of a subsequent process for generating a learning scenario from a test scenario. [Figure 14] 1 is a flowchart showing an example of the operation of the reinforcement learning assistance system. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a reinforcement learning assistance device, a reinforcement learning assistance system, and a reinforcement learning assistance method according to embodiments of the present invention will be described in detail with reference to the drawings. Note that the embodiments described below are examples of embodiments of the present invention, and the present invention should not be interpreted as being limited to these embodiments. Furthermore, in the drawings referred to in this embodiment, identical parts or parts having similar functions are given the same or similar symbols, and repeated explanations thereof may be omitted. Furthermore, for convenience of explanation, the dimensional ratios of the drawings may differ from the actual ratios, and some components may be omitted from the drawings.

[0012] (One embodiment) Fig. 1 is a block diagram showing an example of the configuration of a reinforcement learning assistance system 1 according to this embodiment. The configuration of the reinforcement learning assistance system 1 will be described using Fig. 1. As shown in Fig. 1, the reinforcement learning assistance system 1 according to this embodiment is a system capable of generating and evaluating a learning scenario used in reinforcement learning, for example.

[0013] In this embodiment, data corresponding to a driving pattern is referred to as a test scenario, and a unit of learning data used in reinforcement learning is referred to as a learning scenario. A learning scenario corresponds to, for example, characteristic information of a test scenario, and is used to train a learning model that reproduces the test scenario.

[0014] A learning model that has learned a test scenario can reproduce the test scenario. However, it has become clear that the reproducibility of a learning model that has learned a test scenario with respect to the test scenario is not necessarily optimal. Therefore, in this embodiment, a learning scenario with higher reproducibility with respect to the test scenario is generated. Alternatively, a learning scenario with higher convergence with respect to a predetermined convergence condition is generated.

[0015] As shown in FIG. 1, the reinforcement learning device 100 is configured to include, for example, a CPU (Central Processing Unit) and is a device for acquiring, through reinforcement learning using a learning scenario, a more suitable behavior of a control object, such as a motor, for which a certain operating pattern (test scenario) is assumed. The reinforcement learning device 100 includes an environment unit 110, a state input unit 120, a learning unit 130, a learning scenario acquisition unit 140, a learning parameter storage unit 150, and a learning parameter output unit 160. Details of the reinforcement learning device 100 will be described later. The reinforcement learning device 100 configures each processing unit by, for example, executing a program stored in the learning parameter storage unit 150.

[0016] The reinforcement learning assistance device 200 is configured to include, for example, a CPU (Central Processing Unit) and is capable of creating and evaluating various scenarios. The reinforcement learning assistance device 200 includes an evaluation unit 210, a prediction unit 220, a creation unit 230, a display control unit 240, a display unit 250, and an operation unit 260. The reinforcement learning assistance device 200 configures each processing unit by, for example, executing a program stored in an evaluation parameter storage unit 212 of the evaluation unit 210.

[0017] The evaluation unit 210 evaluates the performance in the convergence process of the learning model trained by the reinforcement learning device 100 using the learning scenario created by the creation unit 230. The evaluation unit 210 includes an evaluation parameter acquisition unit 211, an evaluation parameter storage unit 212, an evaluation environment unit 213, an evaluation state recognition unit 214, an evaluation target recognition unit 215, a test scenario storage unit 216, an inference unit 217, and an evaluation result output unit 218. Details of the evaluation unit 210 will be described later.

[0018] The prediction unit 220 predicts the convergence of the learning model when learning with a learning scenario and the reproducibility of the test scenario using information on the performance of the learning model in the convergence process. The prediction unit 220 includes an evaluation result storage unit 221 and a convergence prediction unit 222.

[0019] The creation unit 230 is capable of creating test scenarios and various learning scenarios corresponding to the test scenarios, and includes a drawing unit 231 and a scenario output unit 232. Note that the creation unit 230 according to this embodiment corresponds to the generation unit. Details of the creation unit 230 will also be described later.

[0020] The display control unit 240 causes the display unit 250 to display various types of data generated by the reinforcement learning assistance device 200 as images. The display unit 250 is, for example, a monitor, and displays various types of data as images under the control of the display control unit 240. The operation unit 260 is composed of, for example, a mouse and a keyboard, and is capable of inputting operation signals according to operations by an operator to the reinforcement learning device 100 and the reinforcement learning assistance device 200. The display unit 250 and the operation unit 260 may be configured as an integrated touch panel.

[0021] (Reinforcement learning device) Figure 2 shows an example of a learning scenario. The horizontal axis represents the step time s, and the vertical axis represents the control value ωr(s). This learning scenario corresponds to the test scenario (see Figure 3), and is generated by extracting information from the acceleration and deceleration regions of the test scenario, for example.

[0022] Here, the reinforcement learning device 100 will be described in detail with reference to FIG. 2. The reinforcement learning device 100 uses a learning scenario to learn a learning model. This reinforcement learning device 100 is capable of generating information on the progress of learning in the learning model. A reinforcement learning assistance device 200 (described later) uses this information on the progress of learning to evaluate the convergence and reproducibility to a test scenario of the learning model learned using the learning scenario.

[0023] The environment unit 110 includes, for example, a control object and a device for acquiring information about the control object. In this embodiment, the environment unit 110 is capable of generating a new state ω(s+1) resulting from operating the control object, which is the environment, based on the current state ω(s) of the control object and an action estimation V(s) calculated from the state ω(s) using a learning model in the learning unit 130. For example, in this embodiment, the current state ω(s) corresponds to a control value, and if the control object is a motor, the current state ω(s) is the current rotation speed of the motor, the action estimation V(s) is the voltage, and the new state ω(s+1) is the next rotation speed resulting from application of the voltage.

[0024] The environment unit 110 generates a reward R(s) using the current state ω(s) acquired from the control object, an action estimation V(s) calculated from this state ω(s) using the learning model of the learning unit 130, and a new state ω(s+1) obtained by manipulating the control object, which is the environment, using this action estimation V(s). For example, the reward R(s) for the action estimation V(s) is calculated based on the state ω(s+1) and the control value ωr(s+1) shown in FIG. 2. For example, the reward R(s) is set to a larger value as the state ω(s+1) approaches the target control value ωr(s+1). Note that, although the present embodiment will be described using one state ω(s), this is not limiting. There may be multiple types of states ω(s).

[0025] The learning unit 130 is supplied with the control value ωr(s) shown in FIG. 2 as a target value in time series from the target recognition unit 122 for each step time s. The learning unit 130 is also supplied with the state ω(s) and the immediate reward R(s) at step time s from the environment unit 110. As a result, the learning unit 130 learns the learning model so as to obtain a large reward R(s). By repeating this learning, the reproducibility of the time-series fluctuation of the control value shown in FIG. 2 is improved in the control of the control object using the learning model. In other words, as the learning of the learning model progresses, it comes to output a behavior estimation V(s) that approaches the state ωr(s+1) for the state ω(s).

[0026] More specifically, the environment unit 110 operates an actual device such as a motor and measures its movement with a sensor or a camera to generate a new state ω(s+1) that has changed due to the action estimation V(s) of the control object, and a reward R(s) that evaluates the result. Alternatively, the environment unit 110 may operate a simulation model by simulation using a simulator of the model of the control object, and generate the state that has changed due to the action estimation V(s) of the control object, the new changed state ω(s+1), and a reward R(s) that evaluates the result. As described above, if the control object is a motor, for example, the state ω(s) is the actually measured rotational speed of the motor, the action estimation V(s) is the voltage applied to the motor, and the control value ωr(s+1) is the target rotational speed of the motor.

[0027] The learning scenario storage unit 121 of the state input unit 120 stores, as a learning scenario, time-series data of the control value ωr(s), for example, as shown in Fig. 2. The goal recognition unit 122 recognizes the control value ωr(s) corresponding to the step time s from the learning scenario data stored in the learning scenario storage unit 121 as a target value, and outputs it to the learning unit 130. That is, the goal recognition unit 122 outputs the control value ωr(s) corresponding to each step time s to the learning unit 130 in time series based on the learning scenario data.

[0028] The learning unit 130 has a learning function, and as described above, learns a learning model (e.g., a neural network) by Q-learning, which is reinforcement learning, using the current state ω(s) supplied from the state recognition unit 123 and the reward R(s) supplied from the environment unit 110. That is, this learning model receives the state ω(s) as an input and outputs an action estimation V(s).

[0029] The learning scenario acquisition unit 140 acquires learning scenario data from the scenario output unit 232 of the reinforcement learning assistance device 200 and outputs it to the learning scenario storage unit 121. The learning parameter storage unit 150 stores each parameter of the learning model learned by the learning unit 130 at a predetermined cycle. The learning model parameters are, for example, information such as weight coefficients of a neural network. That is, the learning parameter storage unit 150 stores the parameters of the learning model in chronological order as the learning progress of the learning model for each predetermined number of times. Then, the learning parameter output unit 160 associates each parameter of the learning model stored in the learning parameter storage unit 150 with the number of times of learning, and supplies the associated parameters to the evaluation parameter acquisition unit 211 of the reinforcement learning assistance device 200.

[0030] (Evaluation unit of reinforcement learning assist device) The evaluation parameter acquisition unit 211 of the evaluation unit 210 acquires the parameters of the learning model associated with the number of times of learning via the learning parameter output unit 160, and supplies them to the evaluation parameter storage unit 212. The evaluation parameter storage unit 212 stores the parameters of the learning model associated with the number of times of learning as the learning progress.

[0031] The inference unit 217 generates an inference model using the parameters of the learning model stored in the evaluation parameter storage unit 212. In other words, this inference model corresponds to the control model. As described above, this inference model receives, for example, the state ωr(s) as input and outputs the behavior estimation V(s).

[0032] 3 is a diagram showing an example of a test scenario, where the horizontal axis represents the step time s and the vertical axis represents the control value ωr(s).

[0033] Here, the evaluation unit 210 will be described in detail with reference to FIG. 3. As described above, the test scenario corresponds to, for example, a driving pattern. In the case of a motor, the patterns of acceleration and deceleration portions may be important. In this case, data on the constant speed driving portion becomes redundant for learning. For this reason, as described above with reference to FIG. 2, data on the acceleration and deceleration portions is used as the learning scenario. On the other hand, evaluation of the control model is performed in correspondence with the driving pattern. For this reason, the evaluation unit 210 uses a test scenario corresponding to the actual driving pattern for evaluation, as shown in FIG. 3. Evaluation of the test scenario is performed using data for one cycle. For example, in FIG. 3, 1 to 30 step times (s) constitute one cycle of data.

[0034] The evaluation environment unit 213 of the evaluation unit 210 receives as input the behavior estimation V(s) calculated by the inference unit 217 for the current state ω(s), and generates the next state ω(s+1). Then, the evaluation environment unit 213 outputs the next state ω(s+1) to the evaluation state recognition unit 214. More specifically, similar to the environment unit 110, the evaluation environment unit 213 operates an actual machine such as a motor and measures its movement with a sensor or a camera, thereby generating the next state ω(s+1) that has changed according to the behavior estimation V(s) for the control object. Furthermore, similar to the environment unit 110, the evaluation environment unit 213 may operate the model by simulation using a simulator of the model of the control object, and generate the next state ω(s+1) that has changed according to the behavior estimation V(s) for the control object.

[0035] The evaluation state recognition unit 214 acquires the next state ω(s+1) of the controlled object from the evaluation environment unit 213 as described above, recognizes the state of the controlled object, and outputs it to the inference unit 217 .

[0036] The evaluation target recognition unit 215 recognizes the control value ωr(s) corresponding to time as a target value from the test scenario data (see FIG. 3) stored in the test scenario storage unit 216, and outputs it to the inference unit 217. The test scenario storage unit 216 stores, for example, time-series data of the control value ωr(s) (e.g., target rotation speed) shown in FIG. 3 as a test scenario.

[0037] 3 is supplied to the inference unit 217 in time series for each step from the evaluation target recognition unit 215. The inference model (control model) of the inference unit 217 generates a behavior estimation V(s) for each step time s according to the state ωr(s), and outputs it to the evaluation environment unit 213.

[0038] FIG. 4 is a diagram showing the output result of the inference unit 217 for the test scenario of FIG. 3. The horizontal axis represents time, and the vertical axis represents the control value. Line L10 represents the test scenario. Line L12 represents the state ω(s+1) output by the evaluation environment unit 213 for the behavior estimation V(s) output by the control model of the inference unit 217. For example, if the controlled object is a motor, line L10 corresponds to the target rotation speed of the motor, and line L12 corresponds to the actual rotation speed of the motor output by the evaluation environment unit 213. The evaluation result output unit 218 associates the state ω(s+1) corresponding to the control value output by the evaluation environment unit 213 with the step time s of the test scenario, and outputs it to the evaluation result storage unit 220 of the prediction unit 220.

[0039] (Prediction part of reinforcement learning assistant device) The evaluation result storage unit 220 of the prediction unit 220 stores the state ω(s+1) associated with the step time s of the test scenario for each learning iteration of the learning model.

[0040] 5 is a diagram showing the calculation progress of the convergence prediction unit 222. The horizontal axis represents time, and the vertical axis represents the difference value between line L10 and line L12 shown in FIG. 4. As shown in FIG. 5, the convergence prediction unit 222 calculates the difference value between line L10 and line L12 for one cycle (e.g., 30 seconds) of the test scenario, and calculates an average error, which is the average value of the absolute values ​​of these difference values, as the evaluation result for the number of learning rounds. Note that the evaluation result calculated by the convergence prediction unit 222 is not limited to the average error, and may be, for example, the maximum error.

[0041] FIG. 6 is a diagram showing the inference process of the convergence prediction unit 222. The horizontal axis represents the number of learning rounds, and the vertical axis represents the average error. Line L14 represents the value of the average error, which is the evaluation result for each learning round. The convergence prediction unit 222 generates a convergence curve L16 based on the average error, which is the evaluation result for each learning round. The convergence prediction unit 222 can use common algorithms, such as least-squares approximation, spline interpolation, multiple regression analysis, and neural networks, to generate the convergence curve L16.

[0042] The convergence prediction unit 222 predicts the convergence value of the learning model based on the convergence curve L16. The convergence prediction unit 222 also predicts the number of learning iterations until a predetermined convergence value (convergence condition), for example, 0.2, is reached based on the convergence curve L16. Note that the closer the convergence value is to 0, the higher the reproducibility of the learning model for the test scenario.

[0043] In this way, the convergence prediction unit 222 can predict the learning results, for example, in the convergence state of the learning model, using the evaluation values ​​up to, for example, 10,000 learning iterations before the learning converges. This makes it possible to evaluate the learning scenario in a shorter time. Therefore, as will be described later, it becomes possible to evaluate the learning scenario in a shorter time each time the operator generates a learning scenario.

[0044] (Creation unit and display control unit of the reinforcement learning assistance device)

[0045] The drawing unit 231 can create test scenarios and learning scenarios using a graphical user interface (GUI) via the display control unit 240. The scenario output unit 232 stores the test scenarios and learning scenarios generated by the drawing unit 231 in the learning scenario storage unit 121 and the test scenario storage unit 216 in association with each other.

[0046] Fig. 7 is a diagram showing an example of a display image M8 for creating a learning scenario generated by the drawing unit 231. As shown in Fig. 7, the drawing unit 231 can cause the display unit 250 to display the display image M8 for the learning scenario via the display control unit 240. The display image M8 for the learning scenario displays data for each step time s of the learning scenario L18. The vertical axis represents the control value ωr(s), and the horizontal axis represents the step time s.

[0047] The operator can input the number of executions of learning A10, the evaluation period A12, the sampling period A14, the learning scenario time A16, and the convergence condition A18 via the operation unit 260. In addition, when the start learning button A20 is pressed, evaluation of the learning scenario L18 begins. The number of executions of learning A10 is the number of times learning has been performed for the learning scenario L18. For example, if the number of executions of learning using the learning scenario L18, which has 30 steps, is 10,000, then 300,000 steps of learning will be performed on the learning model.

[0048] The evaluation period A12 is the evaluation period for the number of learning times. For example, if the number of learning times is 500, the parameters of the learned learning model are used every 500 learning times to evaluate the error of the learning model. The sampling period A14 is the time interval between each step time s when the learning scenario L18 is used as scenario data. For example, if the sampling period is 1 second, 30 steps of data are generated. Similarly, if the sampling period is 0.1 seconds, 300 steps of data are generated.

[0049] The learning scenario time A16 is the time range for one cycle of the learning scenario L18. For example, if it is 30 seconds, 30 seconds of data from the learning scenario L18 is used for one learning of the learning model. The convergence condition A18 sets the convergence condition. For example, if it is 0.2, the convergence prediction unit 222 predicts the number of learning iterations until the error reaches 0.2.

[0050] Furthermore, the drawing unit 231 can change the shape of the learning scenario L18 by moving each point P10 through operation of the operation unit 260. The drawing unit 231 regenerates the learning scenario L18 by connecting each moved point P10 using spline interpolation, least-squares approximation interpolation, or the like. In this way, the learning scenario L18 can be created using a GUI with two axes, time and control value, making it easier to change the learning scenario L18 and understand the changes.

[0051] Next, when the learning start button A20 is pressed, the evaluation of the learning scenario L18 begins. Fig. 8 is a diagram showing an example of the evaluation result of the prediction unit 220 when the learning scenario L18 of Fig. 7 is used.

[0052] Screens M10, M10a, and M12 are example screens displayed on the display unit 250 via the display control unit 240. As described above, screen M10 displays the error L14 and convergence prediction curve L16, which are the processing results of the convergence prediction unit 222. Screen M10a shows the error calculation method used for the convergence error.

[0053] Screen M12 also displays the processing results of the convergence prediction unit 222, including the convergence error estimate, which is the expected error value at convergence, and the estimated number of learning iterations until the error reaches the convergence condition A18 (see FIG. 7). As described above, the convergence error estimate indicates, as an objective numerical value, the reproducibility of the learning model trained with learning scenario L18 with respect to test scenario L10 (see FIG. 4). For example, a value of 0 indicates that test scenario L10 (see FIG. 4) can be perfectly reproduced.

[0054] In this way, the convergence prediction unit 222 predicts and displays the expected number of learning iterations and the expected convergence error, thereby enabling the evaluation of the learning scenario to be completed in a shorter time. In other words, even if the learning of the learning model is stopped midway, the convergence of the learning model can be estimated.

[0055] As a result, an operator who sees such results can move each point P10 by operating the operation unit 260 to regenerate a new learning scenario L18 and re-evaluate it in a shorter time. By repeating such operations, it becomes possible to generate a desired learning scenario. After generating a desired learning scenario, for example, the number of execution learning times A10 (see FIG. 7) can be set with reference to the expected number of required learning times, and the reinforcement learning device 100 can be made to learn a final learning model.

[0056] FIG. 9 is a diagram showing an example result for learning scenario L18 in FIG. 7. Screens M14, M16, and M18 are example screens displayed on the display unit 250 by the drawing unit 231 via the display control unit 240. Screen M14 is a diagram showing the output result of the inference unit 217 for learning scenario L18 in FIG. 7. The horizontal axis represents time, and the vertical axis represents the control value. Line L20 is the test scenario. Line L22 is the state ω(s+1) corresponding to the control value output by the evaluation environment unit 213 for the action value V(s) output by the learning model of the inference unit 217 learned using learning scenario L18. The horizontal axis represents step time s, and the vertical axis represents the control value ωr(s).

[0057] Screen M16 is a diagram showing the calculation progress of the convergence prediction unit 222. The horizontal axis is step time s, and the vertical axis is the difference value between line L20 and line L22 shown in M14. Screen M18 is an image showing the number of executed learnings A10 (see FIG. 7). By presenting such an image, it becomes easy to judge the performance of the learning model when learning scenario L18 is used.

[0058] Fig. 10 is a diagram showing an example of a display image M18 for creating a test scenario, which is generated by the drawing unit 231 via the display control unit 240. As shown in Fig. 10, the drawing unit 231 can cause the display unit 250 to display the display image M18 for the test scenario via the display control unit 240. The display image M18 for the test scenario displays data for each step time s of the test scenario L20. The vertical axis represents the control value ωr(s), and the horizontal axis represents the step time s.

[0059] The operator sets the test scenario period A22 via the operation unit 260. This enables the drawing unit 231 to generate a test scenario L20 according to the test scenario period A22.

[0060] Furthermore, the drawing unit 231 can move each point P12 by operating the operation unit 260. The drawing unit 231 connects each of the moved points P12 using spline interpolation, least-squares approximation interpolation, or the like, to regenerate the test scenario L20. In this way, the test scenario L20 can be created using a GUI with two axes, time and control value, making it easier to change the test scenario L20 and understand the changes.

[0061] FIG. 11 is a diagram showing the output result of the inference unit 217 for the test scenario L20 in FIG. 9. That is, it shows the evaluation result after learning the test scenario L20 itself 10,000 times as the learning scenario. In FIG. 11, the number of learning times indicates the number of times the learning model has been trained using the test scenario L20 itself, and time corresponds to the step time s on the horizontal axis in FIG. 10. The evaluation result indicates the error for each time of the test scenario L20, and the scenario data indicates the data value for each time of the test scenario L20.

[0062] For example, an operator can refer to such analysis results and change the test scenario L20 to generate a learning scenario. For example, referring to Figure 11, it can be seen that the error for the constant values ​​0 and 10 of the test scenario L20 is large, resulting in poor convergence.

[0063] FIG. 12 shows an example process for generating a learning scenario by modifying test scenario L20. The figure shows the process by which an operator modifies test scenario L20 to generate learning scenario L23 using a display image for creating learning scenarios (see FIG. 7). Because test scenario L20 has a large error relative to the constant values ​​0 and 10, resulting in poor convergence, this example shows the generation of learning scenario L23 centered around the range of mark M20. Evaluation of learning scenario L23 can also be easily performed using the diagram showing an example of the evaluation results of prediction unit 220 (see FIG. 8). This allows the operator to objectively grasp, in a short amount of time, that convergence improves by reducing the linear region of test scenario L20, for example.

[0064] FIG. 13 shows an example process for generating learning scenario L18 (see FIG. 7) by modifying learning scenario L23. Using the display image for creating learning scenarios (see FIG. 7), the operator manipulates point P12 of learning scenario L23 to generate learning scenario L24. Evaluation of learning scenario L24 can also be easily performed using a diagram showing an example of the evaluation results of the prediction unit 220 (see FIG. 8). This allows the operator to objectively and quickly grasp, for example, that further reducing the linear region of learning scenario L23 further improves convergence. The operator can then continuously copy learning scenario L24 to generate learning scenario L18 shown in FIG. 7. Alternatively, the operator can generate a learning scenario equivalent to learning scenario L18 of FIG. 7 by setting the learning scenario period A16 to 10 and the number of executions A10 to 30,000 for learning scenario L24. In this way, learning scenarios can be created using a GUI with two axes of time and control value, making it easier to change learning scenario data and understand the changes, and therefore generating the desired learning scenario L18 can be done in a shorter time.

[0065] The above is a description of the configuration according to this embodiment, and an example of control processing will now be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the operation of the reinforcement learning assistance system 1. As shown in Fig. 14, the reinforcement learning assistance device 200 uses a predetermined GUI (see Figs. 7 and 10) to create a test scenario and a learning scenario corresponding to the test scenario (step S1).

[0066] Next, the reinforcement learning device 100 performs learning of the learning model based on the learning scenario generated by the reinforcement learning assistance device 200 (step S2). The reinforcement learning device 100 determines whether the specified number of execution learning times A10 (see FIG. 7) has been completed (step S3). If it determines that the execution learning has not been completed (No in step S3), the reinforcement learning device 100 repeats the process from step S2. During this time, the reinforcement learning device 100 stores the parameters of the learning model at a period according to the evaluation period A12 (see FIG. 7).

[0067] On the other hand, when the reinforcement learning device 100 determines that the learning has ended (Yes in step S3), the reinforcement learning device 100 periodically outputs the stored parameters of the learning model to the reinforcement learning assistance device 200.

[0068] Next, the reinforcement learning assistance device 200 uses the periodically stored parameters of the learning model to generate an average error for each period as an evaluation of the test scenario (see FIG. 6) (step S4).

[0069] Next, the reinforcement learning assistance device 200 predicts convergence using the average error for each period (see FIG. 6) (step S5). Subsequently, the reinforcement learning assistance device 200 causes the display unit 250 to display information related to the convergence prediction (see FIG. 8), and ends the process (step S6).

[0070] As described above, according to this embodiment, the drawing unit 231 generates a learning scenario L18 that serves as a unit of learning data for a learning model, and the prediction unit 220 predicts the convergence of the learning model trained using the learning scenario L18 for a test scenario L20 that corresponds to the learning scenario L18 but is different from the learning scenario L18. This makes it possible to evaluate the test scenario L20 of the learning scenario L18, which is different from the test scenario L20 corresponding to the driving pattern, in a shorter time.

[0071] Furthermore, the learning scenario L18 and test scenario L20 generated by the drawing unit 231 are displayed on the display unit 250 via the display control unit 240, and the shapes of the scenarios L18 and L20 displayed on the display unit 250 can be changed and regenerated via the operation unit 260. This makes it possible to regenerate the scenarios L18 and L20 while observing the changes to the scenarios L18 and L20.

[0072] Furthermore, the prediction unit 220 quantifies the number of learning times that satisfy a predetermined convergence value and the reproducibility for the test scenario L20 based on the evaluation value of the learning model in the middle of learning using the learning scenario L18. This makes it possible to evaluate the learning scenario L18 before the learning model using the learning scenario L18 converges, and to regenerate the learning scenario L18 with an objective evaluation in a shorter time.

[0073] Although several embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel devices, methods, and programs described in this specification can be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications can be made to the forms of the devices, methods, and programs described in this specification without departing from the spirit of the invention. [Explanation of symbols]

[0074] 1: Reinforcement learning assistance system, 100: Reinforcement learning device, 200: Reinforcement learning assistance device, 217: Inference unit, 220: Prediction unit, 230: Creation unit (generation unit), 240: Display control unit, 250: Display unit, 260: Operation unit, L10, L20: Test scenario, L18, L23, L24: Learning scenario.

Claims

1. a generation unit that generates a learning scenario that serves as a unit of learning data for a learning model; a prediction unit that predicts the convergence of the learning model trained using the learning scenario with respect to a test scenario that corresponds to the learning scenario and is different from the learning scenario; A reinforcement learning assist device comprising:

2. a display control unit that displays the learning scenario on a display unit; an operation unit capable of performing an operation to change the shape of the learning scenario displayed on the display unit; Further provided with The reinforcement learning assistance device according to claim 1 , wherein the prediction unit predicts convergence of the learning model using the changed learning scenario with respect to the test scenario.

3. the display control unit causes the display unit to display the test scenario; The reinforcement learning assistance device according to claim 2 , wherein the operation unit is capable of performing an operation to change at least one of a period and a shape of the test scenario displayed on the display unit.

4. The reinforcement learning assistance device according to claim 3 , wherein the prediction unit predicts reproducibility of the learning model for the test scenario as the convergence.

5. The reinforcement learning assist device according to claim 4 , wherein the prediction unit predicts, as the convergence, the number of learning times at which a predetermined convergence condition of the learning model is reached.

6. the learning scenario and the test scenario are time-series control value data, the prediction unit predicts the convergence of the learning model using an error between a control value output in a time series by a control object controlled by the learning model and a control value in a time series of the test scenario; 6. The reinforcement learning assistance device according to claim 5, wherein the reproducibility increases as the error decreases.

7. an evaluation parameter acquisition unit that acquires the number of learning times during learning of the learning model using the learning scenario and parameters of the learning model corresponding to the number of learning times; an inference unit that generates an inference model corresponding to the learning model using the parameters during the learning, The reinforcement learning assistance device according to claim 6 , wherein the prediction unit predicts the convergence of the learning model using an output value of the inference model for the test scenario.

8. According to an execution learning count, which is the number of learning times at which learning of the learning model is completed, and an evaluation period, which is the interval between the number of learning times until the execution learning count is completed, the inference unit generates a plurality of inference models corresponding to the learning model until the number of execution learning times is completed; The reinforcement learning assistance device according to claim 7 , wherein the prediction unit predicts the convergence of the learning model using output values ​​of the plurality of inference models for the test scenario.

9. The reinforcement learning assistance device according to claim 8 , wherein the generation unit performs sampling at a sampling period set in the learning scenario and regenerates the data of the learning scenario.

10. The reinforcement learning assistance device according to claim 9 , wherein the display control unit causes the display unit to display information about convergence with respect to the test scenario in the form of a chart.

11. The reinforcement learning assistance device of claim 9, wherein the display control unit displays, on the display unit, an average error or a maximum error between the output value of the inference model for the test scenario and the value of the test scenario as the evaluation result for each evaluation period.

12. The reinforcement learning assistance device according to claim 11, wherein the display control unit displays time series data of the test scenario and time series data of the error between the output value of the inference model for the test scenario and the value of the test scenario on the display unit.

13. The operation unit includes: The reinforcement learning assistance device according to claim 12 , wherein an operation signal corresponding to a position on the screen of the display unit can be input.

14. The reinforcement learning assistance device of claim 13, wherein the operation unit is capable of inputting at least one of the drawing of the learning scenario to be displayed on the display unit, the number of times the learning is executed, the evaluation period, the sampling period of the learning scenario, the time for one period of the learning scenario, and the allowable error conditions used to determine convergence in the prediction unit by an operation signal corresponding to a screen position of the display unit.

15. The reinforcement learning assistance device according to claim 1; a reinforcement learning device that learns the learning model using the learning scenario; Equipped with A reinforcement learning assistance system, wherein the learning model uses the learning scenario to learn an action for a current state.

16. A reinforcement learning assistance method executed by a reinforcement learning assistance device, comprising: a generation step in which a generation unit generates a learning scenario that serves as a unit of learning data for a learning model; a prediction step in which a prediction unit predicts the convergence of the learning model trained using the learning scenario with respect to a test scenario that corresponds to the learning scenario and is different from the learning scenario; A reinforcement learning assistance method comprising:

Images