Control device, control method, and program

Abstract

The present invention provides a control device, control method, and program capable of improving the recognition accuracy and recognition speed of a machine learning model for selecting actions of a robot that performs active perception. [Solution] The control device comprises an environment control unit that dynamically controls the state of the environment around an agent that acts based on a policy output from a machine learning model into which training data has been input; a sensor control unit that dynamically controls the state of a sensor that generates training data according to the state of the environment; and an action control unit that dynamically controls the agent's actions. The environment control unit may dynamically control the state of the environment based on environment control parameters stored in the replay memory. The sensor control unit may dynamically control the state of the sensor based on sensor control parameters stored in the replay memory. The action control unit may dynamically control the agent's actions based on action control parameters stored in the replay memory.

Description

Technical Field

[0001] The present invention relates to a control device, a control method, and a program.

Background Art

[0002] The progress of robot technology is remarkable, and the scope of application of robot technology covers a wide range, including industry, medical care, and disaster response. A robot may maximize the obtained reward (effect) by actively selecting actions and perceiving the state of the environment (hereinafter referred to as "active perception"). Here, the action selected to perceive the state of the environment and the understanding of the state of the environment are closely related. For example, a vehicle robot may improve the appearance of an obstacle placed on the ground (sensor data obtained from a camera mounted on the vehicle robot) by selecting an action to change the position of the vehicle robot on the ground. The vehicle robot understands the position of the obstacle based on the improved appearance and travels on the ground while avoiding the obstacle.

[0003] For the purpose of improving the performance of an image recognition model (machine learning model), a method of dynamically selecting the perspective of an agent observing the environment and the attention area in a real image using reinforcement learning is disclosed in Non-Patent Document 1. In Non-Patent Document 1, the perspective of each camera arranged on a partial sphere is determined using an azimuth angle and an elevation angle. The agent selects the perspective with the highest information value (the perspective that maximizes the obtained reward) based on the state of the environment. Specifically, the agent predicts the spherical angle with respect to the current position of the agent. The agent selects the camera having the perspective with the azimuth angle and elevation angle closest to the predicted spherical angle. As a result, the camera with the perspective having the highest information value is selected, so that the recognition accuracy and recognition speed of the image recognition model are improved.

[0004] The method disclosed in Non-Patent Document 1 improves the recognition accuracy and speed of the image recognition model compared to conventional methods that rely on a static viewpoint and a generalized attention area. This enables a more adaptive and efficient image recognition system in applications such as estimating the agent's pose in three dimensions or semantic segmentation. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Aleksis Pirinen, "Reinforcement Learning for Active Visual Perception", [online], LUND UNIVERSITY, [Retrieved September 12, 2024], Internet<URL: https: / / lup.lub.lu.se / search / ws / files / 97743536 / aleksis_phd_thesis.pdf> [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] In offline reinforcement learning in a real-world environment, even if it is possible to freely select training data as explanatory variables input to the machine learning model, it is necessary to obtain the policy (target variable) output by the machine learning model in advance. However, in offline reinforcement learning in a real-world environment, the pose and parameters of the camera that generates the training data (real-world images) are limited. Therefore, there is a problem in that it is not possible to improve the recognition accuracy and recognition speed of the machine learning model equipped to the robot (agent).

[0007] In view of the above circumstances, the present invention aims to provide a control device, a control method, and a program that can improve the recognition accuracy and recognition speed of a machine learning model for selecting actions of a robot that performs active perception. [Means for solving the problem]

[0008] One aspect of the present invention is a control device comprising: an environment control unit that dynamically controls the state of the environment surrounding an agent that acts based on a policy output from a machine learning model into which training data has been input; a sensor control unit that dynamically controls the state of a sensor that sequentially generates the training data according to the state of the environment; and an action control unit that dynamically controls the actions of the agent.

[0009] One aspect of the present invention is a control method performed by the control device described above, comprising the steps of: dynamically controlling the state of the environment surrounding an agent that acts based on a policy output from a machine learning model into which learning data has been input; dynamically controlling the state of a sensor that sequentially generates the learning data according to the state of the environment; and dynamically controlling the actions of the agent.

[0010] One aspect of the present invention is a program for causing a computer to execute the following steps: a procedure for dynamically controlling the state of the environment surrounding an agent that acts based on a policy output from a machine learning model into which training data has been input; a procedure for dynamically controlling the state of a sensor that sequentially generates the training data according to the state of the environment; and a procedure for dynamically controlling the actions of the agent. [Effects of the Invention]

[0011] The present invention makes it possible to improve the recognition accuracy and recognition speed of machine learning models for selecting actions of robots that perform active perception. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of the configuration of the learning system in the first embodiment. [Figure 2] This flowchart shows an example of agent operation in the first embodiment. [Figure 3]This is a flowchart illustrating an example of the operation of the control device in the first embodiment. [Figure 4] This figure shows an example of the configuration of the learning system in the third embodiment. [Figure 5] This is a flowchart illustrating an example of the operation of the control device in the third embodiment. [Figure 6] This figure shows examples of the hardware configuration of the control device in each embodiment. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will be described in detail with reference to the drawings. (First Embodiment) Figure 1 shows an example configuration of the learning system 1a in the first embodiment. The learning system 1a is a system that performs reinforcement learning on a machine learning model (not shown) using learning data obtained by active perception as explanatory variables. The machine learning model includes a neural network such as a deep neural network.

[0014] The learning system 1a comprises an agent 2 and a control device 3a. Agent 2 is provided in the environment 4. Agent 2 comprises a memory 21, one or more sensors 22, a learning unit 23, a selection unit 24, and an action unit 25. The control device 3a comprises an environment control unit 31, a sensor control unit 32, and an action control unit 33.

[0015] Agent 2 is an agent that performs active perception, such as a robot. The robot could be, for example, a ground-based robot (e.g., a vehicle), a water-based robot (e.g., a submarine), or a flying robot (e.g., an aircraft).

[0016] The control device 3a is a device that controls the actions of the agent 2, the state of the sensor 22, and the state of the environment 4. The environment 4 may be a real environment (the real world) or a virtual environment created using computer graphics or the like. The environment 4 includes one or more objects 5. The object 5 is not limited to a specific type of object, but for example, is a vehicle, a lighting fixture, a sound source, or a blower.

[0017] The agent 2 acts based on a policy output from a machine learning model into which learning data (observation results of the state of the environment 4) is input. Here, the agent 2 performs reinforcement learning on a machine learning model (not shown) using sequentially generated learning data.

[0018] The memory 21 stores in advance a machine learning model having a neural network and a computer program. The memory 21 may include, for example, a replay memory. The memory 21 stores, as past control parameters, the environment control parameters used for controlling the state of the environment 4, the sensor control parameters used for controlling the state of the sensor 22, and the action control parameters used for controlling the actions of the agent 2, for example, in association with time information. Here, the memory 21 does not necessarily store the learning data and policies (actions) used in past reinforcement learning.

[0019] The sensor 22 is a sensing device that observes the state of the environment 4, and for example, is a camera. The sensor 22 sequentially generates learning data according to the state of the environment 4. For example, when the sensor 22 is a camera, the sensor 22 generates, as learning data, at least partial images of the environment 4 at a predetermined period. The object 5 may be imaged, for example, in the image of the environment 4 (learning data). The state of the sensor 22 (for example, the orientation) is dynamically controlled based on the sensor control parameters input from the sensor control unit 32.

[0020] The learning unit 23 acquires the machine learning model from the memory 21. The learning unit 23 acquires the sequentially generated learning data from the sensor 22. The learning unit 23 inputs the learning data (explanatory variable) into the machine learning model. The learning unit 23 acquires one or more policies (objective variables) from the machine learning model.

[0021] The selection unit 24 selects the action of the agent 2 that executes the active perception so as to maximize the reward (effect) obtained by the active perception by the agent 2 from among one or more policies. The reward is predefined using, for example, a function or the like.

[0022] The action unit 25 causes the selected action to be executed by the agent 2. For example, the action unit 25 moves the agent 2 in the direction based on the selected action. Also, the action unit 25 may dynamically actuate the agent 2 based on the action control parameter input from the action control unit 33. For example, the action unit 25 may move the agent 2 in the environment 4 in a random direction indicated by the action control parameter. For example, the action unit 25 may increase the moving speed of the agent 2 in the environment 4 based on the moving speed indicated by the action control parameter.

[0023] As described above, while the agent 2 dynamically acts based on at least one of the policy and the action control parameter, the agent 2 inputs the learning data acquired from the sensor 22 whose state is dynamically controlled based on the sensor control parameter into the machine learning model. Thereby, the agent 2 executes the active perception.

[0024] The environmental control unit 31 dynamically controls the state of the environment 4 around the agent 2. Here, the environmental control unit 31 outputs environmental control parameters used to control the state of the environment 4 to the environment 4. Here, the environmental control unit 31 may also output environmental control parameters to an object 5 (e.g., a vehicle, lighting fixture, sound source, or blower) placed in the environment 4. The environmental control parameters are parameters used to control the state of the environment 4. The environmental control parameters represent, for example, operating lighting, emitting sound, operating airflow, moving object 5, shaking object 5, placing a new object 5 in the environment 4, or affecting sensor 22. Affecting sensor 22 means, for example, changing the sensitivity of sensor 22, introducing noise into the training data, changing the latency of reading the training data, or causing training data to be lost. Object 5 may move in the environment 4 based on the environmental control parameters.

[0025] The environmental control unit 31 may store environmental control parameters in memory 21 (replay memory). The environmental control unit 31 may dynamically control the state of environment 4 based on the stored environmental control parameters (past environmental control parameters).

[0026] The sensor control unit 32 dynamically controls the state of the sensor 22. Here, the sensor control unit 32 outputs sensor control parameters used to control the state of the sensor 22 to the sensor 22. The sensor control parameters represent, for example, sensor parameters such as magnification and focal length. The sensor control parameters may also represent, for example, sensor parameters such as the position, orientation and attitude of the sensor 22.

[0027] The sensor control unit 32 may store the sensor control parameters in the memory 21 (replay memory). The sensor control unit 32 may dynamically control the state of the sensor 22 based on the stored sensor control parameters (past sensor control parameters).

[0028] The action control unit 33 dynamically controls the actions of agent 2 (the machine). Here, the action control unit 33 outputs action control parameters used to control the actions of agent 2 to the action unit 25. Action control parameters represent, for example, changing the attitude of agent 2, moving agent 2, or changing the movement speed of agent 2.

[0029] The behavior control unit 33 may store behavior control parameters in memory 21 (replay memory). The behavior control unit 33 may dynamically control the behavior of agent 2 based on the stored behavior control parameters (past behavior control parameters).

[0030] Next, we will explain an example of how the learning system 1a works. Figure 2 is a flowchart showing an example of the operation of agent 2 in the first embodiment. Sensor 22 sequentially generates learning data according to the state of environment 4. Sensor 22 may dynamically change its state based on sensor control parameters. The state of environment 4 may also be dynamically changed based on environment control parameters (step S101).

[0031] The learning unit 23 inputs the learning data into the machine learning model (step S102). The learning unit 23 obtains one or more policies from the machine learning model (step S103). The selection unit 24 selects an action for agent 2 to perform active perception from among the one or more policies (step S104). The action unit 25 causes agent 2 to perform the selected action. The action unit 25 may dynamically change agent 2's action based on action control parameters (step S105).

[0032] The action unit 25 determines whether or not to terminate the execution of the action. For example, if the action unit 25 receives a signal from the operation unit (not shown) to agent 2 instructing it to terminate, it determines to terminate the execution of the action (step S106).

[0033] If it is determined that the action should be continued (step S106: NO), the action unit 25 returns to step S101. If it is determined that the action should be terminated (step S106: YES), agent 2 terminates the operation illustrated in Figure 2.

[0034] Figure 3 is a flowchart showing an example of the operation of the control device 3a in the first embodiment. The environment control unit 31 dynamically controls the state of the environment 4 by outputting environment control parameters to the environment 4 (step S201). The sensor control unit 32 dynamically controls the state of the sensor 22 by outputting sensor control parameters to the sensor 22 (step S202). The behavior control unit 33 dynamically controls the behavior of agent 2 by outputting behavior control parameters to the behavior unit 25 (step S203).

[0035] The action control unit 33 determines whether or not to terminate the execution of the action. For example, if the action control unit 33 receives a signal to terminate the action from the control device 3a (not shown), it determines to terminate the execution of the action (step S204).

[0036] If it is determined that the action should be continued (step S204: NO), the action control unit 33 returns to step S201. If it is determined that the action should be terminated (step S204: YES), the control device 3a terminates the operation illustrated in Figure 3.

[0037] As described above, the environment control unit 31 dynamically controls the state of the environment 4 around agent 2, which acts based on the policy output from the machine learning model. The sensor control unit 32 dynamically controls the state of sensor 22, which sequentially generates learning data according to the state of environment 4. The behavior control unit 33 dynamically controls the behavior of agent 2.

[0038] Thus, in reinforcement learning, the dynamic control of the state of sensor 22, the actions of agent 2, and the state of environment 4 improves the quality and diversity of the training data (explanatory variables). This makes it possible to improve the recognition accuracy and speed of the machine learning model used to select actions for a robot that performs active perception.

[0039] Improving the quality and diversity of training data makes reinforcement learning more efficient. Furthermore, by using control parameters in real-world environments, the reproducibility of training data in those environments can be increased, making it possible to perform online reinforcement learning in real-world environments using highly reproducible training data.

[0040] Since the size of the control parameters stored in the replay memory is small, reinforcement learning can be performed with a small amount of memory. Furthermore, instead of storing the training data and policies (actions) used in past reinforcement learning, the replay memory used to perform reinforcement learning may also store environment control parameters for regenerating the training data used in past reinforcement learning, and policy control parameters for having agent 2 re-execute the policies used in past reinforcement learning.

[0041] Furthermore, by equipping the control device 3a with a separate machine learning model, reinforcement learning may also be performed on the control device 3a. The control device 3a may perform reinforcement learning on the machine learning model equipped in the control device 3a using a predetermined method.

[0042] (Second Embodiment) The closer the quality of the virtual environment image (computer graphics image) is to the quality of the real-world image (image captured in the real environment), the higher the rendering cost of the virtual environment image becomes. Therefore, it is difficult to dynamically generate images of the virtual environment in online reinforcement learning in a virtual environment. In other words, for a machine learning model to be reinforced using training data generated from virtual environment images, and for that machine learning model (the learning result in the virtual environment) to be usable in online reinforcement learning in the real environment, it is necessary to generate computer graphics images from the virtual environment that have the same quality as real-world images, which increases the rendering cost of the virtual environment image. For this reason, it is difficult to apply a machine learning model that has been reinforced using virtual environment images as training data to reinforcement learning in the real environment.

[0043] Therefore, the main difference between the second embodiment and the first embodiment is that predetermined intermediate data is generated as training data. The second embodiment will be explained focusing on the differences from the first embodiment.

[0044] In online reinforcement learning in a virtual environment, when sensor 22 sequentially generates training data based on sensing results of the virtual environment, sensor 22 (renderer) sequentially generates event data (intermediate data) of the virtual environment image as training data, instead of sequentially generating images of the virtual environment (rendered images of the virtual environment) as training data. In contrast, in online reinforcement learning in a real environment (inference stage), sensor 22 (camera) generates event data (intermediate data) of real-world images as training data from real-world images. Here, sensor 22 may include, for example, event-based vision sensors.

[0045] In online reinforcement learning in a virtual environment, when sensor 22 sequentially generates training data based on sensing results from the virtual environment, sensor 22 (renderer) may sequentially generate feature quantities (intermediate data) of images from the virtual environment as training data instead of sequentially generating images from the virtual environment as training data. In contrast, in online reinforcement learning in a real environment (inference stage), sensor 22 (camera) generates feature quantities (intermediate data) of real-world images as training data from real-world images. Here, sensor 22 may be equipped with a feature extractor (not shown) that extracts feature quantities from real-world images. Note that the types of feature quantities in real-world images and the types of feature quantities in images from the virtual environment are the same.

[0046] The feature extractor of sensor 22 generates features for real-world images so that the feature quality of the real-world images and the virtual images are approximately equivalent. Therefore, it is not necessary for the virtual environment images to be rendered to have the same image quality as the real-world images. The image quality of the virtual environment images only needs to be sufficiently guaranteed as intermediate data. For this reason, it is possible to generate virtual environment images at low cost. For example, if event vision data is generated as training data, regions in the virtual environment images that do not change over time do not need to be rendered at all.

[0047] As described above, sensor 22 sequentially generates intermediate data representing the state of environment 4 as training data. This makes it possible to improve the recognition accuracy and recognition speed of the machine learning model for selecting actions for a robot that performs active perception, at low cost. Intermediate data is sparser than images, etc. Therefore, it is possible to reduce the amount of memory required for training.

[0048] (Third embodiment) In the third embodiment, the main difference from the first and second embodiments is that the training data (additional training data) lacking in reinforcement learning is generated based on active perception. The third embodiment will be explained focusing on the differences from the first and second embodiments.

[0049] Figure 4 shows an example configuration of the learning system 1b in the third embodiment. The learning system 1b comprises an agent 2 and a control device 3b. The agent 2 is provided in the environment 4. The agent 2 comprises a memory 21, a sensor 22, a learning unit 23, a selection unit 24, and an action unit 25. The control device 3b comprises an environment control unit 31, a sensor control unit 32, an action control unit 33, and a decision unit 34.

[0050] When the sensor 22 sequentially generates training data based on a real or virtual environment, and the learning unit 23 performs reinforcement learning of a machine learning model using the training data, the decision unit 34 analyzes past training data stored in the memory 21. The decision unit 34 may also analyze the state of the machine learning model being trained by the learning unit 23.

[0051] The decision unit 34 determines a method for generating additional training data to be used for additional reinforcement learning of the machine learning model, based on the results of analyzing the training data that has been generated sequentially in the past. The decision unit 34 generates generation method data that represents the method for generating the additional training data.

[0052] Additional training data is, for example, training data that is lacking for reinforcement learning. For example, the decision unit 34 analyzes the characteristics of the lacking training data. The environment control unit 31 dynamically controls the state of environment 4 based on the analysis results so that the training data lacking for reinforcement learning (necessary additional training data) is generated. The sensor control unit 32 dynamically controls the state of sensor 22 based on the analysis results so that the training data lacking for reinforcement learning is generated. The action control unit 33 dynamically controls the action of agent 2 based on the analysis results so that the training data lacking for reinforcement learning is generated.

[0053] The relationship between the characteristics of the additional training data and the control parameters (for example, control parameters representing the orientation of sensor 22) is measured in advance so that training data missing for reinforcement learning is generated. Here, random control may be performed in a virtual environment and the distribution of the characteristics of the generated training data may be measured in advance. For example, the decision unit 34 determines the region (characteristics of the additional training data) that has not been used to generate training data in environment 4 by analyzing the distribution of past training data. Based on the analysis results of the distribution of past training data, the decision unit 34 determines that directing the sensitivity of sensor 22 to the region that has not been used to generate training data is the method for generating additional training data.

[0054] For example, the decision unit 34 may input predetermined test data as training data into a machine learning model and analyze the attributes of the training data that was incorrectly judged (for example, real-world images used as training data when an incorrect policy is output from the machine learning model). Based on the attribute analysis results, the decision unit 34 determines the method for generating the incorrectly judged training data as the method for generating additional training data.

[0055] For example, the decision unit 34 may evaluate the uncertainty of a machine learning model (e.g., prediction probability, entropy, and variance) by inputting test data or past training data into the machine learning model. Any active learning algorithm may be used to evaluate the uncertainty of the machine learning model. Based on the evaluation results, the decision unit 34 determines a method for generating training data with high uncertainty as a method for generating additional training data.

[0056] Next, we will explain an example of how learning system 1b works. Figure 5 is a flowchart showing an example of the operation of the control device 3b in the third embodiment. The decision unit 34 determines a method for generating additional training data to be used for additional reinforcement learning of the machine learning model, based on the sequentially generated training data. The decision unit 34 generates generation method data representing the method for generating additional training data.

[0057] The method for generating additional training data is not limited to a specific method. For example, the method for generating additional training data could be to direct the sensitivity of sensor 22 to an area in environment 4 that has not yet been sensed. Alternatively, the method for generating additional training data could be to direct the field of view of sensor 22 (camera) to the location of an object 5 in environment 4 that has not yet been imaged (step S301).

[0058] The environment control unit 31 dynamically controls the state of environment 4 by outputting, for example, environment control parameters to environment 4 based on the generation method data (step S302). As a result, additional learning data is sequentially generated by the sensor 22 according to the state of environment 4 (for example, the position of object 5) which has been controlled so that learning data lacking for reinforcement learning is generated.

[0059] The sensor control unit 32 dynamically controls the state of the sensor 22 (for example, the direction of sensitivity) by outputting, for example, sensor control parameters to the sensor 22 based on the generation method data (step S303). As a result, additional training data is sequentially generated by the sensor 22 according to the state of the sensor 22, which is controlled so that training data lacking for reinforcement learning is generated.

[0060] The action control unit 33 dynamically controls the actions of agent 2 by outputting, for example, action control parameters to the action unit 25 based on the generation method data (step S304). As a result, additional learning data is sequentially generated by the sensor 22 in accordance with the actions of agent 2 (for example, movement in the direction indicated by the action control parameters) that are controlled to generate learning data that is lacking for reinforcement learning.

[0061] The action control unit 33 determines whether or not to terminate the execution of the action. For example, if the action control unit 33 receives a signal to terminate the action from the control device 3b (not shown), it determines to terminate the execution of the action (step S305).

[0062] If it is determined that the action should be continued (step S305: NO), the action control unit 33 returns to step S301. If it is determined that the action should be terminated (step S305: YES), the control device 3b terminates the operation illustrated in Figure 5.

[0063] As described above, the decision unit 34 determines a method for generating additional training data to be used for additional reinforcement learning of the machine learning model, based on the sequentially generated training data. The decision unit 34 generates generation method data representing the method for generating additional training data. The environment control unit 31 dynamically controls the state of the environment 4 based on the generation method data. The sensor control unit 32 dynamically controls the state of the sensor 22 based on the generation method data. The behavior control unit 33 dynamically controls the behavior of agent 2 based on the generation method data.

[0064] This dynamically generates images of the virtual environment that are input to the machine learning model as training data, thereby improving the recognition accuracy and speed of the machine learning model for selecting actions for the robot performing active perception. Furthermore, it improves the efficiency of reinforcement learning compared to cases where the dynamic control of the state of environment 4, the dynamic control of the state of sensor 22, and the dynamic control of agent 2's actions are all controlled randomly.

[0065] (Example hardware configuration) Figure 6 shows examples of the hardware configuration of the control device in each embodiment. The control device 3 corresponds to the control device 3a in the first embodiment and the control device 3b in the second embodiment, respectively.

[0066] Some or all of the functions of the control device 3 are implemented as software by a processor 300, such as a CPU (Central Processing Unit), executing a program stored in a storage unit 301 having a non-volatile recording medium (non-temporary recording medium). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs (Read Only Memory), CD-ROMs (Compact Disc Read Only Memory), hard disks built into computer systems, and non-temporary recording media such as solid-state drives. The storage unit 301 may also be a replay memory. The communication unit 302 transmits the processing results from the control device 3 to an external device (not shown). The communication unit 302 may also receive the program via a communication line.

[0067] Some or all of the functional units of the control device 3 may be implemented using hardware including electronic circuits (or circuits) such as LSI (Large Scale Integrated Circuit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array).

[0068] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0069] The present invention is applicable to robots that perform active perception using sensing devices (sensors). [Explanation of Symbols]

[0070] 1a, 1b...Learning system, 2...Agent, 3, 3a, 3b...Control device, 4...Environment, 5...Object, 21...Memory, 22...Sensor, 23...Learning unit, 24...Selection unit, 25...Action unit, 31...Environment control unit, 31...Sensor control unit, 33...Action control unit

Claims

1. An environment control unit dynamically controls the state of the environment surrounding an agent that acts based on a policy output from a machine learning model that has been input with training data, A sensor control unit dynamically controls the state of a sensor that sequentially generates the learning data according to the state of the environment, The behavior control unit dynamically controls the agent's actions. A control device equipped with the following features.

2. The control device according to claim 1, wherein the environmental control unit stores environmental control parameters used to control the state of the environment in a replay memory, and dynamically controls the state of the environment based on the stored environmental control parameters.

3. The control device according to claim 1, wherein the sensor control unit stores sensor control parameters used to control the state of the sensor in a replay memory, and dynamically controls the state of the sensor based on the stored sensor control parameters.

4. The control device according to claim 1, wherein the behavior control unit stores behavior control parameters used to control the agent's behavior in a replay memory, and dynamically controls the agent's behavior based on the stored behavior control parameters.

5. The control device according to claim 1, wherein the sensor sequentially generates intermediate data representing the state of the environment as the learning data.

6. The system further comprises a determination unit that determines a method for generating additional training data used for additional reinforcement learning of the machine learning model based on the sequentially generated training data, and generates generation method data representing the method for generating the additional training data, The environmental control unit dynamically controls the state of the environment based on the generated method data. The sensor control unit dynamically controls the state of the sensor based on the generated method data. The control device according to claim 1, wherein the behavior control unit dynamically controls the behavior of the agent based on the generated method data.

7. A control method performed by a control device, A step in which the state of the environment surrounding an agent that acts based on a policy output from a machine learning model that has been input with training data is dynamically controlled, A step of dynamically controlling the state of the sensor that sequentially generates the learning data according to the state of the environment, The steps include dynamically controlling the agent's behavior and A control method including

8. On the computer, A procedure for dynamically controlling the state of the environment surrounding an agent that acts based on a policy output from a machine learning model that has been input with training data, A procedure for dynamically controlling the state of a sensor that sequentially generates the learning data according to the state of the environment, A procedure for dynamically controlling the actions of the aforementioned agent and A program to execute.

Images