Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program that enhance efficiency of machine learning.SOLUTION: In a system including an information processing device, a robot, and a sensor, an information processing device 10 includes: an acquisition unit 11 that acquires an expert trajectory for a specific task; a generator 12 that performs reinforcement learning on the basis of at least a first reward based on behavior information being identified as the expert trajectory, and a second reward based on the specific task being executed on the basis of the behavior information, and generates the behavior information on the basis of the acquired information and a result of the reinforcement learning; and a discriminator 13 that discriminates whether input information is the generated behavior information or the expert trajectory. The generator 12 performs reinforcement learning at a first point of time by setting a ratio of the second reward to the first reward as a first ratio, and performs reinforcement learning at a second point of time later than the first point of time by setting the ratio of the second reward to the first reward as a second ratio higher than the first ratio.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, a method called GAIL (Generative Adversarial Imitation Learning), which combines an imitation learning algorithm using inverse reinforcement learning with a generative adversarial network (GAN), has been attracting attention (see Non-Patent Document 1).

[0003] In addition, since inverse reinforcement learning is a method to estimate the reward function of the environment from the expert's action trajectory (expert data, Expert Trajectory), imitation learning can be performed even if rewards cannot be obtained from the environment. Imitation learning using inverse reinforcement learning requires solving two problems: finding the reward function from the expert's action trajectory, and finding the expert policy from the obtained reward function through reinforcement learning. On the other hand, GAIL uses the mechanism of GAN to find the expert policy from the expert's action trajectory.

[0004] Non-Patent Document 2 discloses a technique for machine learning control of a robot arm using GAIL. In Non-Patent Document 2, an image captured by a camera and information indicating the position and angular velocity of the joints of the robot arm are acquired. In addition, the reward γ gail and the reinforcement learning reward γ task A generator is trained using a hybrid reward γ, which is a combination of the above and the above as shown in the following equation (1). Note that the reward γ for imitation learning is gail is the reward based on being able to fool the discriminator. Also, the reinforcement learning reward γ taskis a reward based on the robot completing a task.

[0005] In addition, λ is a preset constant and is a value between 0 and 1. t is the input data to the generator, and a t is the output data from the generator. Note that when λ is 0, only reinforcement learning (RL) is used, and when λ is 1, normal GAIL (for example, as described in Non-Patent Document 1) is used. γ(s t ,a t )=λγ gail (s t ,a t )+(1-λ)γ task (s t ,a t ) · · · (1) [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Jonathan Ho and Stefano Ermon. "Generative adversarial imitation learning" NIPS, 2016. [Non-Patent Document 2] Yuke Zhu, et al. "Reinforcement and Imitation Learning for Diverse Visuomotor Skills" RSS, 2018 Summary of the Invention [Problem to be solved by the invention]

[0007] However, conventional techniques may not be able to efficiently execute machine learning, for example.

[0008] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that can improve the efficiency of machine learning. [Means for solving the problem]

[0009] In a first aspect of the present disclosure, an information processing device is provided that includes an acquisition unit that acquires information indicating an expert's behavior trajectory for a specific task, a generator that performs reinforcement learning based at least on a first reward based on identification of behavior information as information indicating the expert's behavior trajectory and a second reward based on execution of the specific task based on the behavior information, and generates behavior information based on the information acquired by the acquisition unit and a result of the reinforcement learning, and a discriminator that identifies whether input information is behavior information generated by the generator or information indicating the expert's behavior trajectory, wherein the generator performs the reinforcement learning at a first time point with a ratio of the second reward to the first reward set as a first ratio, and performs the reinforcement learning at a second time point after the first time point with the second reward to the first reward set as a second ratio higher than the first ratio.

[0010] In addition, a second aspect of the present disclosure provides an information processing method that performs reinforcement learning based at least on a process of acquiring information indicating an expert's behavior trajectory for a specific task, a first reward based on the behavior information being identified as information indicating the expert's behavior trajectory, and a second reward based on the specific task being performed based on the behavior information, and performs a process of generating behavior information based on the information acquired by the acquiring process and a result of the reinforcement learning, and a process of identifying whether input information is information generated by the generating process or information indicating the expert's behavior trajectory, wherein in the generating process, the reinforcement learning is performed at a first time point with a ratio of the second reward to the first reward set to a first ratio, and the reinforcement learning is performed at a second time point after the first time point with the second reward to the first reward set to a second ratio higher than the first ratio.

[0011] In addition, in a third aspect of the present disclosure, a program is provided that causes a computer to execute the following processes: a process of acquiring information indicating an expert's action trajectory for a specific task; a process of performing reinforcement learning based at least on a first reward based on identification of the action information as information indicating the expert's action trajectory and a second reward based on execution of the specific task based on the action information; a process of generating action information based on the information acquired by the acquiring process and a result of the reinforcement learning; and a process of identifying whether input information is information generated by the generating process or information indicating the expert's action trajectory, wherein in the generating process, the reinforcement learning is performed at a first time point with a ratio of the second reward to the first reward set as a first ratio, and the reinforcement learning is performed at a second time point after the first time point with the second reward to the first reward set as a second ratio higher than the first ratio. Effect of the Invention

[0012] According to one aspect, the efficiency of machine learning can be improved. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to an embodiment. [Figure 4] 11 is a flowchart illustrating an example of a learning process of the information processing device according to the embodiment. [Diagram 5] 11 is a flowchart illustrating an example of an inference process of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The principles of the present disclosure are described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the present disclosure. The disclosure described herein may be implemented in various ways other than those described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0015] <System configuration> A configuration of an information processing system 1 according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of a configuration of the information processing system 1 according to an embodiment. In the example of FIG. 1, the information processing system 1 has an information processing device 10, a robot 20, and a sensor 30. Note that the numbers of the information processing devices 10, the robot 20, and the sensors 30 are not limited to those shown in the example of FIG. Note that the information processing device 10 and the sensor 30 may be housed inside the housing of the robot 20. The information processing device 10, the robot 20, and the sensor 30 are connected to each other so as to be able to communicate wirelessly or by wire.

[0016] The information processing device 10 is a device that controls the robot 20 using machine learning. For example, the information processing device 10 acquires, by a sensor 30, the actions of a human or the like when performing a task (work) using a tool as information indicating an expert's action trajectory, and performs learning based on the acquired information. Then, the information processing device 10 causes the robot 20 to use the tool in the same way as a human or the like to perform the task.

[0017] The robot 20 is a robot that performs tasks using various tools with an arm or the like. The robot 20 may be a device that can perform tasks using tools, and the external shape is not limited. The robot 20 can be used for various purposes, such as for home use, exploration, and factory use. The sensor 30 is a sensor that measures the surroundings of the robot 20. The sensor 30 may be, for example, a camera or a LiDAR.

[0018] <Hardware configuration> Fig. 2 is a diagram showing an example of a hardware configuration of an information processing device 10 according to an embodiment. In the example of Fig. 2, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These units may be connected by a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0019] When the program 104 is executed by the processor 101, memory 102, etc. in cooperation, the computer 100 performs at least some processing of the embodiments of the present disclosure. The memory 102 may be of any type suitable for a local technology network. The memory 102 may be, as a non-limiting example, a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as a non-limiting example. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0020] The embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor or other computing device.

[0021] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0022] The program codes for carrying out the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus. When the program codes are executed by the processor or controller, the functions / operations in the flowcharts and / or the block diagrams they implement are performed. The program codes are executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0023] The program can be stored and provided to the computer using various types of non-transitory computer-readable media. The non-transitory computer-readable media include various types of tangible recording media. Examples of the non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, semiconductor memories, and the like. The magnetic recording media include, for example, flexible disks, magnetic tapes, hard disk drives, and the like. The magneto-optical recording media include, for example, magneto-optical disks, and the like. The optical disk media include, for example, Blu-ray disks, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), CD-RW (ReWritable), and the like. The semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (random access memories), and the like. The program may also be provided to the computer by various types of temporary computer-readable media. Examples of the temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0024] <Configuration> Next, the configuration of the information processing device 10 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the information processing device 10 according to the embodiment. In the example of Fig. 3, the information processing device 10 has an acquisition unit 11, a generator 12, a discriminator 13, and a control unit 14. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor 101 and a memory 102 of the information processing device 10.

[0025] The acquisition unit 11 acquires information indicating the action trajectory of the expert for a specific task and information indicating the environment related to the robot 20 from a storage device inside the information processing device 10 or an external device.

[0026] The generator 12 generates information indicating an action based on the information acquired by the acquisition unit 11 and the result of reinforcement learning. The generator 12 generates a reward γ based on a classification result by the classifier 13 when the classifier 13 outputs either the information indicating the expert's action trajectory acquired by the acquisition unit 11 or the information indicating an action generated based on the result of reinforcement learning. gail (an example of a “first reward”). The generator 12 also determines the reward γ task (An example of the “second remuneration”).

[0027] The generator 12 also generates the reward γ gail Reward for γ task The ratio of is set as the first ratio, and reinforcement learning is performed at the first time point, and the reward γ gail Reward for γ task is set to a second rate higher than the first rate, and reinforcement learning is performed at a second time point after the first time point.

[0028] The discriminator 13 discriminates whether the information input from the generator 12 is information generated by the generator 12 or information indicating the action trajectory of the expert acquired by the acquisition unit 11. The control unit 14 controls the robot 20 based on the information indicating the action generated by the generator 12.

[0029] <Processing> <<Learning Phase>> Next, an example of the learning process of the information processing device 10 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the learning process of the information processing device 10 according to the embodiment.

[0030] In step S101, the acquisition unit 11 of the information processing device 10 acquires information indicating the expert's behavior trajectory for a specific task. The specific task may be, for example, hammering a nail or scooping water with a cup. Here, the acquisition unit 11 may acquire, as the information indicating the expert's behavior trajectory, information indicating, for example, the position and posture of a human arm and a tool (for example, a hammer, a cup, etc.) at each point in time when the tool is used by a human. The information indicating the expert's behavior trajectory may be generated, for example, by analyzing an image captured by a sensor 30 that is a camera using a convolutional neural network (CNN). In addition, the information indicating the expert's behavior trajectory may be generated based on data measured by a sensor 30 attached to at least one of the human arm and the tool.

[0031] Next, the acquisition unit 11 of the information processing device 10 acquires information indicating the environment (environmental information, information indicating the behavioral trajectory of the robot 20) (step S102). The environmental information may be generated, for example, by analyzing an image captured by the sensor 30, which is a camera, using CNN. The environmental information may also be generated, for example, based on data measured by the sensor 30 provided (mounted) on at least one of the arm and the tool of the robot 20. The environmental information may include, for example, information indicating the position and posture of the tool. The environmental information may also include, for example, information indicating the position and angular velocity of the joint of the arm of the robot 20.

[0032] Next, the generator 12 of the information processing device 10 generates a hybrid reward γ s Here, the generator 12 determines (updates) the hybrid reward γ s may be calculated. Gamma s (s t ,a t )=αγ gail (s t ,a t )+(1-α)γ task(s t ,a t ) · · · (2)

[0033] Here, α can be a variable between 0 and 1. t is the environmental information, and a t is information indicating the behavior (behavioral information). Note that when α is 0, only normal reinforcement learning is performed, and when α is 1, only normal GAIL (imitation learning) is performed.

[0034] Reward γ gail is the reward for imitation learning by normal GAIL. Reward γ task (An example of a "second reward.") is a reinforcement learning reward based on the robot 20 performing (completing, succeeding) a specific task.

[0035] Reward γ for imitation learning using GAIL gail is the reward in normal GAIL. That is, the reward γ gail is a reward based on the fact that the behavioral information generated by the generator 12 has been identified (determined, judged, determined) as information indicating the behavioral trajectory of the expert acquired by the acquisition unit 11 (the fact that the classifier 13 has been deceived, and the similarity of the generated behavioral information to the behavioral trajectory of the expert).

[0036] The generator 12 may, for example, use input data s t and a, which is the output data from the generator 12 t The data of the combination of the above may be input to the classifier 13. Then, in step S107 described later, the classifier 13 may calculate (estimate, infer) a value (accuracy, reliability) of the similarity of the data to an expert's behavior trajectory. The generator 12 increases the reward γ gail The value of may be determined to be high.

[0037] In addition, the reward γ task is a reward based on the execution of a specific task by the robot 20. The control unit 14, for example,t The generator 12 controls the robot based on, for example, the behavior information a t When the position and posture of the tool used by the arm of the robot controlled based on the above-mentioned is changed (transitioned) from the position and posture at the start of the task in the expert's action trajectory to the position and posture at the end of the task, the reward γ task The value of may be a specific value other than 0.

[0038] The generator 12 may determine the value of α to be smaller as the learning progresses. This allows, for example, the influence of imitation learning by GAIL to be relatively large in the early stages of learning, and the influence of reinforcement learning due to the success of the task to be relatively large in the later stages of learning. Therefore, it is considered possible to improve the efficiency of machine learning by a learning process similar to that of a human, in which the robot learns by watching and imitating at first, and then fine-tunes itself by trial and error once it has learned to a certain extent. In this case, the generator 12 determines the reward γ gail Reward for γ task is the first rate, and at the second time point after the first time point, the reward γ gail Reward for γ task may be a second rate higher than the first rate.

[0039] In this case, the generator 12 may determine the first rate and the second rate according to, for example, at least one of the number of times the reinforcement learning in step S104 was performed and the performance of the reinforcement learning result. Here, the generator 12 may determine (specify, calculate) a value of the performance of the reinforcement learning result based on, for example, at least one of the time required for the robot 20 to execute a specific task, the power consumption required for the execution of the specific task, and the number of times information indicating the behavior related to the execution of the specific task was generated. In this case, the generator 12 may determine the value of the performance to be higher, for example, the shorter the time required from the start of the specific task by the robot 20 to its completion. Also, the generator 12 may determine the value of the performance to be higher, for example, the less power consumed for the robot 20 to execute the specific task. Also, the generator 12 may determine the value of the performance to be higher, for example, the fewer the number of pieces of behavior information generated before the robot 20 executes the specific task. The power consumption value of the robot 20 may be measured by the sensor 30 of the robot 20. In the example of the above formula (2), the reward γ of imitation learning by GAIL is gail and the reinforcement learning reward γ task Based on and hybrid reward γ s In the above example, the generator 12 determines the reward γ gail and the reinforcement learning reward γ task In addition to the above, we also use the rewards of other learning methods to create a hybrid reward γ s may be determined.

[0040] Next, the generator 12 of the information processing device 10 calculates the hybrid reward γ s (Step S104). Next, the generator 12 of the information processing device 10 generates the input data, which is the environmental information s t Action information for a t (Step S105) Here, the generator 12 generates the behavior information a tFor example, data for controlling a robot may be generated and output as the behavior information. In this case, the behavior information may include, for example, information indicating the angular velocity of each joint of the arm of the robot.

[0041] Next, the classifier 13 of the information processing device 10 uses the environmental information s t and behavioral information a t The value of the likelihood p of the expert's action trajectory for the data of the combination with is calculated (estimated, inferred) and output to the generator 12 (step S106).

[0042] Next, the classifier 13 of the information processing device 10 uses, for example, a neural network (NN) to classify the behavior information a t If the behavior information a is generated by the generator 12, the value of p is calculated as 0, and the behavior information a t If p is the expert's action trajectory, the value of p is calculated as 1 by supervised learning (step S107). In this way, similar to GAIL, the classifier 13 and the generator 12 can be trained to compete with each other by GAN. Here, the classifier 13 uses, for example, environmental information s t and behavioral information a t and behavioral information a t may be learned by supervised learning based on data of combinations with a correct answer label indicating whether or not the given feature was generated by the generator 12.

[0043] Next, the control unit 14 of the information processing device 10 operates the robot 20 based on the behavioral information generated by the generator 12 (step S108). Here, the control unit 14 may transmit a control command according to the behavioral information to the robot 20. Next, the generator 12 of the information processing device 10 determines whether or not to end the learning (step S109). Here, the generator 12 may determine to end the learning when, for example, at least one of the number of times a specific task has been performed by the robot 20, the number of times the reinforcement learning has been performed in step S104, and the performance as a result of the reinforcement learning is equal to or greater than a threshold value.

[0044] If it is determined that the learning should not be ended (NO in step S109), the process proceeds to step S102. On the other hand, if it is determined that the learning should be ended (YES in step S109), the learning process is ended.

[0045] <<Inference Phase>> Next, an example of the inference processing of the information processing device 10 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the inference processing of the information processing device 10 according to the embodiment.

[0046] In step S201, the control unit 14 of the information processing device 10 determines (recognizes) the content of a task to be executed by the robot 20. Here, the control unit 14 may determine the content of the task based on, for example, an input from a user such as a voice or a button operation. The control unit 14 may determine the content of the task such as, for example, driving a nail with a hammer or scooping water with a cup.

[0047] Next, the acquisition unit 11 of the information processing device 10 acquires environmental information (step S202). The process of step S202 may be the same as the process of step S102 in FIG.

[0048] Next, the generator 12 of the information processing device 10 generates the input data, that is, the environmental information s t Action information for a t (Step S203). The process of step S203 may be the same as the process of step S105 in FIG.

[0049] Next, the control unit 14 of the information processing device 10 operates the robot 20 based on the behavior information generated by the generator 12 (step S204). The process of step S204 may be the same as the process of step S108 in FIG.

[0050] <Modification> The information processing device 10 may be a device contained in one housing, but the information processing device 10 of the present disclosure is not limited to this. Each unit of the information processing device 10 may be realized by cloud computing configured by one or more computers, for example. Such information processing devices are also included as examples of the "information processing device" of the present disclosure.

[0051] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0052] 1. Information Processing Systems 10. Information processing device 11 Acquisition Department 12 Generator 13 Classifier 14 Control section 20. Robot 30 Sensors

Claims

1. an acquisition unit that acquires information indicating an expert's action trajectory for a specific task; a generator that performs reinforcement learning based on at least a first reward based on the fact that the behavioral information is identified as information indicating the behavioral trajectory of the expert and a second reward based on the fact that the specific task is executed based on the behavioral information, and generates behavioral information based on the information acquired by the acquisition unit and a result of the reinforcement learning; a classifier for identifying whether the input information is behavior information generated by the generator or information indicating a behavior trajectory of the expert, The generator performs the reinforcement learning at a first time point by setting a ratio of the second reward to the first reward as a first ratio, and performs the reinforcement learning at a second time point after the first time point by setting the ratio of the second reward to the first reward as a second ratio higher than the first ratio. Information processing device.

2. The generator determines the first rate and the second rate according to the number of times the reinforcement learning has been executed. The information processing device according to claim 1 .

3. The generator determines the first rate and the second rate depending on performance of the reinforcement learning result.

3. The information processing device according to claim 1 or 2.

4. The generator determines the performance value based on at least one of a time required for executing the specific task, a power consumption required for executing the specific task, and a number of times the behavior information required for executing the specific task has been generated. The information processing device according to claim 3 .

5. the expert's action trajectory for the specific task is an action trajectory for a task in which a human uses a specific tool; The behavior information includes information indicating an angular velocity of a joint of an arm of the robot. The information processing device according to claim 1 .

6. obtaining information indicative of an expert's action trajectory for a particular task; A process of performing reinforcement learning based on at least a first reward based on the fact that the behavioral information is identified as information indicating the behavioral trajectory of the expert and a second reward based on the fact that the specific task is executed based on the behavioral information, and generating behavioral information based on the information acquired by the acquiring process and a result of the reinforcement learning; A process of identifying whether the input information is information generated by a generation process or information indicating a behavioral trajectory of the expert is executed; In the generating process, the reinforcement learning is performed at a first time point by setting a ratio of the second reward to the first reward as a first ratio, and the reinforcement learning is performed at a second time point after the first time point by setting the second reward to the first reward as a second ratio higher than the first ratio. Information processing methods.

7. obtaining information indicative of an expert's action trajectory for a particular task; A process of performing reinforcement learning based on at least a first reward based on the fact that the behavioral information is identified as information indicating the behavioral trajectory of the expert and a second reward based on the fact that the specific task is executed based on the behavioral information, and generating behavioral information based on the information acquired by the acquiring process and a result of the reinforcement learning; a process of identifying whether the input information is information generated by a generating process or information indicating a behavior trajectory of the expert; In the generating process, the reinforcement learning is performed at a first time point by setting a ratio of the second reward to the first reward as a first ratio, and the reinforcement learning is performed at a second time point after the first time point by setting the second reward to the first reward as a second ratio higher than the first ratio. program.

Images