Processing system and processing method

Abstract

To smoothly introduce a robot to a work site.SOLUTION: A processing system 1 determines behavior using a behavioral model for determining behavior in accordance with a state. The behavioral model includes a first behavioral model π1 and a second behavioral model π2 that includes at least a simulation model that simulates behavior of a worker 30. The processing system 1 includes: a first determination unit which determines behavior using the first behavioral model π1; a second determination unit which determines behavior by using the second behavioral model π2; and a first learning unit which performs learning of the first behavioral model π1 by reinforcement learning on the basis of the behavior determined by using the second behavioral model π2.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a processing system and a processing method.

Background Art

[0002] In recent years, various technologies related to artificial intelligence (AI) using an action model for determining actions according to a state have been proposed. For example, Patent Document 1 discloses a technology related to learning of an action model by reinforcement learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, with the development of artificial intelligence, it is assumed that some of the operations performed by workers at work sites such as factory production sites will be automated by robots. Here, when introducing a robot to a work site, a situation where a worker and a robot perform a joint operation occurs. In such a situation, it is desired to smoothly introduce a robot to the work site.

[0005] Therefore, in view of such problems, an object of the present invention is to provide a processing system and a processing method capable of smoothly introducing a robot to a work site.

Means for Solving the Problems

[0006] To solve the above problems, the processing system is a processing system that determines an action using an action model for determining an action according to a state, wherein the action model includes a first action model and a second action model which includes at least an imitation model that imitates the actions of a worker, and the processing system comprises a first decision unit that determines an action using the first action model, a second decision unit that determines an action using the second action model, and a first learning unit that learns the first action model by reinforcement learning based on the action determined using the second action model.

[0007] To solve the above problem, the processing method is a method for determining an action using an action model for determining an action according to a state, wherein the action model includes a first action model and a second action model which includes at least an imitation model that imitates the actions of a worker, and includes a first step of determining an action using the first action model, a second step of determining an action using the second action model, and a third step of learning the first action model by reinforcement learning based on the action determined in the second step using the second action model. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to smoothly introduce robots into the work site. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing the general configuration of a processing system according to an embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the functional configuration of the first processing apparatus according to an embodiment of the present invention. [Figure 3] This is a block diagram showing an example of the functional configuration of the second processing apparatus according to an embodiment of the present invention. [Figure 4] This flowchart shows an example of the overall flow of a process performed by a first robot, a second robot, and a worker according to an embodiment of the present invention. [Figure 5] This figure shows the entity that executes the on-site work process according to an embodiment of the present invention. [Figure 6] This figure shows the entity that executes the learning process according to an embodiment of the present invention. [Figure 7] This flowchart shows an example of the processing flow in the learning process according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. The dimensions, materials, and other specific numerical values ​​shown in these embodiments are merely illustrative to facilitate understanding of the invention and do not limit the present invention unless otherwise specified. In this specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals to avoid redundant explanations, and elements not directly related to the present invention are omitted from the illustrations.

[0011] <Configuration of the processing system> The configuration of the processing system 1 according to an embodiment of the present invention will be described with reference to Figures 1 to 3.

[0012] Figure 1 is a schematic diagram showing the general configuration of processing system 1.

[0013] As shown in Figure 1, the processing system 1 includes a first robot 10 and a second robot 20. The processing system 1 is used in workplaces such as factory production sites. Below, we will describe an example of how the processing system 1 is used in a production process at a factory production site. A production process is a process of producing products. However, the workplace where the processing system 1 is used may be a workplace other than the production site where products are produced (for example, a site where products are inspected).

[0014] The first robot 10 and the second robot 20 are robots that determine and execute actions using artificial intelligence. Specifically, the first robot 10 and the second robot 20 have an action model for determining actions according to the state, and determine and execute actions using the action model. The first robot 10 includes a first processing device 11 that performs various processes related to the first robot 10. The first processing device 11 determines the action of the first robot 10 using the first action model π1, and causes the first robot 10 to execute the determined action. The second robot 20 includes a second processing device 21 that performs various processes related to the second robot 20. The second processing device 21 determines the action of the second robot 20 using the second action model π2, and causes the second robot 20 to execute the determined action

[0015] In the production process where the processing system 1 is utilized, the product is produced by the joint work of the first robot 10 and the worker 30. The first robot 10 executes the work related to the production of the product. In the example of FIG. 1, an example where the first robot 10 is a robot arm is shown. However, the first robot 10 may be a robot other than a robot arm

[0016] The second robot 20 is a robot used for learning the first action model π1 used by the first robot 10. The second robot 20 is capable of imitating the actions of the worker 30. In the example of FIG. 1, an example where the second robot 20 is a humanoid robot is shown. However, the second robot 20 may be a robot other than a humanoid robot

[0017] FIG. 2 is a block diagram showing an example of the functional configuration of the first processing device 11

[0018] The first processing device 11 includes a CPU (Central Processing Unit) which is an arithmetic processing device, a ROM (Read Only Memory) which is a storage element that stores programs, arithmetic parameters, etc. used by the CPU, and a RAM (Random Access Memory) which is a storage element that temporarily stores parameters that change as appropriate during the execution of the CPU, etc

[0019] As shown in FIG. 2, the first processing device 11 includes, for example, a first acquisition unit 11a, a first determination unit 11b, a first control unit 11c, a first learning unit 11d, and a first storage unit 11e.

[0020] The first acquisition unit 11a acquires various types of information necessary for controlling the first robot 1️⃣ As described above, the actions of the first robot 10 are determined using the first action model π1. The first acquisition unit 11a acquires information necessary for determining an action using the first action model π1.

[0021] The first determination unit 11b determines an action to be executed by the first robot 10. Specifically, the first determination unit 11b determines the action of the first robot 10 using the first action model π1. The first action model π1 is stored in the first storage unit 11e and is an action model learned by reinforcement learning. The first action model π1 is, for example, a function that outputs an evaluation index (recommendation degree) of the input action when a pair of a state and an action is input.

[0022] The information input as a state to the first action model π1 includes, for example, information related to the production site (e.g., information indicating the type of work or information about the materials used in the work), information related to the first robot 10 (e.g., information indicating the position and orientation of the first robot 10), or information related to the worker 30 (e.g., information indicating the position and orientation of the worker 30). Such information is acquired by the first acquisition unit 11a. The first acquisition unit 11a can acquire information related to the production site and information related to the worker 30 based on, for example, an image captured by a camera provided on the first robot 10. Also, the first acquisition unit 11a can acquire information related to the first robot 10 based on, for example, the output results of various sensors provided on the first robot 10.

[0023] For example, once a state is determined, several candidate actions are determined. The first decision unit 11b can determine the action of the first robot 10 by comparing the evaluation indices obtained by inputting each of these candidate actions into the first action model π1. Specifically, the first decision unit 11b determines that the action that yields the highest evaluation index is the action that the first robot 10 should perform.

[0024] The first control unit 11c causes the first robot 10 to execute the action determined by the first decision unit 11b using the first action model π1. For example, the first control unit 11c can control the first robot 10 by controlling actuators such as motors provided on the first robot 10.

[0025] The first learning unit 11d performs training on the first behavioral model π1. Details of the training of the first behavioral model π1 by the first learning unit 11d will be described later.

[0026] The first memory unit 11e stores various information necessary for controlling the first robot 10. Specifically, the first memory unit 11e stores the first behavior model π1. The first behavior model π1 stored in the first memory unit 11e is updated according to the learning performed by the first learning unit 11d.

[0027] Figure 3 is a block diagram showing an example of the functional configuration of the second processing unit 21.

[0028] The second processing unit 21 includes a CPU, which is an arithmetic processing unit; a ROM, which is a memory element that stores programs and arithmetic parameters used by the CPU; and a RAM, which is a memory element that temporarily stores parameters that change as appropriate during the execution of the CPU.

[0029] As shown in Figure 3, the second processing unit 21 includes, for example, a second acquisition unit 21a, a second determination unit 21b, a second control unit 21c, a second learning unit 21d, a third learning unit 21e, and a second storage unit 21f.

[0030] The second acquisition unit 21a acquires various information necessary for controlling the second robot 20. As described above, the actions of the second robot 20 are determined using the second action model π2. The second acquisition unit 21a acquires information necessary for determining actions using the second action model π2.

[0031] The second decision unit 21b determines the action to be performed by the second robot 20. Specifically, the second decision unit 21b determines the action of the second robot 20 using the second action model π2. The second action model π2 is stored in the second memory unit 21f and includes an imitation model π21 that imitates the actions of the worker 30 and a non-imitation model π22 for determining actions without relying on the actions of the worker 30. The imitation model π21 is an action model learned through imitation learning. The non-imitation model π22 is an action model learned through reinforcement learning.

[0032] The second behavioral model π2 is specifically generated by weighted addition of the imitation model π21 and the non-imitation model π22. As shown in Figure 3, the second behavioral model π2 can be expressed by the formula απ21 + (1-α)π22, for example. Here, the weight coefficient α takes a value between 0 and 1, and the closer the weight coefficient α is to 1, the closer the second behavioral model π2 is to the imitation model π21. Below, we will describe an example in which the second behavioral model π2 includes both the imitation model π21 and the non-imitation model π22. However, as will be discussed later, the second behavioral model π2 only needs to include at least the imitation model π21.

[0033] The imitation model π21 and the non-imitation model π22 are functions that, for example, similar to the first action model π1, take a state-action pair as input and output an evaluation index (recommendation level) for the input action. The second decision unit 21b can determine the action of the second robot 20 by inputting candidate actions corresponding to the state into the second action model π2 and comparing the evaluation indices obtained. Specifically, the second decision unit 21b determines the action that will be performed by the second robot 20 to be the action that yields the highest evaluation index. The information input as a state into the second action model π2 is the same as the information input as a state into the first action model π1, and is acquired by the second acquisition unit 21a in the same way as the acquisition method by the first acquisition unit 11a.

[0034] The second control unit 21c causes the second robot 20 to execute the action determined by the second decision unit 21b using the second action model π2. For example, the second control unit 21c can control the second robot 20 by controlling actuators such as motors provided on the second robot 20.

[0035] The second learning unit 21d trains the imitation model π21 in the second behavior model π2. Details of the training of the imitation model π21 by the second learning unit 21d will be described later.

[0036] The third learning unit 21e trains the non-imitative model π22 in the second behavioral model π2. Details of the training of the non-imitative model π22 by the third learning unit 21e will be described later.

[0037] The second memory unit 21f stores various information necessary for controlling the second robot 20. Specifically, the second memory unit 21f stores the second behavior model π2. The second behavior model π2 stored in the second memory unit 21f is updated according to the learning performed by the second learning unit 21d and the third learning unit 21e.

[0038] <Processing System Operation> The operation of the processing system 1 according to an embodiment of the present invention will be described with reference to Figures 4 to 7.

[0039] As described above, processing system 1 can be used, for example, in the production process at a factory. Below, the overall flow of the process in such an example will be explained with reference to Figures 4 to 6, and then the details of the learning process, which will be described later, will be explained with reference to Figure 7.

[0040] Figure 4 is a flowchart showing an example of the overall flow of a process performed by the first robot 10, the second robot 20, and the worker 30. Step S101 in Figure 4 corresponds to the start of the processing flow shown in Figure 4.

[0041] When the processing flow shown in Figure 4 is initiated, in step S102, the on-site work process, in which the actual product is produced at the production site, is carried out. The on-site work process is carried out, for example, during the daytime. Figure 5 is a diagram showing the entities that carry out the on-site work process. As shown in Figure 5, the on-site work process involves collaborative work between the first robot 10 and the worker 30. Here, the first decision unit 11b of the first robot 10 makes a decision on an action using the first action model π1 when the worker 30 is working. The first robot 10 then executes the action decided using the first action model π1.

[0042] Following step S102 in Figure 4, step S103 involves a learning process in which the first robot 10 learns the first behavior model π1 and the second robot 20 learns the second behavior model π2. The learning process is performed, for example, at night. Figure 6 shows the entities that perform the learning process. As shown in Figure 6, the first robot 10 and the second robot 20 learn each behavior model during the learning process. Details of the learning process will be described later.

[0043] In the processing flow shown in Figure 4, after step S103, the process returns to step S102, and the field work process of step S102 and the learning process of step S103 are repeated. For example, one set of the field work process and learning process is performed in one day, and the next set of the field work process and learning process is performed the following day. As a result, collaborative work between the first robot 10 and the worker 30 is performed on a daily basis, and the first behavior model π1 and the second behavior model π2 are updated on a daily basis.

[0044] Figure 7 is a flowchart showing an example of the processing flow in the learning process. Step S201 in Figure 7 corresponds to the start of the processing flow shown in Figure 7. Step S205 in Figure 7 corresponds to the end of the processing flow shown in Figure 7. The processing flow shown in Figure 7 is an example of the processing flow performed in step S103 of the processing flow shown in Figure 4 described above.

[0045] When the processing flow shown in Figure 7 is initiated, in step S202, the second learning unit 21d of the second robot 20 learns the imitation model π21 in the second action model π2. The learning of the imitation model π21 is performed by the second robot 20.

[0046] In step S202, the second learning unit 21d learns the imitation model π21 through imitation learning based on the actions performed by the worker 30. For example, the actions performed by the worker 30 during the daytime field work process are recorded. In the record of the worker 30's actions, the actions performed by the worker 30 are linked to the state in which those actions were performed (for example, the type of work). Imitation learning of the imitation model π21 is performed using the record of the worker 30's actions obtained in this way. In the imitation learning of the imitation model π21, the second learning unit 21d has the second robot 20 perform various actions and calculates a reward for each action performed by the second robot 20. Here, the second learning unit 21d calculates the reward such that the closer (more similar) the action performed by the second robot 20 in a given state is to the action of the worker 30 corresponding to that state, the higher the reward. The second learning unit 21d performs this reward calculation for each state. The reward obtained in this way corresponds to the evaluation index (recommendation level) of the actions described above. The second learning unit 21d uses the relationship between the obtained state-behavior pairs and rewards to update the relationship between the input (state-behavior pairs) and the output (behavior evaluation index (recommendation level)) in the imitation model π21.

[0047] Following step S202, in step S203, the third learning unit 21e of the second robot 20 learns the non-imitative model π22 in the second behavior model π2. The learning of the non-imitative model π22 is performed by the first robot 10 and the second robot 20.

[0048] In step S203, the third learning unit 21e performs reinforcement learning to train the non-imitation model π22 based on the actions determined by the first decision unit 11b using the first action model π1. For example, in a simulated work environment that mimics an actual production site, the first robot 10 selects and executes actions using the first action model π1. In this situation, reinforcement learning of the non-imitation model π22 is performed. In the reinforcement learning of the non-imitation model π22, the third learning unit 21e causes the second robot 20 to perform various actions and calculates a reward for each action performed by the second robot 20. Here, the third learning unit 21e calculates the reward such that the more efficient the action performed by the second robot 20 is (for example, the shorter the work time or the higher the success rate), the higher the reward. The third learning unit 21e performs this reward calculation for each state. Then, the third learning unit 21e uses the relationship between the obtained state-behavior pairs and rewards to update the relationship between the input (state-behavior pairs) and the output (behavior evaluation index (recommendation level)) in the non-imitative model π22.

[0049] Following step S203, in step S204, the first learning unit 11d of the first robot 10 learns the first behavior model π1, and the processing flow shown in Figure 7 is completed. The learning of the first behavior model π1 is performed by the first robot 10 and the second robot 20.

[0050] In step S204, the first learning unit 11d learns the first action model π1 using reinforcement learning based on the actions determined by the second decision unit 21b using the second action model π2. For example, in a simulated work environment that mimics an actual production site, the second robot 20 selects and executes actions using the second action model π2 updated in steps S202 and S203. In this situation, reinforcement learning of the first action model π1 is performed. In the reinforcement learning of the first action model π1, the first learning unit 11d causes the first robot 10 to perform various actions and calculates a reward for each action performed by the first robot 10. Here, the first learning unit 11d calculates the reward such that the more efficient the action performed by the first robot 10 is (for example, the shorter the work time or the higher the success rate), the higher the reward. The first learning unit 11d performs this reward calculation for each state. Then, the first learning unit 11d uses the relationship between the obtained state-behavior pairs and rewards to update the relationship between the input (state-behavior pairs) and the output (behavior evaluation index (recommendation level)) in the first behavior model π1.

[0051] As explained above, in the processing system 1, the first learning unit 11d of the first robot 10 learns the first action model π1 through reinforcement learning based on the action determined by the second decision unit 21b using the second action model π2. In the above example, reinforcement learning of the first action model π1 is performed by the first robot 10 and the second robot 20. This makes it possible to learn the first action model π1 at night or when the worker 30 is not working.

[0052] Here, if the second behavioral model π2 were to be a behavioral model (for example, a non-imitative model π22) that does not include an imitation model π21 that mimics the actions of worker 30, and does not base its actions on those of worker 30, then the actions of the second robot 20 would deviate significantly from those of worker 30. Therefore, if the first behavioral model π1 of the first robot 10 is learned together with such a second robot 20, there is a risk that worker 30 may not be able to properly adapt to the actions of the first robot 10 determined using the learned first behavioral model π1.

[0053] On the other hand, in this embodiment, the second behavior model π2 includes an imitation model π21 that mimics the actions of the worker 30. This suppresses a large deviation between the actions of the second robot 20 and those of the worker 30. Therefore, it is possible to suppress situations in which the worker 30 cannot properly adapt to the actions of the first robot 10, which are determined using the first behavior model π1 after learning. Thus, according to this embodiment, a robot (the first robot 10 in the above example) can be smoothly introduced into the work site.

[0054] Furthermore, as described above, the first decision unit 11b of the first robot 10 makes decisions about actions using the first action model π1 when the worker 30 is performing work. Specifically, in the daytime field work process (i.e., step S102 in Figure 4), the first robot 10, which works together with the worker 30, executes actions determined using the first action model π1. As a result, the worker 30 performs their work while adapting to the actions of the first robot 10, which are determined and executed using the first action model π1 that was updated in the learning process of the previous day. Therefore, after learning the first action model π1, the worker 30 adapts to the actions of the first robot 10. In this way, in the processing system 1, the worker 30 and the first robot 10 grow by adapting to each other so that the overall efficiency of the system is improved.

[0055] Furthermore, as described above, the second learning unit 21d of the second robot 20 learns the imitation model π21 through imitation learning based on the actions performed by the worker 30. Specifically, in the nighttime learning process (i.e., step S103 in Figure 4), imitation learning of the imitation model π21 in the second action model π2 is performed using the record of the actions performed by the worker 30 in the daytime field work process (i.e., step S102 in Figure 4). As a result, the imitation model π21 can be updated to imitate the actions of the worker 30 after they have adapted to the actions of the first robot 10, which are determined and executed using the first action model π1 after learning. In this way, in the processing system 1, the second robot 20 also grows while adapting to the worker 30 and the first robot 10 so as to improve the efficiency of the entire system.

[0056] Furthermore, as described above, the third learning unit 21e of the second robot 20 learns the non-imitative model π22 by reinforcement learning based on the actions determined by the first decision unit 11b using the first action model π1. Specifically, in the nighttime learning process (i.e., step S103 in Figure 4), reinforcement learning of the non-imitative model π22 in the second action model π2 is performed together with the first robot 10, which uses the first action model π1 updated by the previous day's learning. As a result, the non-imitative model π22 can be updated to adapt to the actions of the first robot 10, which are determined and executed using the first action model π1 after learning.

[0057] In the above, an example was described in which the second behavioral model π2 includes an imitation model π21 and a non-imitation model π22. However, the second behavioral model π2 only needs to include at least the imitation model π21. For example, the second behavioral model π2 may be a behavioral model that imitates the actions of worker 30 (e.g., the imitation model π21) without including a non-imitation model π22 for determining actions without relying on the actions of worker 30. Even in this case, as in the example above, it is possible to suppress situations in which worker 30 cannot properly adapt to the actions of the first robot 10 determined using the learned first behavioral model π1, thereby enabling the smooth introduction of the robot into the work site. However, if the second behavioral model π2 consists only of the imitation model π21, there is a risk that learning to determine actions in situations that worker 30 has not experienced may not proceed smoothly during the learning of the second behavioral model π2. Therefore, it is preferable that the second behavioral model π2 includes a non-imitation model π22 in addition to the imitation model π21.

[0058] In the above example, reinforcement learning of the first behavioral model π1 is described in the nighttime learning process (i.e., step S103 in Figure 4). However, reinforcement learning of the first behavioral model π1 may also be performed in the daytime field work process (i.e., step S102 in Figure 4) in addition to the nighttime learning process.

[0059] In the above description, an example was given in which each functional unit of the first processing unit 11 (i.e., the first acquisition unit 11a, the first decision unit 11b, the first control unit 11c, the first learning unit 11d, and the first storage unit 11e) and each functional unit of the second processing unit 21 (i.e., the second acquisition unit 21a, the second decision unit 21b, the second control unit 21c, the second learning unit 21d, the third learning unit 21e, and the second storage unit 21f) are separated into different robots. However, each of the functional units described above may be contained within a single device. For example, the first robot 10 may have a simulator that internally calculates the processing performed by the second robot 20 described above. In this case, the second robot 20 may be omitted from the processing system 1.

[0060] <Effects of the processing system> The effects of the processing system 1 according to an embodiment of the present invention will be described below.

[0061] In processing system 1, the behavior model includes a first behavior model π1 and a second behavior model π2 which includes at least an imitation model π21 that imitates the behavior of worker 30. Processing system 1 comprises a first decision unit 11b that determines an action using the first behavior model π1, a second decision unit 21b that determines an action using the second behavior model π2, and a first learning unit 11d that learns the first behavior model π1 by reinforcement learning based on the action determined using the second behavior model π2. This allows learning of the first behavior model π1 to be performed at night or when worker 30 is not working. Here, since the second behavior model π2 includes the imitation model π21, it is suppressed that the behavior of the second robot 20 deviates significantly from the behavior of worker 30. Therefore, it is possible to suppress situations in which worker 30 cannot properly adapt to the behavior of the first robot 10 determined using the learned first behavior model π1. Therefore, according to this embodiment, a robot (the first robot 10 in the above example) can be smoothly introduced into the work site.

[0062] Preferably, the processing system 1 includes a second learning unit 21d that learns an imitation model π21 by imitation learning based on the actions performed by the worker 30. This allows the imitation model π21 to be updated to imitate the actions of the worker 30 after they have been adapted to the actions of the first robot 10, which are determined and executed using the first action model π1 after learning.

[0063] Preferably, in the processing system 1, the second behavior model π2 includes, in addition to the imitation model π21, a non-imitation model π22 for making decisions on actions without relying on the actions of the worker 30. This suppresses the possibility that the learning of the second behavior model π2 will not proceed smoothly when learning to make decisions on actions in situations that the worker 30 has not experienced.

[0064] Preferably, in processing system 1, the second behavioral model π2 is generated by weighted addition of the imitation model π21 and the non-imitation model π22. This allows for the appropriate generation of the second behavioral model π2, which includes both the imitation model π21 and the non-imitation model π22. Furthermore, by appropriately setting the weight coefficient α, adjustments can be made to bring the second behavioral model π2 closer to the imitation model π21, or to move it further away from the imitation model π21.

[0065] Preferably, the processing system 1 includes a third learning unit 21e that learns a non-imitative model π22 by reinforcement learning based on actions determined using the first action model π1. This allows the non-imitative model π22 to be updated to adapt to the actions of the first robot 10 determined and executed using the learned first action model π1.

[0066] Preferably, in the processing system 1, the first decision unit 11b makes an action decision using the first action model π1 while the worker 30 is performing the task. As a result, the worker 30 performs the task while adapting to the actions of the first robot 10, which are determined using the first action model π1 that has been updated through learning, in order to improve the overall efficiency of the system. Therefore, after learning the first action model π1, the worker 30 can adapt to the actions of the first robot 10 that are determined using the first action model π1.

[0067] Preferred embodiments of the present invention have been described above with reference to the attached drawings. However, it goes without saying that the present invention is not limited to the embodiments described above, and that various modifications or alterations within the scope of the claims also fall within the technical scope of the present invention.

[0068] For example, the processes described using flowcharts in this specification do not necessarily have to be executed in the order shown in the flowcharts. Some processing steps may be executed in parallel. Additional processing steps may be adopted, and some processing steps may be omitted.

[0069] Furthermore, for example, the series of control processes performed by the processing system 1 described above may be implemented using software, hardware, or a combination of software and hardware. The programs constituting the software are pre-stored in a storage medium provided inside or outside the information processing device, for example. [Explanation of symbols]

[0070] 1. Processing System 10. Robot No. 1 11. First Processing Unit 11a 1st acquisition part 11b 1st decision section 11c First Control Unit 11d First Learning Department 11e 1st memory section 20. Second Robot 21. Second Processing Unit 21a 2nd acquisition part 21b Second decision section 21c Second Control Unit 21d Second Learning Department 21e Third Learning Department 21f 2nd storage section 30 workers π1 First Behavioral Model π² Second Behavioral Model π21 imitation model π22 Non-imitation Model

Claims

1. A processing system (1) that determines an action using an action model for determining an action according to the state, The behavioral model includes a first behavioral model (π1) and a second behavioral model (π2) which includes at least an imitation model (π21) that imitates the behavior of the worker (30). The processing system (1) is A first decision unit (11b) that determines the action using the first behavior model (π1), A second decision unit (21b) that determines the action using the second behavior model (π2), A first learning unit (11d) performs reinforcement learning to learn the first behavior model (π1) based on the actions determined using the second behavior model (π2), Equipped with, The second behavioral model (π2) includes, in addition to the imitation model (π21), a non-imitation model (π22) for determining actions without relying on the actions of the worker (30). Processing system.

2. The second behavioral model (π2) is generated by weighting and adding the imitation model (π21) and the non-imitation model (π22). The processing system according to claim 1.

3. The system includes a third learning unit (21e) that performs reinforcement learning to learn the non-imitation model (π22) based on the actions determined using the first behavior model (π1). The processing system according to claim 1 or 2.

4. The system includes a second learning unit (21d) that learns the imitation model (π21) through imitation learning based on the actions performed by the worker (30), The processing system according to any one of claims 1 to 3.

5. The first decision unit (11b) makes a decision on the action of the worker (30) using the first action model (π1) while the worker (30) is performing the work. The processing system according to any one of claims 1 to 4.

6. A method for determining an action using an action model for determining an action according to the state, The behavioral model includes a first behavioral model (π1) and a second behavioral model (π2) which includes at least an imitation model (π21) that imitates the behavior of the worker (30). The first step is to determine the action using the first behavioral model (π1) described above, The second step involves determining the action using the aforementioned second behavioral model (π²), A third step involves learning the first behavior model (π1) through reinforcement learning based on the behavior determined in the second step using the second behavior model (π2), Includes, The second behavioral model (π2) includes, in addition to the imitation model (π21), a non-imitation model (π22) for determining actions without relying on the actions of the worker (30). Processing method.

Images