Machine learning device for assisting an operator's work, and machine learning method

Abstract

To provide a robot system for assisting a worker to allow the worker to perform work more smoothly.SOLUTION: A robot system 10 includes: a robot 12; a detection device 14 configured to detect motion of a worker when the worker performs predetermined work; an end determination section 30 configured to determine whether the work is ended or not on the basis of detection data of the detection device 14; and a robot controller 28 configured to cause the robot 12 to carry out an article-feed operation for transporting an article for the work to a predetermined position to feed the article to the worker or an article-fetch operation for fetching the article used in the work and transporting the article to a predetermined storage location when the end determination section 30 determines that the work is ended.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a robot system for assisting a worker, a method for controlling a robot, a machine learning device, and a machine learning method.

Background Art

[0002] A robot system for assisting a worker is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need for a robot system that assists a worker so that the worker can work more smoothly.

Means for Solving the Problems

[0005] In one aspect of the present disclosure, a robot system for assisting a worker includes a robot, a detection device that detects the movement of the worker when the worker is performing a predetermined task, an end determination unit that determines whether the task has ended based on the detection data of the detection device, and when it is determined by the end determination unit that the task has ended, an article supply operation of transporting the article for the task to a predetermined position to supply the article to the worker, or an article collection operation of collecting the article used in the task and transporting it to a predetermined storage location, and a robot control unit that causes the robot to execute the operation.

[0006] In another aspect of the present disclosure, a method for controlling a robot that assists a worker's operation detects the movement of the worker by a detection device when the worker is performing a predetermined operation, determines whether the operation has ended based on the detection data of the detection device, and when it is determined that the operation has ended, an article supply operation of transporting the article for the operation to a predetermined position to supply the article to the worker, or an article collection operation of collecting the article used in the operation and transporting it to a predetermined storage location is executed by the robot.

[0007] In still another aspect of the present disclosure, a machine learning device that learns the timing at which a worker's operation ends includes a learning data acquisition unit that acquires, as a learning data set, the detection data of a detection device that detects the movement of the worker when the worker is performing a predetermined operation, and label information indicating the stage of the operation or the time required for the operation, and a learning unit that generates a learning model representing the correlation between the movement and the stage or time using the learning data set.

[0008] In still another aspect of the present disclosure, a machine learning method for learning the timing at which a worker's operation ends acquires, as a learning data set, the detection data of a detection device that detects the movement of the worker when the worker is performing a predetermined operation, and label information indicating the stage of the operation or the time required for the operation, and generates a learning model representing the correlation between the movement and the stage or time using the learning data set.

Advantages of the Invention

[0009] According to the present disclosure, since the robot control unit causes the robot to execute the article supply operation or the article collection operation at the timing when the operation ends, it is possible to assist the operation of the worker so that the worker can perform the operation smoothly.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the various embodiments described below, the same elements are denoted by the same reference numerals, and duplicate descriptions are omitted. First, with reference to FIGS. 1 and 2, a robot system 10 according to an embodiment will be described. The robot system 10 works in cooperation with an operator A and includes a robot 12, a detection device 14, and a control device 16.

[0012] In this embodiment, the robot 12 is a vertically articulated robot and includes a base portion 18, a swivel body 20, a robot arm 22, a wrist portion 24, and a robot hand 26. The base portion 18 is fixed on the floor of the work cell. The swivel body 20 is provided on the base portion 18 so as to be rotatable about a vertical axis.

[0013] The robot arm 22 includes a lower arm portion 23 rotatably attached to the swivel body 20 about a horizontal axis, and an upper arm portion 25 rotatably attached to the tip of the lower arm portion 23. The wrist portion 24 is connected to the tip of the upper arm portion 25 and rotatably supports the robot hand 26. The robot hand 26 has a plurality of finger portions 26a that can be opened and closed, and the finger portions 26a grip or release an article.

[0014] Each component of the robot 12 (base portion 18, swivel body 20, robot arm 22, wrist portion 24, and robot hand 26) incorporates a servo motor (not shown). The servo motor drives the movable components of the robot 12 (swivel body 20, robot arm 22, wrist portion 24) about their drive shafts under the command from the control device 16, thereby operating the robot 12.

[0015] A robot coordinate system C R is set for the robot 12. The robot coordinate system C R is a coordinate system for automatically controlling the movable components of the robot 12 (swivel body 20, robot arm 22, wrist portion 24) in a three-dimensional space. In this embodiment, the robot coordinate system C R is set such that its origin is located at the center of the base portion 18, its z-axis is arranged parallel to the vertical direction, and the swivel body 20 is rotated about the z-axis.

[0016] Based on the robot coordinate system C R the control device 16 generates commands to the respective servo motors incorporated in the robot 12, and operates the movable components of the robot 12 by the servo motors, so that the robot hand 26 is in the robot coordinate system C RPlace it at any position and orientation.

[0017] The detection device 14 detects the movement of the worker A when the worker A is performing a predetermined task. In the present embodiment, the detection device 14 includes an imaging element, an optical system such as a focus lens, and an image processing processor (e.g., GPU), and images the worker A during work along a predetermined line-of-sight direction VD, and based on the captured image, detects the movement of the worker A (so-called optical motion capture). Note that the work performed by the worker A will be described later.

[0018] The control device 16 controls the robot 12 and the detection device 14. The control device 16 is, for example, a computer having a processor (CPU, GPU, etc.), a memory (ROM, RAM, etc.), an input device (keyboard, mouse, touch sensor, etc.), and a display (liquid crystal display, organic EL display, etc.). The processor of the control device 16 is responsible for arithmetic processing for executing various functions described later.

[0019] Next, with reference to FIG. 3, the operation of the robot system 10 will be described. The flow shown in FIG. 3 starts when the control device 16 receives a work start command from an operator, a higher-level controller, or a computer program. In step S1, the control device 16 acquires information on the work performed by the worker A.

[0020] In the present embodiment, the worker A performs the first work, the second work, ··· the nth work, ··· the nth MAX work (n MAX is a positive integer) in order, and performs a total of n MAX types of work. In the nth work (n = 1 to n MAX ), it is assumed that the work is performed using the nth tool T n . Note that the nth tool T n may be different from each other, or at least two tools (the (n - 1)th tool T n-1 and the nth tool T n ) may be the same as each other.

[0021] In this step S1, the control device 16 acquires work information from an operator, a higher-level controller, or a computer program. The work information includes information specifying the type and order of the nth work and the type of the nth tool (e.g., identification number, etc.). The control device 16 determines, from this work information, the order in which the work is to be performed and the tool T to be supplied to worker A n and can recognize it.

[0022] Note that worker A may operate the input device of the control device 16 to input the work information. Alternatively, worker A may use voice recognition technology to input the work information. In this case, the control device 16 is further provided with a microphone capable of inputting voice, and a computer program for voice recognition technology is installed in advance.

[0023] In step S2, the control device 16 sets the number "n" identifying the nth work (i.e., the nth tool T n ) to "1". In step S3, the control device 16 causes the robot 12 to perform an article supply operation. Here, in the present embodiment, as shown in FIG. 4, a mounting table C is installed at a predetermined position within the work cell. On this mounting table C, there are provided a first jig D 1 and a second jig D 2 .

[0024] On the first jig D 1 , the tool T to be used by worker A for the next work is set by the robot 12. On the other hand, on the second jig D 2 , the tool T used for the work is manually set after worker A has performed the work. Also, a storage table E for storing the tool T is installed at a predetermined position within the work cell. The nth tool T used for the nth work n is stored stationary at a predetermined position on the storage table E. The robot coordinate system C of the first jig D 1 , the second jig D 2 , and the nth tool T on the storage table E n R ​The position data (coordinate data) in [the relevant context] is pre-stored in the memory of the control device 16.

[0025] In this step S3, the control device 16 causes the robot 12 to perform an article supply operation based on the position data in the robot coordinate system C 1 of the first tool T 1 and the first jig D R The robot 12 grips the first tool T stored on the storage table E under the command from the control device 16 with the robot hand 26, and sets the first tool T 1 on the first jig D 1 1 .

[0026] In step S4, the detection device 14 starts the operation of detecting the movement of the operator A. Specifically, the detection device 14 receives a command from the control device 16, continuously captures images of the operator A (in other words, shoots a video), and detects the movement of the operator A based on the captured images. In this way, the detection device 14 transmits detection data DD indicating the movement of the operator A (for example, image data continuously showing the movement of the operator A in time series) to the control device 16.

[0027] In step S5, the control device 16 determines whether the nth operation has ended based on the detection data DD of the detection device 14. Hereinafter, with reference to FIGS. 5 and 6, a method for determining the timing when the nth operation ends will be described. As an example, the nth operation can be defined as an operation in which the operator A uses the nth tool T n (FIG. 6) to fasten fasteners G (such as bolts) to the five fastening holes F 1 ~F 5 shown in FIG. 5 in the order of fastening hole F 1 →F 2 →F 3 →F 4 →F 5 .

[0028] In this case, the operator A first takes the nth tool T 1 set on the first jig D n with the hand H, and the nth tool T​n Use it to fasten the fastening holes F 1 ~F 5 Perform the nth operation of sequentially fastening the fasteners G to. Then, after the operator A finishes the nth operation (that is, after fastening the fastener G to the last fastening hole F 5 ), the nth tool T used n is set to the second jig D 2 .

[0029] The control device 16 analyzes the detection data DD acquired from the detection device 14 and determines whether the nth operation including a series of movements of sequentially fastening the fasteners G to the fastening holes F 1 ~F 5 is completed. As an example, from the acquired detection data DD, the control device 16 determines the body part of the operator A (in this case, the hand H) and the fastening hole F which is the last working location 5 continuously calculates the distance I (Fig. 6) between and monitors whether the distance I becomes equal to or less than a predetermined threshold value I th (I ≦ I th ).

[0030] When the control device 16 detects that I ≦ I th , it determines that the nth operation is completed. In this case, the position data of the fastening hole F R in the robot coordinate system C 5 is stored in advance in the memory of the control device 16. The control device 16 can calculate the distance I from the position data of the fastening hole F R in the robot coordinate system C 5 and the detection data DD of the detection device 14.

[0031] Alternatively, when a predetermined time (for example, 5 seconds) has elapsed since the control device 16 detected that the distance I has become equal to or less than the predetermined threshold value I th , it may determine that the nth operation is completed. Or, after the distance I has become equal to or less than the predetermined threshold value I th , when the distance I exceeds the threshold value I th again (I > I th ), the control device 16 may determine that the nth operation is completed.

[0032] As another example, the control device 16 may detect the fastener hole F 5 The reference motion pattern for fastening the fastener G to the last fastening hole F is stored in advance. 5 The detection data DD may be generated based on the detection data DD obtained by the detection device 14 detecting the movement of the worker A (or another worker) when the worker A (or another worker) is performing a reference operation of fastening the fastener G.

[0033] The control device 16 monitors whether the movement of the worker A indicated by the detection data DD acquired after the start of step S4 matches the reference movement pattern, and determines that the nth task is completed when the movement of the worker A matches the reference movement pattern. to fit It may be determined that the n-th task is completed when a predetermined time has elapsed since it was determined that the n-th task matches the n-th task.

[0034] By using the above method, the control device 16 can determine the timing when the nth task is completed. When the control device 16 determines that the nth task is completed (i.e., YES), it proceeds to step S6, whereas when the control device 16 determines that the nth task is not completed (i.e., NO), it loops to step S5. Thus, in this embodiment, the control device 16 functions as a completion determination unit 30 (FIG. 1) that determines whether the nth task is completed or not based on the detection data DD.

[0035] In step S6, the control device 16 increments the number "n" for identifying the nth job (the nth tool) by "1" (n=n+1). In step S7, the number "n" for identifying the nth job is incremented by "1" (n=n+1). MAX Exceeded (n>n MAX The control device 16 determines whether n>n MAX If it is determined that n is equal to or smaller than n (i.e., YES), the process proceeds to step S10. MAX If it is determined that the answer is YES (that is, NO), the process proceeds to step S8.

[0036] In step S8, the control device 16 causes the robot 12 to perform an article supply operation. Specifically, the control device 16 uses the nth tool T n and the first jig D 1 in the robot coordinate system C R to cause the robot 12 to perform an article supply operation, and the robot 12 grips the nth tool T n stored on the storage table E with the robot hand 26, and sets the nth tool T n to the first jig D 1 .

[0037] Thus, in the present embodiment, the control device 16 functions as a robot control unit 28 (FIG. 1) that causes the robot 12 to perform an article supply operation for supplying the nth tool T n for the nth operation to the worker A, and transports the nth tool T n to a predetermined position (the first jig D 1 ).

[0038] In step S9, the control device 16 functions as the robot control unit 28 and causes the robot 12 to perform an article collection operation. Specifically, the control device 16 controls the robot 12 based on the position data in the robot coordinate system C 2 of the second jig D R , and the robot 12 grips the (n - 1)th tool T 2 used in the immediately preceding (n - 1)th operation and set on the second jig D n-1 with the robot hand 26.

[0039] Next, the control device 16 controls the robot 12 based on the position data in the robot coordinate system C n-1 of the (n - 1)th tool T R to be set on the storage table E, and the robot 12 returns the (n - 1)th tool T n-1 gripped by the robot hand 26 to a predetermined position on the storage table E. For example, when n = 4 is set at the start of this step S9, the control device 16 causes the robot 12 to use the third tool T used in the third operation3 is recovered by the second jig D 2 and transported to a predetermined position on the storage table E.

[0040] Thus, the control device 16 causes the robot 12 to perform an article recovery operation of recovering the (n - 1)th tool T used in the immediately preceding n - 1th operation n-1 and transporting it to a predetermined storage location (storage table E). Before executing this step S9, the control device 16, based on the image captured by the detection device 14 (or another detection device), determines whether the second jig D n-1 is set on the (n - 1)th tool T 2 and may start step S9 when the second jig D n-1 is set on the (n - 1)th tool T 2 .

[0041] Alternatively, an article detection sensor (for example, a proximity switch) for detecting that the (n - 1)th tool T 2 is arranged on the second jig D n-1 is provided on the second jig D 2 , and the control device 16 determines whether the second jig D n-1 is set on the (n - 1)th tool T 2 based on the output signal of the article detection sensor. After step S9, the control device 16 returns to step S5 and loops steps S5 to S9 until it determines YES in step S7.

[0042] When it is determined YES in step S7, in step S10, the control device 16 functions as a robot control unit 28 and causes the robot 12 to perform an article recovery operation. Specifically, the control device 16 controls the robot 12 based on the position data in the robot coordinate system C 2 of the second jig D R , and the robot 12 grips, with the robot hand 26, the nth tool T 2 set on the second jig D MAX and used in the nth operation MAX . nMAX

[0043] Next, the control device 16 controls the robot 12 based on the position data in the robot coordinate system C of the nth tool T to be set on the storage table E. The robot 12 returns the nth tool T held by the robot hand 26 to a predetermined position on the storage table E. MAX of the tool T nMAX in the robot coordinate system C R Based on the position data, the robot 12 is controlled. The robot 12 returns the nth tool T held by the robot hand 26 to a predetermined position on the storage table E. MAX of the tool T nMAX to a predetermined position on the storage table E.

[0044] As described above, in this embodiment, when the control device 16 determines that the nth operation has ended, the control device 16 causes the robot 12 to execute the article supply operation (step S8). According to this configuration, when the worker A performs the (n + 1)th operation following the nth operation, the (n + 1)th tool T for the (n + 1)th operation is automatically set by the robot 12 on the first jig D. Thereby, it is possible to assist the worker A in smoothly performing the nth operation. n+1 is automatically set by the robot 12 on the first jig D 1 . Thereby, it is possible to assist the worker A in smoothly performing the nth operation.

[0045] Also, in this embodiment, when the control device 16 determines that the nth operation has ended, the control device 16 causes the robot 12 to execute the article collection operation (steps S9 and S10). According to this configuration, when the worker A completes the nth operation, the used nth tool T can be automatically collected by the robot 12. Thereby, it is possible to assist the work of the worker A so that the worker A can perform the work smoothly. n is automatically collected by the robot 12. Thereby, it is possible to assist the work of the worker A so that the worker A can perform the work smoothly.

[0046] Note that in step S5, the control device 16 may determine whether or not the nth operation has ended based on a learning model LM that represents the correlation between the movement of the worker A performing the nth operation and the stage of the nth operation or the time t required for the nth operation. Hereinafter, a machine learning device 50 that learns the learning model LM will be described with reference to FIG. 7.

[0047] The machine learning device 50 can be composed of, for example, a computer having a processor (CPU, GPU, etc.), a memory (ROM, RAM), an input device (keyboard, mouse, touch sensor, etc.), and a display (liquid crystal display, organic EL display, etc.), or software such as a learning algorithm executed by the computer.

[0048] The machine learning device 50 learns the timing at which the worker A finishes the nth task. The machine learning device 50 includes a learning data acquisition unit 52 and a learning unit 54. As one embodiment of the functions of the machine learning device 50, the learning data acquisition unit 52 uses the detection data DD of the detection device 14 described above and the label information LI indicating the stage (initial stage, middle stage, final stage, etc.) of the nth task to form a learning data set DS 1 and acquires it.

[0049] More specifically, when the worker A is performing the nth task, the detection device 14 captures an image of the worker A along the line of sight direction VD, and detects the movement of the worker A based on the captured image. The detection device 14 supplies detection data DD indicating the movement of the worker A (for example, image data continuously showing the movement of the worker A in time series) to the learning data acquisition unit 52.

[0050] Fig. 8 schematically shows the detection data DD continuously acquired by the detection device 14 m . The horizontal axis in Fig. 8 represents time t. Note that Fig. 8 illustrates the case where the detection device 14 detects a total of 300 pieces of detection data DD m (m = 1 to 300) from the start to the end of the nth task by the worker A.

[0051] The label information LI includes, for example, initial stage label information LI indicating that the nth task is in the "initial stage" 1 and middle stage label information LI indicating that the nth task is in the "middle stage" 2 and final stage label information LI indicating that the nth task is in the "final stage" 3 . Here, the detection data DD shown in Fig. 8 1 ~DD99 The aggregate is such that worker A shows the movement of fastening the first two fastening holes F 1 ~F 5 among those in FIG. 5. For example, the fastening holes F 1 and F 2 ).

[0052] Also, the aggregate of detection data DD 100 ~DD 199 is such that worker A shows the movement of fastening the next two fastening holes F 1 ~F 5 among those in FIG. 5. For example, the fastening holes F 3 and F 4 ). Also, the aggregate of detection data DD 200 ~DD 300 is such that worker A shows the movement of fastening the last one fastening hole F 1 ~F 5 among those in FIG. 5. For example, the fastening hole F 5 ).

[0053] In this case, the operator may attach the initial stage label information LI 1 ~DD 99 to the detection data DD, attach the mid-stage label information LI 1 to the detection data DD 100 ~DD 199 , and attach the final stage label information LI 2 to the detection data DD 200 ~DD 300 . When the operator attaches the label information LI, the display of the machine learning device 50 displays the detection data DD 3 (image data). m (image data).

[0054] Then, while visually observing the image of the detection data DD m displayed on the display, the operator operates the input device of the machine learning device 50 to attach to the detection data DD m the initial stage label information LI 1 indicating the progress stage of the nth operation, the mid-stage label information LI 2 , and the final stage label information LI3 Optionally assign any one of them.

[0055] In this way, the operator optionally assigns the label information LI to the detection data DD m in accordance with the movement of the worker A indicated by the detection data DD m to the detection data DD. The learning data acquisition unit 52 obtains the detection data DD m acquired from the detection device 14 and the label information LI input by the operator as the learning data set DS 1 and acquires it.

[0056] In the example shown in FIG. 8, for ease of understanding, the case where the detection device 14 detects a total of 300 detection data DD 1 ~DD 300 during the nth operation is described. However, the detection device 14 may detect any number of detection data DD m during the nth operation. The operator arbitrarily determines how many detection data DD m in the detected set of detection data DD m to which any type of label information LI 1 , LI 2 , or LI 3 is to be assigned.

[0057] In addition, the label information LI is not limited to the above-described initial-stage label information LI 1 , intermediate-stage label information LI 2 , and final-stage label information LI 3 . For example, as the final-stage label information LI 3 , a plurality of label information such as "final stage 1", "final stage 2", "final stage 3" may be set in chronological order. Similarly, a plurality of label information may be set as the initial-stage label information LI 1 or the intermediate-stage label information LI 2 . Also, only the final-stage label information LI 3 may be assigned to the detection data DD m .

[0058] The learning unit 54 uses the learning data set DS1 Using 1 , a learning model LM is generated that represents the correlation between the movements of worker A who is performing the nth task and the stage of the nth task (initial stage, intermediate stage, final stage). 1 (function). For example, the learning unit 54 generates the learning model LM by performing supervised learning. 1 In this case, worker A repeatedly tries the nth task, and the learning data acquisition unit 52 repeatedly acquires the learning data set DS 1 as teacher data every time worker A performs the nth task.

[0059] The learning unit 54 learns the learning model LM by identifying features that imply the correlation between the detection data DD m acquired as teacher data and the label information LI indicating the stage of the nth task (initial stage label information LI 1 , intermediate stage label information LI 2 , final stage label information LI 3 ). As such supervised learning, for example, algorithms such as a support vector machine (SVM) or a Gaussian mixture model (GMM) can be used. 1

[0060] Hereinafter, with reference to FIG. 9, the flow of the learning cycle performed by the machine learning device 50 will be described. In step S11, worker A performs the nth task, and the detection device 14 detects the movements of worker A during the nth task. In step S12, the learning data acquisition unit 52 acquires the learning data set DS m of the detection data DD 1 and the label information LI, and stores them in the memory of the machine learning device 50 in association with each other.

[0061] In step S13, the learning unit 54 uses the learning data set DS 1 acquired in step S12 to generate a learning model LM that represents the correlation between the movements of worker A during the execution of the nth task and the stage of the nth task. 1 And the flow returns to step S11.

[0062] ​ By executing such a learning cycle, the learning of the learning model LM 1 progresses, and the learning model LM 1 will be led to the optimal solution. According to the present embodiment, the learning model LM 1 that quantitatively represents the correlation between the movement of the worker A during the execution of the nth task and the stage of the nth task can be obtained automatically and with high accuracy.

[0063] Next, another form of the function of the machine learning device 50 will be described. In the present embodiment, the learning data acquisition unit 52 uses the above-described detection data DD m and the time t required for the nth task as the learning data set DS 2 to obtain. For example, the learning data acquisition unit 52 uses, as the learning data set DS 2 the detection data DD 1 ~DD 5 showing the movement of the worker A who fastens the first p (p is an integer of 4 or less) fastening holes F out of the fastening holes F 1 ~DD m_p and the time t from the time when the fastening of the pth fastening hole F is completed to the time when the nth task ends.

[0064] Specifically, the learning data acquisition unit 52 uses, as the learning data set DS 2 for example, the detection data DD 1 ~DD 2 showing the movement of the worker A who fastens the first two fastening holes F (fastening holes F 1 and F 99 ) and the time t from the time when the fastening of the second fastening hole F (fastening hole F 2 ) is completed (that is, the detection time of the detection data DD 99 ) to the time when the nth task ends (the detection time of the detection data DD 300 ).

[0065] The learning unit 54 uses the learning data set DS m of the detection data DD 2Using this, a learning model LM 2 (function) is generated to represent the correlation between the movements of worker A who is performing the nth task and the time t required for the nth task (for example, the time t from the point when fastening for the pth fastening hole F is completed until the nth task is completed).

[0066] Hereinafter, with reference to FIG. 9, the learning cycle flow of the learning model LM 2 will be described. The learning flow of the learning model LM 2 differs from the learning flow of the above-described learning model LM 1 in steps S12 and S13. Specifically, in step S12, the learning data acquisition unit 52 acquires a learning data set DS m of the detection data DD 2 and time t, and stores them in the memory of the machine learning device 50 in association with each other.

[0067] In step S13, the learning unit 54 uses the learning data set DS 2 acquired in step S12 to generate a learning model LM 2 representing the correlation between the movements of worker A who is performing the nth task and time t. For example, the learning unit 54 learns the learning model LM 2 by executing a supervised learning algorithm (SVM, GMM, etc.). Then, the flow returns to step S11.

[0068] By executing such a learning cycle, the learning of the learning model LM 2 progresses, and the learning model LM 2 will be led to the optimal solution. According to this embodiment, a learning model LM 2 that quantitatively represents the correlation between the movements of worker A who is performing the nth task and the time t required for the nth task can be obtained automatically and with high accuracy.

[0069] Note that the learning algorithm executed by the learning unit 54 is not limited to supervised learning, and known learning algorithms in machine learning, such as unsupervised learning, reinforcement learning, neural networks, etc., can be adopted. As an example, FIG. 10 schematically shows a model of a neuron. FIG. 11 schematically shows a model of a three-layer neural network configured by combining the neurons shown in FIG. 10. The neural network can be configured by, for example, an arithmetic device or a storage device that mimics the model of a neuron.

[0070] The neuron shown in FIG. 10 outputs a result y for a plurality of inputs x (inputs x1 to x3 as an example in the figure). Each input x (x1, x2, x3) is multiplied by a weight w (w1, w2, w3). The relationship between the input x and the result y can be expressed by Equation 1 below. Note that the input x, the result y, and the weight w are all vectors. Also, in Equation 1, θ is a bias, and f k is an activation function.

[0071]

Equation

[0072] In the three-layer neural network shown in FIG. 11, a plurality of inputs x (inputs x1 to input x3 as an example in the figure) are input from the left side, and a result y (results y1 to results y3 as an example in the figure) is output from the right side. In the illustrated example, each of the inputs x1, x2, x3 is multiplied by a corresponding weight (collectively represented by W1), and each of the individual inputs x1, x2, x3 is input to three neurons N11, N12, N13.

[0073] In FIG. 11, the outputs of each of the neurons N11 to N13 are collectively represented by Z1. Z1 can be regarded as a feature vector that extracts the feature amount of the input vector. In the illustrated example, each of the feature vectors Z1 is multiplied by a corresponding weight (collectively represented by W2), and each of the individual feature vectors Z1 is input to two neurons N21, N22. The feature vector Z1 represents the feature between the weight W1 and the weight W2.

[0074] In FIG. 11, the outputs of each of neurons N21 to N22 are collectively represented by Z2. Z2 can be regarded as a feature vector obtained by extracting the feature amounts of the feature vector Z1. In the illustrated example, weights corresponding to each of the feature vectors Z2 (collectively represented by W3) are multiplied, and each of the individual feature vectors Z2 is input to three neurons N31, N32, and N33.

[0075] The feature vector Z2 represents the features between the weights W2 and W3. Finally, neurons N31 to N33 output results y1 to y3, respectively. The machine learning device 50 takes the learning data set DS (DS 1 or DS 2 ) as input, and by performing operations of a multi-layer structure according to the neural network described above, a learning model LM (LM 1 or LM 2 ) can be learned.

[0076] The configuration of the machine learning device 50 described above can be described as a machine learning method (or software) executed by a computer's processor. This machine learning method is such that the processor, when worker A is performing the nth task, uses the detection data DD of the detection device 14 that detects the movement of worker A m and the label information LI indicating the stage of the nth task or the time t required for the nth task as the learning data set DS 1 or DS 2 , and uses this learning data set DS 1 or DS 2 to generate a learning model LM 1 or LM 2 representing the correlation between the movement of worker A and the stage or time t of the nth task.

[0077] Next, referring to FIGS. 2 and 12, a robot system 60 according to another embodiment will be described. The robot system 60 includes a robot 12, a detection device 14, and a control device 62. The control device 62 is a computer having a processor 64 (such as a CPU or GPU), a memory 66 (such as a ROM or RAM), an input device 68 (such as a keyboard, mouse, touch sensor, etc.), and a display 70 (such as a liquid crystal display or an organic EL display). The processor 64, the memory 66, the input device 68, and the display 70 are communicably connected to each other via a bus 72.

[0078] In the present embodiment, the machine learning device 50 is implemented in the control device 62 as hardware or software, and the processor 64 executes various operations for performing the functions of the machine learning device 50 while communicating with the memory 66, the input device 68, and the display 70. That is, in the present embodiment, the processor 64 functions as the learning data acquisition unit 52 and the learning unit 54. The memory 66 stores in advance a learning model LM (LM 1 or LM 2 ) learned by the machine learning device 50.

[0079] Next, referring to FIG. 3, the operation of the robot system 60 will be described. The processor 64 functions as the above-described robot control unit 28 and end determination unit 30, and executes the flow shown in FIG. 3 while communicating with the memory 66, the input device 68, and the display 70. Here, the operation flow of the robot system 60 is different from the operation flow of the above-described robot system 10 in the following processes.

[0080] Specifically, in step S4, the detection device 14 starts the operation of detecting the movement of worker A. At this time, the detection device 14 continuously images (shoots a video) of worker A under the same conditions as the above-mentioned step S11 executed during the learning stage of the learning model LM, and detects the movement of worker A. Specifically, the detection device 14 images worker A along the same line-of-sight direction VD as in the above-mentioned step S11 under the same shooting conditions (shutter speed, shooting speed, etc.). The detection device 14 sequentially transmits the continuously acquired detection data DD m ' to the processor 64.

[0081] In step S5, the processor 64 determines whether the nth operation has ended based on the detection data DD m ' of the detection device 14 and the learning model LM. Here, when the memory 66 stores the learning model LM 1 , the processor 64 inputs the continuous detection data DD m ' acquired from the detection device 14 after the start of step S4 into the learning model LM 1 .

[0082] Then, the learning model LM 1 estimates and outputs label information LI (i.e., initial-stage label information LI m , mid-stage label information LI 1 , or final-stage label information LI 2 ) having a correlation with the input detection data DD 3 . As an example, in this step S5, when the processor 64 detects that the learning model LM 1 outputs the final-stage label information LI 3 , it determines YES. As another example, the processor 64 may determine YES when a predetermined time has elapsed since it detects that the learning model LM outputs the final-stage label information LI 3 .

[0083] On the other hand, when the memory 66 does not store the learning model LM 2When storing, in step S5, the processor 64 uses the continuous detection data DD acquired from the detection device 14 after the start of step S4 m ’ as input to the learning model LM 2 . For example, when the detection data DD 1 ’ to DD m_p ’ indicating the movement of the worker A who fastens the first p fastening holes F is input, the learning model LM 2 estimates and outputs the time t from the time when the fastening of the p-th fastening hole F is completed in the n-th operation to the time when the n-th operation ends.

[0084] As a result, the processor 64 can recognize the time when the n-th operation ends and can determine the timing when the n-th operation ends. For example, the processor 64 measures the elapsed time τ from the time when it acquires the detection data DD m_p ’ indicating the movement of the worker A who fastens the p-th fastening hole F, and determines YES when the elapsed time τ reaches the time t output by the learning model LM 2 (or the time that is a predetermined time earlier than the time t).

[0085] In this way, the processor 64 functions as the end determination unit 30 and determines whether the n-th operation has ended based on the detection data DD m ’ and the learning model LM. According to this embodiment, the processor 64 can more accurately determine the timing when the n-th operation ends by using the learning model LM.

[0086] Note that in the above-described embodiment, the case where the robot systems 10 and 60 assist the work of one worker A has been described. However, the robot systems 10 and 60 may be configured to assist the work of a plurality of workers A. Such a form is shown in FIGS. 13 and 14.

[0087] The robot system 80 shown in FIGS. 13 and 14 includes three workers A 1 , A 2 and A 3is configured to assist with the work. Specifically, the robot system 80 includes a robot 12, detection devices 14A, 14B, and 14C, and a control device 62. The detection devices 14A, 14B, and 14C have the same configuration as the above-described detection device 14. The detection device 14A detects the movement of worker A 1 and the detection device 14B detects the movement of worker A 2 and the detection device 14C detects the movement of worker A 3 .

[0088] Also, at predetermined positions within the work cell, there are placement tables C 1 for worker A 1 and placement tables C 2 for worker A 2 and placement tables C 3 for worker A 3 respectively installed. On each of the placement tables C 1 , C 2 and C 3 , a first jig D 1 and a second jig D 2 are provided.

[0089] The processor 64 of the control device 62 of the robot system 80 executes the flow shown in FIG. 3 in parallel for each of worker A 1 , A 2 and A 3 to supply articles (tool T) for work to worker A 1 , A 2 and A 3 respectively (article supply operation), and also to collect the articles used by worker A 1 , A 2 and A 3 in the work (article collection operation).

[0090] Note that the learning model LM (LM 1 , LM 2 ) stored in the memory 66 of the control device 62 of the robot system 80 is the learning model LM 1 learned using the detection data obtained when the detection device 14A detected the movement of worker A _A (LM1_A , LM 2_A ) and the learning model LM learned using the detection data detected by the detection device 14B for the movements of the worker A 2 The learning model LM _B (LM 1_B , LM 2_B ) and the learning model LM learned using the detection data detected by the detection device 14C for the movements of the worker A 3 The learning model LM _C (LM 1_C , LM 2_C ).

[0091] Here, among the workers A 1 , A 2 and A 3 , the types and orders of the operations to be performed can be different from each other. In this case, when the operations shown in the flowchart of FIG. 3 are executed in parallel for each of the workers A 1 , A 2 and A 3 , the operation information obtained in step S1 will be different for the workers A 1 , A 2 and A 3 .

[0092] Also, when the operations shown in the flowchart of FIG. 3 are being executed in parallel for each of the workers A 1 , A 2 and A 3 , the timing at which the operation ends, as determined by the processor 64 in step S5, may occur in close proximity in time among at least two of the workers A 1 , A 2 , A 3 . In this case, the processor 64 may determine the order in which the at least two workers A 1 , A 2 , A 3 end their operations, and in accordance with that order, step S8 may be executed for the workers A 1 , A 2 , A 3 .

[0093] Hereinafter, the workers A 1 and A 2This section explains the case where the timing for the 2 operation to end is approaching. In step S5, the processor 64 uses the above-mentioned learning model LM 1 to estimate the time when worker A 2 and A 1 will finish their work. Therefore, the processor 64 can determine whether the time when worker A 2 and A 1 will finish their work is approaching, and can also determine the order in which the work of worker A 2 and A

[0094] ends. For example, if the processor 64 determines that worker A 1 will finish the work before worker A 2 , it first executes step S8 for worker A 1 , and then executes step S8 for worker A 2 . In this case, in step S5 of the flow related to worker A 1 , the processor 64 may determine YES when the time reaches a predetermined time before the time estimated by the learning model LM 2_A that the work will end.

[0095] On the other hand, in step S5 of the flow related to worker A 2 , the processor 64 may determine YES at the time estimated by the learning model LM 2_B that the work will end. Thus, when the timing for the work of worker A 1 and A 2 to end is approaching, the processor 64 can execute step S8 at different timings for worker A 1 and A 2 according to the order in which the work ends.

[0096] Alternatively, the processor 64 of the robot system 80, similar to step S5 in the above-mentioned robot system 10, uses the detection data DD mMonitor whether the movement of worker A shown by matches the reference movement pattern, and when the movement of worker A matches the reference movement pattern, it can also be determined that the nth operation is completed. In this case, if worker A 1 and A 2 are about to finish their operations close in time, the processor 64 can determine which of the movements of worker A 1 and A 2 matches the reference movement pattern first.

[0097] For example, if it is determined that the movement of worker A 1 matches the reference movement pattern earlier than that of worker A 2 , first, step S8 is executed for worker A 1 , and then step S8 is executed for worker A 2 . In this case, in step S5 of the flow regarding worker A 1 , when it is determined that the movement of worker A 1 matches the reference movement pattern, it is determined as YES, and step S8 is executed for worker A 1 .

[0098] On the other hand, in step S5 of the flow regarding worker A 2 , when it is determined as YES after a predetermined time has elapsed since it was determined that the movement of worker A 2 matches the reference movement pattern, step S8 is executed for worker A 2 . As a result, when the timings at which the operations of worker A 1 and A 2 are about to finish are close, for worker A 1 and A 2 , step S8 can be executed at different timings according to the order in which the operations are completed.

[0099] With the control method as described above, even when the timings at which the operations are completed are close in time among at least two workers A 1 , A 2 , A 3 , for worker A1 、A 2 、and A 3 For A, steps S8 (article supply operation) and S9 (article collection operation) can be smoothly executed without the operation of robot 12 or worker A 1 、A 2 、A 3 from stalling.

[0100] Also, since robot 12 can efficiently supply articles, the operating rate of robot 12 can be increased. As a result, it is not necessary to provide a plurality of robots to assist the respective operations of a plurality of workers A 1 、A 2 、A 3 ; one robot 12 can assist the operations of a plurality of workers A 1 、A 2 、A 3 . Therefore, the cost of the robot system 80 can be reduced.

[0101] Note that in the robot system 80, the processor 64 may execute the flow of FIG. 3 (i.e., the flow without using the learning model LM in step S5) executed by the above-described robot system 10 for any one of workers A 1 、A 2 and A 3 . In this case, the processor 64 will execute the flow of FIG. 3 without using the learning model LM and the flow of FIG. 3 using the learning model LM in parallel for different workers. Also, in the robot system 80, one detection device 14 may be configured to detect the movements of workers A 1 、A 2 and A 3 .

[0102] Note that in the above-described embodiment, the case where the robot 12 sets the nth tool T n to the first jig D 1 was described in step S8. However, the present invention is not limited to this, and the robot 12 may be a worker A (or A 1 、A 2 、A3 ) to the nth tool T n may be directly handed over.

[0103] In this case, the control devices 16, 62 (processor 64) perform, as the article supply operation in step S8, the nth tool T gripped by the robot hand 26 of the robot 12 n to a predetermined handover position. This handover position is determined as a position close to the operator A for handing over the nth tool T n to the operator A. The position data in the robot coordinate system C R of the handover position is stored in advance in the memory (66). The operator A (or A 1 , A 2 , A 3 ) receives, with the hand H, the nth tool T placed by the robot 12 at the handover position in step S8 n .

[0104] In this case, a force sensor for detecting an external force F applied to the robot hand 26 may be provided on the robot 12. When the operator A (or A 1 , A 2 , A 3 ) receives the nth tool T n from the robot 12 with the hand H, the external force F applied to the robot hand 26 will vary. The control devices 16, 62 (processor 64) monitor the external force F detected by the force sensor, and when the external force F varies beyond a predetermined threshold value, it may be determined that the operator A has received the nth tool T n and proceed to step S9.

[0105] Also, in the above-described embodiment, in step S9, the case where the robot 12 grips and retrieves the (n - 1)th tool T used in the immediately preceding (n - 1)th operation, which is set on the second jig D 2 , with the robot hand 26 has been described. However, not limited to this, the robot 12 may receive the (n - 1)th tool T from the operator A (or A n-1 ), A 1 , A 2 , A 3 ) n-1It may be received directly.

[0106] In this case, a force sensor for detecting an external force F applied to the robot hand 26 may be provided in the robot 12. Then, in step S9, the control devices 16, 62 (processor 64) first place the tip (TCP) of the robot hand 26 in the handover position with the fingers 26a of the robot hand 26 open as an article recovery operation. Worker A (or, A 1 、A 2 、A 3 ) inserts the (n - 1)-th tool T n-1 between the fingers 26a of the robot hand 26 and applies an external force F to the robot hand 26 through the (n - 1)-th tool T n-1 . The force sensor detects the external force F at this time.

[0107] When the external force F detected by the force sensor exceeds a predetermined threshold value, the control devices 16, 62 (processor 64) close the fingers 26a of the robot hand 26 to grip the (n - 1)-th tool T n-1 . In this way, the robot 12 can directly receive the (n - 1)-th tool T 1 、A 2 、A 3 ) from worker A (or, A n-1 . Then, the control devices 16, 62 (processor 64) transport the received (n - 1)-th tool T n-1 to a predetermined position on the storage table E.

[0108] In the above-described embodiment, the case where the detection device 14 detects the movement of worker A (or, A 1 、A 2 、A 3 ) based on an image captured by an optical motion capture has been described. However, it is not limited to this, and the detection device may detect the movement of worker A (or, A 1 、A 2 、A 3 ) by a so-called inertial sensor type motion capture. 1 、A 2 、A 3The movement of ( ) may be detected. In this case, the detection device has a plurality of acceleration sensors provided on each part of the body of worker A (or A 1 、A 2 、A 3 ), and detects the movement of worker A (or A 1 、A 2 、A 3 ) from the output signals of the acceleration sensors.

[0109] When training the learning model LM in the machine learning device 50 using a detection device having such acceleration sensors, the processor 64 (learning data acquisition unit 52) may acquire, as the learning data set DS, the output signals of the respective acceleration sensors as detection data. Alternatively, the processor 64 (learning data acquisition unit 52) may acquire, as the learning data set DS, detection data (image data) indicating the movement of worker A (or A 1 、A 2 、A 3 ) obtained from the output signals of the respective acceleration sensors. Further, the detection device may use any type of motion capture technology such as mechanical or magnetic.

[0110] In the above-described embodiment, the case where the control device 16 (processor 64) functions as the end determination unit 30 and determines whether or not the nth operation has ended based on the detection data DD of the detection device 14 has been described. However, the present invention is not limited to this, and an image processing processor incorporated in the detection device 14 may function as the end determination unit 30 and determine whether or not the nth operation has ended based on the detection data DD.

[0111] In the above-described embodiment, the case where the nth operation is an operation of fastening the fastener G to the fastening holes F 1 ~F 5 in the order of fastening holes F 1 →F 2 →F 3 →F 4 →F 5 has been described. However, for the fastening holes F 1 ~F 5The order of fastening the fastener G may be any order. Also, the fastening holes F 1 ~F 5 may be any number.

[0112] Furthermore, the nth operation is not limited to the fastening operation. For example, the nth operation may be an operation in which the operator A performs welding (e.g., spot welding) on the first welding point, the second welding point, ··· the nth welding point with the nth tool which is a welding torch. Or, the nth operation may be an operation in which the operator A applies paint in order to the first painting location, the second painting location, ··· the nth painting location with the nth tool which is a paint applicator. Or, the nth operation may be an operation in which the operator A solders electronic components at the first mounting position, the second mounting position, ··· the nth mounting position on the substrate with the nth tool which is a soldering iron.

[0113] Also, in the above-described embodiment, the case where the robot 12 supplies and retrieves the tool T to and from the operator as an article for work has been described. However, it is not limited thereto, and the robot 12 may supply and retrieve, for example, parts to be attached to the product (fasteners G such as bolts, electronic components such as IC chips) as articles for work.

[0114] Also, the robot hand 26 is not limited to gripping an article with the finger portion a, and may have a suction portion and suck the article by the suction portion. Also, the robot 12 is not limited to a vertically articulated type, and may be any type of robot such as a horizontally articulated type or a parallel link type. Although the present disclosure has been described through the embodiments above, the above-described embodiments do not limit the invention according to the claims.

Description of Reference Numerals

[0115] 10, 60, 80 Robot system 12 Robot 14, 14A, 14B, 14C Detection device 16, 62 Control device 28 Robot control unit 30 End determination unit 50 Machine learning device 52 Learning data acquisition unit 54 Learning unit

Claims

1. A machine learning device that learns the timing at which an operation by a worker who sequentially performs operations on a plurality of work locations ends for the last of the work locations, comprising: a learning data acquisition unit that acquires, as a learning data set, detection data of a detection device that detects the movement of the worker while the operation is being performed on one of the work locations, and the time from when the operation on the one work location ends until the operation on the last work location ends; a learning unit that generates a learning model representing the correlation between the movement and the time using the learning data set. A machine learning device.

2. A robot system that assists a worker's operation, comprising: a robot; the machine learning device according to claim 1; an end determination unit that receives an input of the detection data and determines whether or not the operation has ended based on the time output by the learning model; a robot control unit that causes the robot to execute an article supply operation of transporting an article for the operation to a predetermined position to supply the article to the worker, or an article recovery operation of recovering the article used in the operation and transporting it to a predetermined storage location, when it is determined by the end determination unit that the operation has ended. A robot system.

3. A plurality of the workers each perform a different one of the operations, the end determination unit determines the order in which the operations by the plurality of workers end based on the time output by the learning model, The robot control unit according to claim 2, which executes the article supply operation or the article recovery operation for the plurality of workers in accordance with the order. A robot system.

4. The end determination unit determines whether or not the times at which at least two of the workers end the operation are close based on the time output by the learning model, When it is determined that the ending times are close, the timing at which it is determined that one of the at least two workers has ended the operation is shifted from the time output by the learning model when receiving an input of the detection data detecting the movement of the one worker. The robot system according to claim 2.

5. A machine learning method for learning the timing at which an operation by a worker who sequentially performs operations on a plurality of work locations ends for the last of the work locations, comprising: During the performance of the work on one of the work locations, detection data of a detection device that detects the movement of the worker, and the time from the completion of the work on the one work location until the completion of the work on the last work location are acquired as a learning dataset. A machine learning method for generating a learning model representing the correlation between the movement and the time using the learning dataset.

Images