Robotic system, control method, machine learning device, and machine learning method

Abstract

A robot system, a control method, a machine learning device, and a machine learning method are provided. The robot system assists a worker so that the worker more smoothly performs work. The robot system has a robot, a detection device that detects an activity of the worker when the worker is performing predetermined work, an end judgment section that judges whether or not the work ends based on detection data of the detection device, and a robot control section that causes the robot to perform an article supply action or an article recovery action when it is judged by the end judgment section that the work ends, the article supply action being an action of transporting an article for the work to a predetermined position in order to supply the article to the worker, and the article recovery action being an action of recovering the article used in the work and transporting the article to a predetermined storage place.

Description

Technical Field

[0001] This invention relates to a robot system for assisting operators in their work, a robot control method, a machine learning device, and a machine learning method. Background Technology

[0002] Robotic systems that assist operators in their work are known (e.g., Japanese Patent Application Publication No. 2018-058178). Operators seek robotic systems that assist them to perform their work more smoothly. Summary of the Invention

[0003] One aspect of this disclosure provides a robot system for assisting an operator's work, comprising: a robot; a detection device that detects the operator's activities while the operator is performing a predetermined task; an end-of-task determination unit that determines whether the task has ended based on the detection data from the detection device; and a robot control unit that, upon the end-of-task determination unit determining that the task has ended, causes the robot to perform a material supply action or a material retrieval action, wherein the material supply action is an action of supplying the operator with the material used in the task and transporting it to a predetermined location, and the material retrieval action is an action of retrieving the material used in the task and transporting it to a predetermined storage location.

[0004] Another aspect of this disclosure provides a method for controlling a robot that assists an operator in their work. When the operator is performing a predetermined task, the operator's activities are detected by a detection device. The detection data from the detection device is used to determine whether the task has ended. If the task has ended, the robot performs a material supply action or a material retrieval action. The material supply action is the action of supplying the operator with the material used in the task and transporting the material to a predetermined location. The material retrieval action is the action of retrieving the material used in the task and transporting it to a predetermined storage location.

[0005] Another aspect of this disclosure provides a machine learning apparatus for learning the timing of a worker's work completion, comprising: a learning data acquisition unit that acquires detection data from a detection device that detects the worker's activities while the worker is performing a predetermined task, label information representing the task stage, or the time required for the task, as a learning dataset; and a learning unit that uses the learning dataset to generate a learning model representing the correlation between activity and stage or time.

[0006] Another aspect of this disclosure provides a machine learning method for learning the timing of a worker's work completion. This method obtains detection data from a detection device that detects the worker's activities while the worker is performing a predetermined task, label information representing the work stage, or the time required for the task, as a learning dataset. Using this learning dataset, a learning model representing the correlation between activity and stage or time is generated.

[0007] According to this disclosure, the robot control unit causes the robot to perform a material supply action or a material retrieval action at a timed end of the operation, thereby assisting the operator in performing the operation more smoothly. Attached Figure Description

[0008] Figure 1 This is a block diagram of a robot system according to one implementation method.

[0009] Figure 2 yes Figure 1 The diagram shows a 3D representation of the robot system.

[0010] Figure 3 It means Figure 1 The flowchart shows an example of the motion flow of the robot system.

[0011] Figure 4 It's enlarged. Figure 2 An enlarged view of the workbench shown.

[0012] Figure 5 This is used to illustrate an example of a task performed by an operator.

[0013] Figure 6 This indicates that the operator used tools to... Figure 5 The fastening holes are shown in the state of the fasteners.

[0014] Figure 7 This is a block diagram of a machine learning device according to one embodiment.

[0015] Figure 8 This schematically represents the detection data continuously acquired by the detection device.

[0016] Figure 9 It means Figure 7 The flowchart shown is an example of a learning process performed by a machine learning device.

[0017] Figure 10 A schematic representation of a neuron model.

[0018] Figure 11 A schematic representation of a multilayer neural network model.

[0019] Figure 12This is a block diagram of a robot system with other implementation methods.

[0020] Figure 13 This is a perspective view of a robot system according to another embodiment.

[0021] Figure 14 yes Figure 13 The diagram shows a block diagram of the robot system. Detailed Implementation

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, in the various embodiments described below, the same elements will be labeled with the same reference numerals, and repeated descriptions will be omitted. First, refer to... Figure 1 as well as Figure 2 A robot system 10 according to one embodiment will be described. The robot system 10 performs work in cooperation with operator A, and includes: robot 12, detection device 14, and control device 16.

[0023] In this embodiment, robot 12 is a vertical articulated robot, comprising: a base 18, a rotating body 20, a robot arm 22, a wrist 24, and a manipulator 26. The base 18 is fixed to the floor of the work unit. The rotating body 20 is rotatably mounted on the base 18 about a vertical axis.

[0024] The robotic arm 22 has a lower wrist 23 rotatably mounted on a rotating body 20 about a horizontal axis; and an upper wrist 25 rotatably mounted on the front end of the lower wrist 23. A wrist 24 is connected to the front end of the upper wrist 25 to support the robotic hand 26 in a manner that allows the robotic hand 26 to rotate. The robotic hand 26 has a plurality of fingers 26a that can be opened and closed, through which objects are grasped or released.

[0025] Each component of robot 12 (base 18, rotating body 20, robot arm 22, wrist 24, and manipulator 26) has a built-in servo motor (not shown). Under the command of the control device 16, the servo motor causes the movable components of robot 12 (rotating body 20, robot arm 22, and wrist 24) to rotate around the drive shaft, thereby causing robot 12 to move.

[0026] A robot coordinate system C was set for robot 12. R Robot coordinate system C R This is the coordinate system of the movable components (rotator 20, robot arm 22, wrist 24) of the automatically controlled robot 12 in three-dimensional space. In this embodiment, the robot coordinate system C... R The origin of the rotating body 20 is set at the center of the base 18, and its z-axis is configured to be parallel to the vertical direction. The rotating body 20 rotates around the z-axis.

[0027] Control device 16 uses robot coordinate system C R Based on this, instructions are generated for each servo motor built into robot 12, which in turn causes the movable components of robot 12 to move, thereby configuring manipulator 26 into robot coordinate system C. R Any position and posture within.

[0028] The detection device 14 detects the activities of operator A while operator A is performing a predetermined task. In this embodiment, the detection device 14 includes an imaging element, an optical system such as a focusing lens, and an image processing processor (e.g., a GPU). It captures images of operator A during the task along a predetermined line of sight VD and detects operator A's activities based on the captured images (so-called optical motion capture). The task performed by operator A will be described later.

[0029] The control device 16 controls the robot 12 and the detection device 14. The control device 16 is, for example, a computer having a processor (CPU, GPU, etc.), memory (ROM, RAM, etc.), input devices (keyboard, mouse, touch sensor, etc.), and a display (LCD, OLED, etc.). The processor of the control device 16 performs arithmetic processing for executing the various functions described later.

[0030] Next, refer to Figure 3 The operation of the robot system 10 is described below. When the control device 16 receives a start command from the operator, the host controller, or the computer program, it begins... Figure 3 The process is shown below. In step S1, the control device 16 obtains information about the work performed by operator A.

[0031] In this embodiment, operator A assigns tasks according to the order of task 1, task 2, ..., task n, ..., task n. MAX Homework (n) MAX In the order of positive integers, sum up n. MAX This type of assignment, in the nth assignment (n = 1 to n) MAX In ), the nth tool T is used. n To perform the task. Additionally, the nth tool T... n These can be different tools from each other, or at least two tools (the (n-1)th tool T). n-1 With the nth tool T n They are the same as each other.

[0032] In step S1, control device 16 obtains job information from the operator, a higher-level controller, or a computer program. This job information includes the type and sequence of the nth job, and information for identifying the type of the nth tool (e.g., identification number). Control device 16 can identify the sequence of operations and the tool T to be supplied to operator A from this job information. n .

[0033] Alternatively, operator A can operate the input device of control device 16 to input work information. Or, operator A can use voice recognition technology to input work information. In this case, control device 16 is also equipped with a microphone for voice input and pre-installed with a computer program for voice recognition technology.

[0034] In step S2, the control device 16 will be used to identify the nth job (i.e., the nth tool T). n The number "n" of the control device 16 is set to "1". In step S3, the control device 16 causes the robot 12 to perform the item supply action. Here, in this embodiment, as... Figure 4 As shown, a platform C is set at a predetermined position within the work unit. A first clamp D1 and a second clamp D2 are installed on the platform C.

[0035] Robot 12 sets the tool T to be used by operator A in the next task at the first fixture D1. Conversely, after operator A performs the task, the tool T to be used in that task is manually set at the second fixture D2. Furthermore, a storage table E for storing the tool T is set at a predetermined location within the work unit. The nth tool T used in the nth task... n The first clamp D1, the second clamp D2, and the nth tool T on the storage platform E are stored statically at their predetermined positions. n In robot coordinate system C R The position data (coordinate data) is pre-stored in the memory of the control device 16.

[0036] In step S3, the control device 16, based on the first tool T1 and the first fixture D1, positions itself in the robot coordinate system C. R The position data in the system enables the robot 12 to perform the item supply action. Under the instruction from the control device 16, the robot 12 uses the robotic arm 26 to grasp the first tool T1 stored on the storage table E and place the first tool T1 in the first clamp D1.

[0037] In step S4, the detection device 14 begins detecting the activities of operator A. Specifically, the detection device 14 receives instructions from the control device 16 and continuously captures images of operator A (in other words, performs video recording), detecting operator A's activities based on the captured images. Thus, the detection device 14 sends detection data DD (e.g., image data representing operator A's activities in a continuous time series) representing operator A's activities to the control device 16.

[0038] In step S5, the control device 16 determines whether the nth operation has ended based on the detection data DD from the detection device 14. (See below for further details.) Figure 5 as well as Figure 6 The method for determining the end of the nth task is explained. As an example, the nth task can be defined as operator A using the nth tool T. n ( Figure 6 ),right Figure 5 The five fastening holes F1 to F5 shown are used to fasten the fasteners G (bolts, etc.) in the order of F1→F2→F3→F4→F5.

[0039] In this situation, operator A first performs the following nth operation: using hand H to pick up the nth tool T set in the first fixture D1. n Using the nth tool T n Tighten fasteners G sequentially through fastening holes F1 to F5. Then, after completing the nth operation (i.e., tightening fastener G in the last fastening hole F5), operator A will use the nth tool T. n Set in the second clamp D2.

[0040] The control device 16 analyzes the detection data DD obtained from the detection device 14 to determine whether the nth operation has ended. This nth operation includes a series of actions to sequentially tighten fasteners G into fastening holes F1 to F5. For example, based on the obtained detection data DD, the control device 16 continuously calculates the distance I between a part of operator A's body (in this case, hand H) and the last work area, i.e., fastening hole F5. Figure 6 Monitor whether the distance I is equal to a predetermined threshold I. th The following (I≤I th ).

[0041] Control device 16 detects I≤I th When the nth operation is completed, the fastening hole F5 is positioned in the robot coordinate system C. R The position data is pre-stored in the memory of the control device 16. The control device 16 can determine the position data based on the robot's coordinate system C. RThe distance I is calculated using the position data of the fastening hole F5 and the detection data DD of the detection device 14.

[0042] Alternatively, the control device 16 can be configured such that the distance I from the detected point is a predetermined threshold I. th The nth operation is considered complete when a predetermined time (e.g., 5 seconds) has elapsed since the beginning of the following time point. Alternatively, the control device 16 can be set to operate at a distance I that is a predetermined threshold I. th After that, the distance I exceeds the threshold I again. th (I>I th When the value reaches 0, the nth task is considered complete.

[0043] As another example, the control device 16 pre-stores the reference action pattern of operator A's activity on the final work area, namely, fastening the fastener G to the fastening hole F5. The reference action pattern can be generated based on the detection data DD obtained by the detection device 14 detecting the activity of operator A (or other operator) when operator A (or other operator) is performing the reference action on the final fastening hole F5 to fasten the fastener G.

[0044] Control device 16 monitors whether the activity of operator A, represented by the detection data DD obtained after the start step S4, is consistent with the reference action pattern. When operator A's activity is consistent with the reference action pattern, it determines that the nth operation has ended. In addition, control device 16 can determine that the nth operation has ended after a predetermined time has elapsed since it was determined that operator A's activity is consistent with the reference action pattern.

[0045] Using the method described above, the control device 16 can determine the timing of the end of the nth task. When the control device 16 determines that the nth task has ended (i.e., yes), it proceeds to step S6; conversely, when it determines that the nth task has not ended (i.e., no), it repeatedly performs step S5. Thus, in this embodiment, the control device 16 serves as the end determination unit 30 for determining whether the nth task has ended based on the detection data DD. Figure 1 To fulfill its function.

[0046] In step S6, the control device 16 increments the number "n" used to identify the nth job (nth tool) by "1" (n = n + 1). In step S7, it is determined whether the number "n" used to identify the nth job exceeds the maximum number of jobs n. MAX (n>n MAX The control device 16 determines that n > n MAX (That is, if it is), proceed to step S10; on the other hand, when it is determined that n≤n MAX If (i.e., no), proceed to step S8.

[0047] In step S8, the control device 16 causes the robot 12 to perform a material supply action. Specifically, the control device 16 determines the material supply action based on the nth tool T used in the next operation. n With the first fixture D1 in robot coordinate system C R The location data in the system enables robot 12 to perform the item supply action. Robot 12 uses robotic arm 26 to grasp the nth tool T stored on storage table E. n And the nth tool T n Set in the first fixture D1.

[0048] Thus, in this embodiment, the control device 16 serves as the robot control unit 28. Figure 1 To perform its function, the robot control unit 28, in order to control the nth tool T used in the nth operation, n Provided to operator A, so that robot 12 can perform the task of placing the nth tool T. n The operation of supplying items to a predetermined location (clamp D1).

[0049] In step S9, the control device 16 functions as the robot control unit 28, causing the robot 12 to perform the item retrieval operation. Specifically, the control device 16 adjusts the position of the second gripper D2 in the robot coordinate system C. R The robot 12 is controlled by the position data in the second gripper D2. The robot 12 uses the manipulator 26 to grasp the (n-1)th tool T used in the (n-1)th operation immediately preceding the second gripper D2. n-1 .

[0050] Next, control device 16 will adjust the settings according to the (n-1)th tool T that should be placed on storage table E. n-1 In robot coordinate system C R The robot 12 is controlled by the position data in the middle, and the robot 12 will grasp the (n-1)th tool T held by the manipulator 26. n-1 Return to the predetermined position on storage table E. For example, when the start time of this step S9 is set to n=4, the control device 16 retrieves the third tool T3 used in the third operation from the second gripper D2 via the robot 12 and transports it to the predetermined position on storage table E.

[0051] Thus, control device 16 causes robot 12 to execute the (n-1)th tool T to be used in the (n-1)th preceding task. n-1 The item is retrieved and transported to a predetermined storage location (storage station E). Additionally, before performing step S9, the control device 16 can determine whether the (n-1)th tool T is positioned in the second clamp D2 based on images captured by the detection device 14 (or other detection devices). n-1 When the (n-1)th tool T is set in the second fixture D2 n-1At that time, begin step S9.

[0052] Alternatively, settings can be configured in the second fixture D2 to indicate whether the (n-1)th tool T is configured in the second fixture D2. n-1 The control device 16 determines, based on the output signal of the object detection sensor (e.g., a proximity switch), whether the (n-1)th tool T is placed in the second fixture D2. n-1 After step S9, the control device 16 returns to step S5 until it is determined to be yes in step S7, and repeats steps S5 to S9.

[0053] When the determination is yes in step S7, in step S10, the control device 16 functions as the robot control unit 28, causing the robot 12 to perform the item retrieval action. Specifically, the control device 16, based on the second gripper D2 in the robot coordinate system C... R The robot 12 is controlled by the position data in the second gripper D2. The robot 12 uses the manipulator 26 to grasp the nth position of the gripper. MAX The nth one used in the assignment MAX Tool T nMAX .

[0054] Next, control device 16 will adjust the settings according to the nth position that should be set on storage table E. MAX Tool T nMAX In robot coordinate system C R The position data is used to control robot 12, which will be grasped by robotic arm 26. MAX Tool T nMAX Return to the designated location on storage table E.

[0055] As described above, in this embodiment, the control device 16 determines that the nth task has ended and causes the robot 12 to perform a material supply operation (step S8). According to this structure, when operator A performs the (n+1)th task after the nth task, the robot 12 automatically sets the (n+1)th tool T for that (n+1)th task on the first gripper D1. n+1 Therefore, it can help operator A to smoothly carry out the nth task.

[0056] Furthermore, in this embodiment, when the control device 16 determines that the nth task has ended, it causes the robot 12 to perform an item retrieval operation (steps S9 and S10). According to this structure, when operator A completes the nth task, the robot 12 can automatically retrieve the used nth tool T. n Therefore, it can assist operator A in their work, enabling operator A to carry out the work smoothly.

[0057] Additionally, in step S5, the control device 16 can determine whether the nth task has ended based on the learning model LM, whereby the learning model LM represents the correlation between the activities of operator A performing the nth task and the stage of the nth task or the time t required for the nth task. Hereinafter, refer to... Figure 7 The machine learning device 50 used to learn the learning model LM will be described.

[0058] The machine learning device 50 may, for example, consist of a computer having a processor (CPU, GPU, etc.), memory (ROM, RAM), input devices (keyboard, mouse, touch sensor, etc.), and a display (liquid crystal display, organic EL display, etc.), or software such as learning algorithms executed by the computer.

[0059] The machine learning device 50 learns the timing of when operator A finishes the nth job. The machine learning device 50 has a learning data acquisition unit 52 and a learning unit 54. As one embodiment of the function of the machine learning device 50, the learning data acquisition unit 52 acquires the detection data DD from the detection device 14 and the label information LI representing the stage (initial stage, intermediate stage, final stage, etc.) of the nth job, and uses it as a learning dataset DS1.

[0060] More specifically, the detection device 14 takes a picture of operator A along the line of sight VD while operator A is performing the nth operation, and detects operator A's activities based on the captured image.

[0061] The detection device 14 provides the detection data DD (e.g., image data representing the activities of operator A in a time series) representing the activities of operator A to the learning data acquisition unit 52.

[0062] Figure 8 This schematically represents the detection data DD continuously acquired by the detection device 14. m . Figure 8 The horizontal axis represents time t. Additionally, Figure 8 The following scenario illustrates this: From the start of the nth operation by operator A until the end of the operation, a total of 300 data points DD were detected by detection device 14. m (m = 1 ~ 300).

[0063] The tag information LI includes, for example, initial stage tag information LI1 indicating that the nth task is in the "initial stage", intermediate stage tag information LI2 indicating that the nth task is in the "intermediate stage", and final stage tag information LI3 indicating that the nth task is in the "final stage". Here, Figure 8 The detection data shown are DD1~DD 99 The collection represents operator A's work on... Figure 5The first two fastening holes F (e.g., fastening holes F1 and F2) in the fastening holes F1 to F5 are fastened.

[0064] In addition, the detection data DD 100 ~DD 199 The collection represents operator A's work on... Figure 5 The next two fastening holes F (e.g., fastening holes F3 and F4) from fastening holes F1 to F5 are used for fastening. Furthermore, the detection data DD... 200 ~DD 300 The collection represents operator A's work on... Figure 5 The activity of tightening is performed on the last fastening hole F (e.g., fastening hole F5) among the fastening holes F1 to F5.

[0065] At this point, the operator can process the detection data DD1 to DD2. 99 Initial stage labeling information LI1, and detection data DD 100 ~DD 199 Labeling the intermediate stage information LI2, and applying it to the detection data DD 200 ~DD 300 Labeling information LI3 in the final stage. While the operator is labeling information LI, the display of the machine learning device 50 shows the detection data DD. m (Image data).

[0066] Furthermore, the operator visually monitors the detection data DD displayed on the screen. m The image, while operating the input device of the machine learning device 50, is fed into the detection data DD. m Arbitrarily assign one of the following labels to represent the progress stage of the nth task: initial stage label LI1, intermediate stage label LI2, and final stage label LI3.

[0067] Thus, the operator uses the detection data DD m The activity represented by operator A arbitrarily assigns the tag information LI to the detection data DD. m The learning data acquisition unit 52 acquires the detection data DD obtained from the detection device 14. m The training dataset DS1 consists of the label information LI input by the operator.

[0068] In addition, Figure 8 In the example shown, for ease of understanding, it is described that in the nth operation, the detection device 14 detected a total of 300 detection data points DD1 to DD2. 300 However, the detection device 14 can detect any number of detection data DD in the nth operation. m The operator uses the detected data DDm Arbitrarily decide on any number of detection data DD m The collection is assigned any kind of tag information LI1, LI2 or LI3.

[0069] Furthermore, the label information LI is not limited to the initial stage label information LI1, intermediate stage label information LI2, and final stage label information LI3 mentioned above. For example, as the final stage label information LI3, multiple label information such as "Final Stage 1," "Final Stage 2," and "Final Stage 3" can be set in chronological order. Similarly, multiple label information can also be set as the initial stage label information LI1 or the intermediate stage label information LI2. In addition, the detection data DD can also be... m Only assign the final stage label information LI3.

[0070] The learning unit 54 uses the learning dataset DS1 to generate a learning model LM1 (function). This learning model LM1 represents the correlation between the activities of operator A performing the nth task and the stages (initial stage, intermediate stage, final stage) of the nth task. For example, the learning unit 54 generates the learning model LM1 by performing supervised learning. In this case, operator A repeatedly attempts the nth task, and the learning data acquisition unit 52 repeatedly acquires the learning dataset DS1 as training data each time operator A performs the nth task.

[0071] Learning unit 54 learns the learning model LM1 by identifying the following features, which are implied by the detection data DD obtained as training data. m The correlation between the label information LI representing the stage of the nth task (initial stage label LI1, intermediate stage label LI2, and final stage label LI3). Supervised learning for this purpose can be achieved using algorithms such as Support Vector Machines (SVM) or Gaussian Mixture Models (GMM).

[0072] The following is for reference Figure 9 The process of the learning loop performed by the machine learning device 50 is explained. In step S11, operator A performs the nth task, and the detection device 14 detects operator A's activity during the nth task. In step S12, the learning data acquisition unit 52 acquires the detection data DD. m The learning dataset DS1, along with the label information LI, is correlated and stored in the memory of the machine learning device 50.

[0073] In step S13, the learning unit 54 uses the learning dataset DS1 obtained in step S12 to generate a learning model LM1, which represents the correlation between the activities of operator A performing the nth task and the stage of the nth task. Then, the process returns to step S11.

[0074] By executing such a learning cycle, the learning model LM1 is learned and guided towards the optimal solution. According to this embodiment, the learning model LM1 can be automatically and accurately determined, which quantitatively represents the correlation between the activities of operator A performing the nth task and the stage of the nth task.

[0075] Next, other aspects of the functionality of the machine learning device 50 will be described. In this embodiment, the learning data acquisition unit 52 acquires the aforementioned detection data DD. m The training dataset DS2 is obtained from the time t required for the nth task. For example, as the training dataset DS2, the training data acquisition unit 52 obtains data representing the time required for task A. Figure 5 The detection data DD1 to DD5 are the activity data of operator A who performs the tightening of the first p (p is an integer less than 4) fastening holes F1 to F5. m_p And the time t from the time when the fastening of the p-th fastening hole F is completed to the time when the n-th operation ends.

[0076] Specifically, as the learning dataset DS2, the learning data acquisition unit 52 acquires, for example, detection data DD1 to DD2 representing the activities of operator A in tightening the initial two fastening holes F (fastening holes F1 and F2). 99 And the time point from when the tightening of the second fastening hole F (fastening hole F2) was completed (i.e., the detection data DD) 99 From the detection time point to the end time point of the nth job (detection data DD) 300 The time t up to the detection time point.

[0077] Learning Department 54 uses test data DD m The learning dataset DS2 with time t generates a learning model LM2 (function) that represents the correlation between the activities of operator A performing the nth job and the time t required for the nth job (e.g., the time t from the time when the fastening of the pth fastening hole F is completed to the time when the nth job ends).

[0078] The following is for reference Figure 9The learning loop of learning model LM2 will be explained. The difference between the learning process of learning model LM2 and that of learning model LM1 lies in steps S12 and S13. Specifically, in step S12, the learning data acquisition unit 52 acquires the detection data DD. m The learning dataset DS2 with time t is correlated with each other and stored in the memory of the machine learning device 50.

[0079] In step S13, the learning unit 54 uses the learning dataset DS2 obtained in step S12 to generate a learning model LM2, which represents the correlation between the activities of operator A performing the nth task and time t. For example, the learning unit 54 learns the learning model LM2 by executing a supervised learning algorithm (SVM, GMM, etc.). Then, the process returns to step S11.

[0080] By executing such a learning cycle, the learning model LM2 is learned and guided towards the optimal solution. According to this embodiment, the learning model LM2 can be automatically and accurately determined, which quantitatively represents the correlation between the activities of operator A performing the nth task and the time t required for the nth task.

[0081] Furthermore, the learning algorithms executed by Learning Department 54 are not limited to supervised learning; for example, unsupervised learning, reinforcement learning, neural networks, and other well-known machine learning algorithms can be used. As an example, Figure 10 A schematic representation of a neuron model. Figure 11 Indicatively representing that Figure 10 The diagram shows a three-layer neural network model composed of combined neurons. A neural network can, for example, consist of a computing device, a storage device, etc., that simulates a neuron model.

[0082] Figure 10 The neuron shown outputs a result y for multiple inputs x (x1 to x3 in the figure as an example). Each input x (x1, x2, x3) is multiplied by a weight w (w1, w2, w3). The relationship between the inputs x and the result y can be represented by the following mathematical expression 1. Furthermore, the inputs x, the result y, and the weights w are all vectors. Also, in mathematical expression 1, θ is the bias, and fk is the activation function.

[0083]

[0084] Figure 11The three-layer neural network shown takes multiple inputs x (x1 to x3 in the diagram as an example) from the left and outputs results y (y1 to y3 in the diagram as an example) from the right. In the example shown, inputs x1, x2, and x3 are multiplied by their corresponding weights (represented by W1), and each input x1, x2, and x3 is fed into three neurons N11, N12, and N13.

[0085] exist Figure 11 In this diagram, Z1 is used to represent the outputs of neurons N11 to N13. Z1 can be considered as a feature vector obtained by extracting features from the input vector. In the illustrated example, each feature vector Z1 is multiplied by its corresponding weight (represented by W2), and each feature vector Z1 is input into two neurons N21 and N22. The feature vector Z1 represents the feature between weights W1 and W2.

[0086] exist Figure 11 In this example, Z2 is used to represent the outputs of neurons N21 to N22. Z2 can be considered as a feature vector obtained by extracting the feature value of feature vector Z1. In the example shown, feature vector Z2 is multiplied by the corresponding weight (represented by W3), and each feature vector Z2 is input into three neurons N31, N32, and N33.

[0087] The feature vector Z2 represents the feature between weights W2 and W3. Finally, neurons N31 to N33 output results y1 to y3 respectively. The machine learning device 50 takes the learning dataset DS (DS1 or DS2) as input and performs multi-layer structure operations according to the above neural network, thereby learning the learning model LM (LM1 or LM2).

[0088] The structure of the aforementioned machine learning device 50 can be described as a machine learning method (or software) executed by a computer processor. Regarding this machine learning method, the processor acquires detection data DD from the detection device 14, which detects the activities of operator A while operator A is performing the nth task. m The training dataset DS1 or DS2 is used to generate a training model LM1 or LM2 that represents the stage of the nth task and the time t required for the nth task.

[0089] Next, refer to Figure 2 as well as Figure 12Other embodiments of the robot system 60 will be described. The robot system 60 includes a robot 12, a detection device 14, and a control device 62. The control device 62 is a computer having a processor 64 (CPU, GPU, etc.), a memory 66 (ROM, RAM, etc.), an input device 68 (keyboard, mouse, touch sensor, etc.), and a display 70 (liquid crystal display, organic EL display, etc.). The processor 64, memory 66, input device 68, and display 70 are communicatively connected to each other via a bus 72.

[0090] In this embodiment, the machine learning device 50 is implemented as hardware or software in the control device 62. The processor 64 communicates with the memory 66, the input device 68, and the display 70, and performs various operations to implement the functions of the machine learning device 50. That is, in this embodiment, the processor 64 functions as both the learning data acquisition unit 52 and the learning unit 54. The memory 66 pre-stores the learning model LM (LM1 or LM2) learned by the machine learning device 50.

[0091] Next, refer to Figure 3 The operation of the robot system 60 will be described. The processor 64 functions as the aforementioned robot control unit 28 and termination determination unit 30, communicating with the memory 66, input device 68, and display 70 to execute... Figure 3 The process is shown below. Here, the motion flow of robot system 60 differs from that of robot system 10 in the following aspects.

[0092] Specifically, in step S4, the detection device 14 begins detecting the activities of operator A. At this time, the detection device 14 continuously captures (video captures) images of operator A under the same conditions as in step S11 performed during the learning phase of the learning model LM, detecting operator A's activities. Specifically, the detection device 14 captures images of operator A along the same line of sight VD as in step S11, using the same shooting conditions (shutter speed, shooting speed, etc.). The detection device 14 then transmits the continuously acquired detection data DD... m 'Send to processor 64 in sequence.'

[0093] In step S5, the processor 64 determines the detection data DD from the detection device 14. m 'And the learning model LM, to determine whether the nth task has ended. Here, when the learning model LM1 is stored in the memory 66, the processor 64 will use the continuous detection data DD obtained from the detection device 14 after starting step S4.' m 'Input into the learning model LM1.'

[0094] Therefore, the learning model LM1 estimates and outputs the detection data DD input to it. m 'Relevant label information LI (i.e., initial stage label information LI1, intermediate stage label information LI2, or final stage label information LI3).' As an example, in step S5, processor 64 determines "yes" when it detects that the learning model LM1 has output the final stage label information LI3. As another example, processor 64 may also determine "yes" when a predetermined time has elapsed since the time point when the learning model LM1 output the final stage label information LI3.

[0095] On the other hand, when the learning model LM2 is stored in the memory 66, in step S5, the processor 64 will process the continuous detection data DD obtained from the detection device 14 after the start of step S4. m 'Input to the learning model LM2. For example, inputting detection data DD1' to DD2 representing the activities of operator A in tightening the initial p fastening holes F. m_p At that time, the learning model LM2 estimates and outputs the time t from the time point when the fastening of the p-th fastening hole F is completed to the time point when the n-th operation ends.

[0096] Therefore, processor 64 can identify the end time of the nth job and determine the timing of the nth job's end. For example, processor 64 obtains detection data DD... m_p The time τ elapsed from the starting point of the time interval is used to determine if the above detection data DD is true when the elapsed time τ reaches the time t output by the learning model LM2 (or, a predetermined time earlier than time t). m_p 'This represents the activity of operator A who tightens the p-th fastening hole F.

[0097] In this way, the processor 64 functions as the termination judgment unit 30, based on the detection data DD m The processor 64 uses a learning model LM to determine whether the nth job has ended. According to this embodiment, by using the learning model LM, the processor 64 can determine the timing of the end of the nth job with higher accuracy.

[0098] Furthermore, in the above embodiments, the robot systems 10 and 60 were described in the case of assisting one operator A in their work. However, the robot systems 10 and 60 may also be configured to assist the work of multiple operators A. Figure 13 as well as Figure 14 This indicates that it is done in this way.

[0099] Figure 13 as well as Figure 14The robot system 80 shown is configured to assist three operators, A2 and A3, in performing their tasks. Specifically, the robot system 80 includes: a robot 12, detection devices 14A, 14B and 14C, and a control device 62. Detection devices 14A, 14B and 14C have the same structure as detection device 14. Detection device 14A detects the activities of operator A1, detection device 14B detects the activities of operator A2, and detection device 14C detects the activities of operator A3.

[0100] In addition, a platform C1 for operator A1, a platform C2 for operator A2, and a platform C3 for operator A3 are respectively set at predetermined locations within the work unit. A first fixture D1 and a second fixture D2 are respectively set on platforms C1, C2, and C3.

[0101] The processor 64 of the control unit 62 of the robot system 80 executes commands in parallel for each operator A1, A2, and A3. Figure 3 The process shown involves supplying work-related items (tools T) to operators A1, A2, and A3 respectively (item supply action), and then collecting the items used by operators A1, A2, and A3 during the work (item collection action).

[0102] Furthermore, the learning models LM (LM1, LM2) stored in the memory 66 of the control device 62 of the robot system 80 include learning models LM learned using detection data obtained by detecting the activities of operator A1 by the detection device 14A. _A (LM 1_A LM 2_A The learning model LM is learned using detection data obtained from detecting the activities of operator A2 by detection device 14B. _B (LM 1_B LM 2_B ), and the learning model LM learned using detection data obtained by detecting the activities of operator A3 by detection device 14C. _C (LM 1_C LM 2_C ).

[0103] Here, the types and order of tasks performed by operators A1, A2, and A3 can differ. In this case, the tasks are performed in parallel by each operator A1, A2, and A3. Figure 3 In the process shown, the job information obtained in step S1 is different for operators A1, A2, and A3.

[0104] Furthermore, the work is being carried out in parallel by operators A1, A2, and A3. Figure 3During the illustrated process, the following situation may occur: the timing of the job completion determined by the processor 64 in step S5 is close in time between at least two operators A1, A2, and A3. In this case, the processor 64 can determine the order in which the at least two operators A1, A2, and A3 finish their jobs, and execute step S8 for operators A1, A2, and A3 in that order.

[0105] The following explains the situation where the completion times of operators A1 and A2 are close together. In step S5, when using the learning model LM2 described above, processor 64 can estimate the completion times of operators A1 and A2. Therefore, processor 64 can determine whether the completion times of operators A1 and A2 are close together, and can also determine the order in which operators A1 and A2 complete their tasks.

[0106] For example, when processor 64 determines that operator A1 finishes work before operator A2, it first executes step S8 for operator A1, and then executes step S8 for operator A2. In this case, processor 64 can, in step S5 of the process related to operator A1, reach the point where the learning model LM is used... 2_A If the estimated end time of the operation is earlier than the scheduled time, it is considered as yes.

[0107] On the other hand, processor 64 can, in step S5 of the process related to operator A2, learn model LM... 2_B The estimated end time of the job is determined to be yes. Therefore, when the timer for the job completion of operators A1 and A2 is close, the processor 64 can execute step S8 at different timers for operators A1 and A2 according to the order in which the jobs are completed.

[0108] Alternatively, the processor 64 of the robot system 80 can monitor the detection data DD in the same way as step S5 in the robot system 10 described above. m The processor 64 determines whether the activity of operator A is consistent with the baseline motion pattern. If operator A's activity is consistent with the baseline motion pattern, the nth operation is considered complete. In this case, assuming that the completion times of operators A1 and A2 are close, the processor 64 can determine which operator A1 or A2's activity is consistent with the baseline motion pattern first.

[0109] For example, when it is determined that operator A1's activity matches the reference motion pattern before operator A2, step S8 is executed first for operator A1, and then step S8 is executed for operator A2. In this case, in step S5 of the process related to operator A1, the processor 64 determines that it is true at the time point when it is determined that operator A1's activity matches the reference motion pattern, and then executes step S8 for operator A1.

[0110] On the other hand, in step S5 of the process related to operator A2, starting from the time point when it is determined that operator A2's activity is consistent with the reference action pattern, after a predetermined time has elapsed, step S8 is executed for operator A2. Thus, when the time of completion of the work of operators A1 and A2 is approaching, step S8 can be executed for operators A1 and A2 at different time points according to the order of work completion.

[0111] Using the control method described above, even when the time for the end of the operation is close in time between at least two operators A1, A2, and A3, the robot 12 or the operators A1, A2, and A3 can smoothly and without interruption execute steps S8 (material supply action) and S9 (material retrieval action) for operators A1, A2, and A3.

[0112] Furthermore, items can be supplied efficiently through robot 12, thus increasing the operating rate of robot 12. As a result, it is not necessary to set up multiple robots to assist the work of multiple operators A1, A2, and A3; one robot 12 can assist the work of multiple operators A1, A2, and A3. Therefore, the cost of robot system 80 can be reduced.

[0113] Furthermore, in robot system 80, processor 64 can execute the functions performed by robot system 10 on one of operators A1, A2, and A3. Figure 3 The process (i.e., the process in step S5 that does not use the learning model LM). In this case, the processor 64 executes the process without using the learning model LM in parallel for each operator. Figure 3 The process and the use of the learning model LM Figure 3 The process. In addition, the robot system 80 can also be configured with a detection device 14 to detect the activities of operators A1, A2 and A3.

[0114] Furthermore, in the above embodiment, in step S8, the robot 12 will move the nth tool T n The case of setting the first fixture D1 has been described. However, it is not limited to this; robot 12 can also use the nth tool T. n Hand it directly to operator A (or A1, A2, A3).

[0115] In this case, in step S8, the control devices 16 and 62 (processor 64) perform an item supply action, and the robot 12 grasps the nth tool T held by the robotic arm 26. n Transport to the predetermined delivery location. This delivery location will be designated for the delivery of the nth tool T. nThe location is near operator A and handed over to them. The handover location is in robot coordinate system C. R The position data is pre-stored in memory (66). Operator A (or A1, A2, A3) uses hand H to receive the nth tool T configured by robot 12 at the delivery position in step S8. n .

[0116] In this case, a force sensor can be installed on robot 12 to detect the external force F applied to manipulator 26. When operator A (or A1, A2, A3) receives the nth tool T from robot 12 using hand H... n At that time, the external force F applied to the robotic arm 26 changes. The control devices 16 and 62 (processor 64) monitor the external force F detected by the force sensor. When the external force F changes beyond a predetermined threshold, it can be determined that the operator A has accepted the nth tool T. n Proceed to step S9.

[0117] Furthermore, in the above embodiment, in step S9, the robot 12 uses the manipulator 26 to grasp and retrieve the (n-1)th tool T used in the (n-1)th operation immediately preceding the operation, which is set in the second gripper D2. n-1 The situation has been described. However, it is not limited to this; robot 12 can also directly receive the (n-1)th tool T from operator A (or A1, A2, A3). n-1 .

[0118] In this case, a force sensor can be installed on robot 12 to detect the external force F applied to robotic arm 26. Furthermore, in step S9, as an item retrieval operation, control devices 16 and 62 (processor 64) first position the fingertips (TCP) of robotic arm 26 in the delivery position with the fingertips 26a of robotic arm 26 open. Operator A (or A1, A2, A3) inserts the (n-1)th tool T between the fingertips 26a of robotic arm 26. n-1 Through the (n-1)th tool T n-1 An external force F is applied to the robotic arm 26. The force sensor detects the external force F at this time.

[0119] When the external force F detected by the force sensor exceeds a predetermined threshold, the control devices 16 and 62 (processor 64) close the finger 26a of the robotic arm 26 to grasp the (n-1)th tool T. n-1 In this way, robot 12 can directly receive the (n-1)th tool T from operator A (or A1, A2, A3). n-1 Then, control devices 16 and 62 (processor 64) will receive the (n-1)th tool T. n-1 Transported to the designated location on storage station E.

[0120] Furthermore, in the above embodiment, the method described is that the detection device 14 detects the activities of operator A (or A1, A2, A3) by means of optical motion capture based on images obtained by capturing images of operator A (or A1, A2, A3). However, it is not limited to this method; the detection device may also detect the activities of operator A (or A1, A2, A3) by means of so-called inertial sensor motion capture. In this case, the detection device has multiple accelerometers installed at various parts of operator A's (or A1, A2, A3) body, and detects the activities of operator A (or A1, A2, A3) based on the output signals of these accelerometers.

[0121] When the detection device with such accelerometers learns the learning model LM using the machine learning device 50, the processor 64 (learning data acquisition unit 52) ​​can acquire the output signals of each accelerometer as detection data, which is the learning dataset DS. Alternatively, the processor 64 (learning data acquisition unit 52) ​​can acquire detection data (image data) representing the activities of operator A (or A1, A2, A3) obtained from the output signals of each accelerometer as the learning dataset DS. Furthermore, the detection device can use any type of motion capture technology, such as mechanical or magnetic.

[0122] Furthermore, in the above embodiment, the control device 16 (processor 64) is described as functioning as an end-determination unit 30, determining whether the nth operation has ended based on the detection data DD from the detection device 14. However, it is not limited to this; an image processing processor built into the detection device 14 may also function as an end-determination unit 30, determining whether the nth operation has ended based on the detection data DD.

[0123] Furthermore, in the above embodiment, the nth operation is described as the operation of tightening the fastener G through the fastening holes F1 to F5 in the order of F1→F2→F3→F4→F5. However, the order in which the fastener G is tightened through the fastening holes F1 to F5 can be any order. Furthermore, the number of fastening holes F1 to F5 can also be any number.

[0124] Furthermore, the nth operation is not limited to fastening operations. For example, the nth operation could also be an operation in which operator A uses the nth tool, which functions as a soldering gun, to perform welding (e.g., spot welding) on ​​the first, second, ..., nth solder points. Alternatively, the nth operation could be an operation in which operator A uses the nth tool, which functions as a paint applicator, to sequentially apply paint to the first, second, ..., nth coating areas. Or, the nth operation could also be an operation in which operator A uses the nth tool, which functions as a soldering iron, to solder electronic components to the first, second, ..., nth mounting positions on a substrate.

[0125] Furthermore, in the above embodiment, the robot 12 has been described as supplying and retrieving tools T from the operator as work items. However, it is not limited to this; as work items, the robot 12 can, for example, supply and retrieve components (fasteners G such as bolts, electronic components such as IC chips) installed on the product.

[0126] Furthermore, the robotic arm 26 is not limited to grasping objects using its fingers a; it may also have an adsorption part to adsorb objects. Additionally, the robot 12 is not limited to a vertical multi-joint type; it may be any type of robot, such as a horizontal multi-joint type or a parallel linkage type. The present disclosure has been described above through embodiments, but these embodiments are not intended to limit the scope of the invention as claimed.

Claims

1. A robot system that assists operators in their work, characterized in that, The robot system has the following features: robot; The detection device uses motion capture to detect the movement of parts of the operator's body when the operator performs work on multiple work areas in sequence. The end judgment unit monitors whether a part of the operator's body is close to the last work position among the multiple work positions based on the detection data of the detection device. When a part of the operator's body is close to the last work position, the operation is judged to be over. as well as The robot control unit, upon determining by the termination judgment unit that the operation has ended, causes the robot to perform an item supply action or an item retrieval action. The item supply action is an action of transporting the item used in the operation to a predetermined location to supply the operator with the item used in the operation. The item retrieval action is an action of retrieving the item used in the operation and transporting it to a predetermined storage location.

2. The robot system according to claim 1, characterized in that, The termination determination unit monitors whether the operator's body part is close to the final work area by monitoring whether the distance between a part of the operator's body and the final work area is below a predetermined threshold based on the detection data. The termination determination unit determines that the operation is terminated when the distance is below the threshold.

3. The robot system according to claim 1 or 2, characterized in that, After performing the item supply action, the robot control unit causes the robot to perform the item retrieval action, which retrieves the items used in the preceding operation and transports them to the predetermined storage location.

4. A method for controlling a robot that assists an operator in performing tasks, characterized in that, When an operator performs work on multiple work areas sequentially, a detection device uses motion capture to detect the movement of parts of the operator's body. The detection device monitors whether a part of the operator's body is close to the last work area among the multiple work areas where the work was performed. When a part of the operator's body is close to the last work area, the work is considered to have ended. When the operation is determined to be completed, the robot performs an item supply action or an item retrieval action. The item supply action is the action of transporting the item used in the operation to a predetermined location to supply the operator with the item used in the operation. The item retrieval action is the action of collecting the item used in the operation and transporting it to a predetermined storage location.

Images