Computer program for causing a processor to execute a process for creating a control program for a robot, and method and system for creating a control program for a robot

Abstract

To provide a technique capable of reducing a processing load of a robot control program.SOLUTION: A computer program causes a processor to execute: (a) processing for recognizing a worker operation from an image obtaining by imaging one or more worker operations by an imaging device; (b) processing for recognizing finger positions in a specific finger operation when the worker operation includes the specific finger operation; (c) processing for recognizing the position of a workpiece after the work; and (d) processing for generating a control program of a robot by using the worker operation, the finger positions and the position of the workpiece.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a computer program that causes a processor to execute a process for creating a control program for a robot, and a method and system for creating a control program for a robot. [Background technology]

[0002] Patent Document 1 discloses a technology for creating teaching data for a robot. In this conventional technology, a teaching image including a worker's hand is acquired using a camera, and finger coordinates, which are the positions of each joint and fingertip of the finger, are determined based on the teaching image, and the operation of a robot arm 110 is taught based on the finger coordinates. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-110621 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, hand and finger recognition is always performed even when the object is not being grasped or released, which causes a problem of a large processing load. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a computer program for causing a processor to execute a process of creating a control program for a robot. This computer program causes the processor to execute the following processes: (a) a process of recognizing one or more worker actions included in an operation in which a worker manipulates a workpiece using an arm and fingers from an image captured by an imaging device, (b) a process of recognizing a finger position in a specific finger action from an image captured by the imaging device when the worker action includes a specific finger action accompanied by a movement of the joints of the finger, (c) a process of recognizing a position of the workpiece after the operation from an image captured by the imaging device, and (d) a process of generating a control program for the robot using the worker action recognized in the process (a), the finger position recognized in the process (b), and the position of the workpiece recognized in the process (c).

[0006] According to a second embodiment of the present disclosure, there is provided a method for creating a control program for a robot, the method including the steps of: (a) recognizing one or more worker actions included in an operation in which the worker manipulates a workpiece using an arm and fingers from an image captured by an imaging device, (b) recognizing a finger position in the specific finger action from an image captured by the imaging device when the worker action includes a specific finger action accompanied by a movement of the joints of the fingers, (c) recognizing a position of the workpiece after the operation from an image captured by the imaging device, and (d) generating a control program for the robot using the worker action recognized in the step (a), the finger position recognized in the step (b), and the position of the workpiece recognized in the step (c).

[0007] According to a third embodiment of the present disclosure, there is provided a system for executing a process for creating a control program for a robot. The system includes an information processing device having a processor, and an imaging device connected to the information processing device. The processor executes the following processes: (a) a process for recognizing one or more worker actions included in an operation in which a worker manipulates a workpiece using an arm and fingers from an image captured by the imaging device, (b) a process for recognizing a finger position in the specific finger action from an image captured by the imaging device when the worker action includes a specific finger action involving a movement of the joints of the finger, (c) a process for recognizing a position of the workpiece after the operation from an image captured by the imaging device, and (d) a process for generating a control program for the robot using the worker action recognized in the process (a), the finger position recognized in the process (b), and the position of the workpiece recognized in the process (c). [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram of a robot system according to an embodiment. [Diagram 2] FIG. 2 is a functional block diagram of an information processing device. [Diagram 3] 4 is a flowchart showing the procedure of a control program creation process. [Figure 4] FIG. 4 is an explanatory diagram showing an example of an image frame capturing an image of a workpiece in a first working area. [Diagram 5] FIG. 11 is an explanatory diagram showing a result of work recognition. [Figure 6] FIG. 4 is an explanatory diagram showing an example of an image frame capturing a worker's action. [Figure 7] FIG. 11 is an explanatory diagram showing a recognition result of a worker's action. [Figure 8] 11 is a flowchart showing a detailed procedure of step S40. [Figure 9] FIG. 11 is an explanatory diagram showing how a hand and finger position is recognized. [Figure 10] FIG. 4 is an explanatory diagram showing finger positions to be recognized; [Figure 11] FIG. 11 is an explanatory diagram showing a recognition result of a hand and finger position. [Figure 12] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1 is an explanatory diagram showing an example of a robot system according to an embodiment. The robot system includes a robot 100, a first camera 210, a second camera 220, a third camera 230, and an information processing device 300 having a function of controlling the robot 100. The information processing device 300 is, for example, a personal computer.

[0010] The robot 100 is a multi-axis robot having multiple joints. However, as the robot 100, a robot having any arm mechanism having one or more joints can be used. In addition, although the robot 100 in this embodiment is a vertical multi-joint robot, a horizontal multi-joint robot may also be used. In this embodiment, the end effector of the robot 100 is a gripper that can hold a workpiece, but any end effector can be used.

[0011] In the robot system of FIG. 1, a first working area WA1 where a worker TP performs a teaching operation and a second working area WA2 where the robot 100 performs an operation are set. The worker TP is also called an "instructor." The first working area WA1 can be photographed by a first camera 210. The second working area WA2 can be photographed by a second camera 220. It is preferable that the relative position between the first working area WA1 and the first camera 210 is set to be the same as the relative position between the second working area WA2 and the second camera 220. The first working area WA1 and the second working area WA2 may be the same area.

[0012] A third camera 230 for photographing the fingers and work of the worker TP is installed in the first working area WA1. The third camera 230 is preferably installed in a position closer to the first working area WA1 than the first camera 210 so that the third camera 230 can photograph the fingers and work closer than the first camera 210. If the positions of the fingers and work are recognized using the images captured by the third camera 230, the positions of the fingers and work can be recognized more accurately than when only the first camera 210 is used. However, the third camera 230 may be omitted.

[0013] The first working area WA1 includes a first supply area SA1 and a first target area TA1. The first supply area SA1 is an area where the workpiece WK1 is placed at the start of the teaching work. The first target area TA1 is an area where the workpiece WK1 is moved from the first supply area SA1 by the operation of the operator TP as the teaching work and the workpiece WK1 is placed. The shapes and positions of the first supply area SA1 and the first target area TA1 within the first working area WA1 can be set arbitrarily.

[0014] The second working area WA2 has the same shape as the first working area WA1, and includes a second supply area SA2 and a second target area TA2 having the same shapes as the first supply area SA1 and the first target area TA1, respectively. The second supply area SA2 is an area where the workpiece WK2 is placed at the start of the work by the robot 100. The second target area TA2 is an area where the workpiece WK2 is moved from the second supply area SA2 by the work of the robot 100 and placed therein. The supply areas SA1, SA2 and the target areas TA1, TA2 may each be realized using a tray, or the individual areas SA1, SA2, TA1, TA2 may be drawn by lines on the floor surface or a stand. The supply areas SA1, SA2 and the target areas TA1, TA2 may not be explicitly demarcated.

[0015] The workpiece WK1 to be worked on in the first working area WA1 and the workpiece WK2 to be worked on in the second working area WA2 are objects of the same type and with the same design. However, in order to make it easier to understand the correspondence between the working areas WA1 and WA2, hereinafter, these will be referred to as the "first workpiece WK1" and the "second workpiece WK2."

[0016] 1 illustrates a robot coordinate system Σr set for the robot 100, a first camera coordinate system Σc1 set for the first camera 210, a second camera coordinate system Σc2 set for the second camera 220, and a first camera coordinate system Σc3 set for the third camera 230. These coordinate systems Σr, Σc1, Σc2, and Σc3 are all orthogonal coordinate systems defined by three axes X, Y, and Z. The correspondence between these coordinate systems Σr, Σc1, Σc2, and Σc3 is determined by calibration.

[0017] The position and orientation of the workpiece WK1 in the first working area WA1 and the movement of the worker TP are recognized by the information processing device 300 from the image of the first working area WA1 captured by the first camera 210 and the third camera 230. The position and orientation of the workpiece WK2 in the second working area WA2 is recognized by the information processing device 300 from the image of the second working area WA2 captured by the second camera 220. The cameras 210, 220, and 230 are those capable of capturing an object as a video or multiple image frames. It is preferable to use the cameras 210, 220, and 230 that can recognize an object three-dimensionally. As such a camera, for example, a stereo camera or an RGBD camera capable of capturing a color image and a depth image simultaneously may be used. If an RGBD camera is used, it is possible to recognize the shape of an obstacle using a depth image. The cameras 210, 220, and 230 correspond to the "imaging device" in this disclosure.

[0018] 2 is a block diagram showing the functions of the information processing device 300. The information processing device 300 has a processor 310, a memory 320, an interface circuit 330, and an input device 340 and a display unit 350 connected to the interface circuit 330. The cameras 210, 220, and 230 are further connected to the interface circuit 330.

[0019] The processor 310 has the functions of an object recognition unit 311, an action recognition unit 312, a hand and finger position recognition unit 313, a work description list creation unit 314, and a control program creation unit 315. The object recognition unit 311 recognizes the first work WK1 from the image captured by the first camera 210 or the third camera 230, and recognizes the second work WK2 from the image captured by the second camera 220. The action recognition unit 312 recognizes the action of the worker TP from the image captured by the first camera 210. The hand and finger position recognition unit 313 recognizes the hand and finger position of the worker TP from the image captured by the first camera 210 or the third camera 230. The recognition by the object recognition unit 311, the action recognition unit 312, and the hand and finger position recognition unit 313 can be realized by using a machine learning model by deep learning or a feature extraction model. The work description list creation unit 314 creates a work description list WDL (described later) using the recognition results of the other units. The control program creation unit 315 uses the recognition results of the other units or the work description list WDL to create a control program for the robot 100. The functions of these units 311 to 315 are realized by the processor 310 executing a computer program stored in the memory 320. However, some or all of the functions of each unit may be realized by a hardware circuit.

[0020] The memory 320 stores robot characteristic data RD, work attribute data WD, a task description list WDL, and a robot control program RP. The robot characteristic data RD includes characteristics such as the geometric structure of the robot 100, the rotation angle of the joints, the weight, and the inertia value. The work attribute data WD includes attributes such as the type and shape of the workpieces WK1 and WK2. The task description list WDL is data representing the operation content recognized from a video or multiple image frames of the motion of the worker TP and the workpiece WK1, and is data describing the operation in a robot-independent coordinate system that is independent of the type of robot. The robot control program RP is composed of multiple commands for operating the robot 100. The robot control program RP is configured to control, for example, a pick-and-place operation for moving the second workpiece WK2 from the second supply area SA2 to the second target area TA2 using the robot 100. The robot characteristic data RD and the work attribute data WD are prepared in advance before the control program creation process described later. The task description list WDL and the robot control program RP are created by the control program creation process.

[0021] 3 is a flowchart showing the procedure of the control program creation process executed by the processor 310. The control program creation process starts when the operator TP inputs an instruction to start a teaching operation into the information processing device 300. Steps S10 to S40 described below correspond to the teaching operation in which the operator TP performs teaching. However, in the following description, the term "operation" simply means the operation of moving a workpiece.

[0022] In step S10, the first workpiece WK1 and the movement of the worker TP are photographed in the first working area WA1 for the worker using the first camera 210 and the third camera 230. In step S20, the object recognition unit 311 recognizes the first workpiece WK1 present in the first working area WA1 from the image photographed by the first camera 210 or the third camera 230.

[0023] 4 is an explanatory diagram showing examples of image frames MF001 and MF600 capturing the first workpiece WK1 in the first working area WA1. The upper image frame MF001 is an image before the worker TP moves the first workpiece WK1, and the lower image frame MF600 is an image after the worker TP moves the first workpiece WK1.

[0024] In the image frame MF001 before the moving operation, a plurality of first workpieces WK1a and WK1b are arranged in the first supply area SA1, and no workpieces are arranged in the first target area TA1. In this example, two types of first workpieces WK1a and WK1b are arranged in the first supply area SA1. As the first workpiece WK1, only one type of part may be used, or N types of parts may be used, where N is an integer equal to or greater than 2. When N types of parts are used, the workpiece attribute data WD includes data representing the type and shape of each of the N types of parts. The object recognition unit 311 refers to this workpiece attribute data WD to recognize the type and position and orientation of the first workpieces WK1a and WK1b from the image frame MF001. Around these first workpieces WK1a and WK1b, frame lines surrounding each workpiece are drawn. The color and shape of these frame lines change depending on the type of the recognized workpiece. The worker TP can distinguish the type of each workpiece by observing the frame lines drawn around each workpiece. However, these frame lines can be omitted. In the image frame MF001, coordinate axes U and V of an image coordinate system indicating positions within the image frame MF001 are drawn. In the image frame MF600 after the movement operation, the multiple first workpieces WK1a and WK1b have moved from the first supply area SA1 into the first target area TA1. The object recognition unit 311 recognizes the type and position and orientation of the first workpieces WK1a and WK1b from this image frame MF600 as well.

[0025] FIG. 5 is an explanatory diagram showing the recognition result for the first workpiece WK1. In each record of the recognition result, an image frame number, a workpiece ID, a workpiece type ID, an image coordinate point, and a reference coordinate system position and orientation are registered. The recognition result of the workpiece is time-series data in which records are arranged in chronological order. In the example of FIG. 5, the recognition results of the two first workpieces WK1a and WK1b are registered for the image frame MF001 before the moving operation, and the recognition results of the two first workpieces WK1a and WK1b are also registered for the image frame MF600 after the moving operation. The "workpiece ID" is an identifier for distinguishing each workpiece. The "workpiece type ID" is an identifier for indicating the type of workpiece. The "image coordinate point" is a value that expresses the representative point of each workpiece in image coordinates (U, V). For example, the center of gravity of the workpiece or the upper left point of the frame surrounding the workpiece shown in FIG. 4 can be used as the representative point of the workpiece. However, the image coordinate point may be omitted. The "reference coordinate system position and orientation" is a value expressing the position and orientation of the workpiece in a reference coordinate system, which is a robot-independent coordinate system that does not depend on the robot 100. In this disclosure, the camera coordinate system Σc1 of the first camera 210 is used as the reference coordinate system. However, other coordinate systems may be used as the reference coordinate system. Among the reference coordinate system position and orientation, the parameters θx, θy, and θz indicating the orientation or rotation indicate the rotation angles around three axes, respectively. Note that, instead of the rotation angles, any expression such as a rotation matrix or quaternion indicating rotation can be used to express the parameters indicating the orientation or rotation.

[0026] The object recognition unit 311 recognizes the work before and after the work, and when a change occurs in the position and orientation of the work, and the recognition results are stored as time-series data. During the work, it is preferable to execute object recognition only when a change occurs in the position and orientation of the work. This can reduce the processing load on the processor 310 and also reduce the resources required for processing. Note that, if only the position of the object after the work is used in the robot control program, the object recognition unit 311 may recognize the object only after the work.

[0027] In step S30 of FIG. 3, the action recognition unit 312 recognizes the worker's action from the image captured by the first camera 210.

[0028] FIG. 6 is an explanatory diagram showing an example of an image frame capturing a worker's action. Here, three image frames MF200, MF300, and MF400, which are part of a plurality of image frames captured in time series, are superimposed. In the image frame MF200, the worker TP stretches out his arm AM and grasps the first workpiece WK1a in the first supply area SA1. The action recognition unit 312 sets a bounding box BB that surrounds the arm AM and the first workpiece WK1a in this image frame MF200. The same is true for the other image frames MF300 and MF400.

[0029] The bounding box BB can be used for example for the following purposes: (1) To perform contact detection on the image using the results of workpiece recognition and hand position recognition. (2) To identify the gripping position on the image using the workpiece recognition results and the hand position recognition results. (3) Draw a bounding box BB on the image to show that the arm AM is correctly recognized.

[0030] FIG. 7 is an explanatory diagram showing the recognition result of a worker's motion. In each record of the recognition result, for each worker's motion included in the work, the image frame number, individual ID, motion number, motion name, and the top left point position and bottom right point position of the bounding box BB are registered. The recognition result of the worker's motion is also time-series data in which the records are arranged in chronological order. The "individual ID" is an identifier that distinguishes the arms AM. For example, when the right arm and the left arm appear in the image, separate individual IDs are assigned. The top left point position and bottom right point position of the bounding box BB are expressed as positions in the camera coordinate system Σc1 as the reference coordinate system.

[0031] The "motion name" indicates the type of the worker motion in the image frame. In the example of FIG. 7, a pick motion is recognized in the image frame MF200, a place motion is recognized in the image frame MF300, and a pointing motion is recognized in the image frame MF400. These motions can be recognized by analyzing a plurality of consecutive image frames. The pointing motion means a pointing motion using the index finger. The pointing motion can be used to set a teaching point at the tip of the index finger or to recognize a workpiece on a straight line extending along the joints of the index finger as a conveyance target. A specific motion of the fingers other than those described above may be used as a motion for instructing a specific motion of the robot. For example, a method of gripping the workpiece may be instructed by a finger gesture.

[0032] In addition, since a normal task includes multiple worker actions, multiple worker actions are recognized in step S30. However, a task can also be composed of one or more worker actions. Therefore, in step S30, one or more worker actions included in the task related to the work are recognized.

[0033] The recognition process of the worker's motion in step S30 may be performed using the "SlowFast Networks for Video Recognition" technology. This technology recognizes motion using a first processing result obtained by inputting a first group of image frames extracted in a first period from among a plurality of image frames into a first neural network, and a second processing result obtained by inputting a second group of image frames extracted in a second period longer than the first period from among the plurality of image frames into a second neural network. By using such a technology, the worker's motion can be recognized more accurately.

[0034] In step S40, hand and finger position recognition section 313 recognizes the hand and finger position from the image captured by first camera 210 or third camera 230.

[0035] FIG. 8 is a flowchart showing the detailed procedure of step S40. In step S41, the hand position recognition unit 313 reads a plurality of image frames captured by the first camera 210 or the third camera 230. In step S42, the hand position recognition unit 313 recognizes the motion of the fingers in the plurality of image frames. In step S43, it is determined whether the recognized motion of the fingers corresponds to a specific hand motion. The "specific hand motion" is a motion involving the movement of the joints of the fingers, and is a motion designated in advance by the worker TP. As the specific hand motion, for example, one or more of a gripping motion by the fingers, a releasing motion by the fingers, and a pointing motion by the fingers are designated. In this embodiment, the pick motion corresponds to the "grip motion by the fingers", the place motion corresponds to the "releasing motion by the fingers", and the pointing motion corresponds to the "pointing motion by the fingers". If the finger movement corresponds to a specific finger movement, the finger position recognition unit 313 recognizes the finger position in step S44, and the process proceeds to step S45, which will be described later. The recognition result of the finger position will be described later. If the finger movement does not correspond to a specific finger movement, the process in FIG. 8 is terminated without executing the processes in and after step S44. In other words, if the worker movement does not include a specific finger movement, the process of recognizing the finger position is not performed. In this way, the process of recognizing the finger position is performed only when the worker movement includes a specific finger movement, and the processing load can be reduced.

[0036] In step S45, it is determined whether the specific hand motion is a pointing motion. If it is not a pointing motion, the processing in FIG. 8 is terminated. On the other hand, if the specific hand motion is a pointing motion, in step S46, the hand position recognition unit 313 estimates the pointing direction from multiple image frames. In step S47, the hand position recognition unit 313 identifies the workpiece to be pointed at from the multiple image frames. In step S48, the hand position recognition unit 313 identifies the pointing position as a position indicating the direction of the workpiece identified in step S47. This pointing position is additionally registered in the recognition result of the hand position. Note that the processing in steps S45 to S48 may be omitted.

[0037] 9 is an explanatory diagram showing how the finger position is recognized. Here, in the image frame MF200 shown in FIG. 6, a plurality of reference points JP are specified on the arm AM and fingers of the worker TP. The plurality of reference points JP are connected by links JL. The reference points JP are set at the tips and joints of the fingers, respectively. These reference points JP and links JL are the results of recognition by the finger position recognition unit 313.

[0038] 10 is an explanatory diagram showing the reference points JP of the finger positions to be recognized. Here, the following are set as the reference points JP of the finger positions to be recognized. (1) Tip of thumb JP10 and joint points JP11-JP13 (2) Tip of index finger JP20 and joint points JP21-JP23 (3) Tip of middle finger JP30 and joint points JP31-JP33 (4) Tip of ring finger JP40 and joint points JP41-JP43 (5) Tip of thumb JP50 and joint points JP51-JP53 (6) Wrist joint point JP60 Some or all of these reference points are used as the finger positions recognized by the finger position recognition unit 313. In order to accurately recognize the finger position, it is preferable to use all of the above-mentioned reference points as recognition targets, but from the viewpoint of reducing the processing load, it is preferable to use at least the tip JP10 of the thumb and the tip JP20 of the index finger as recognition targets.

[0039] FIG. 11 is an explanatory diagram showing the recognition result of the finger position. In each record of the recognition result, an image frame number, an individual ID, a finger position ID, a hand designation, image coordinate points of the finger position, and a reference coordinate system position of the finger are registered. The recognition result of the finger position is also time-series data in which records are arranged in chronological order. The "individual ID" is an identifier for distinguishing the arm AM. The "finger position ID" is an identifier for distinguishing the reference points shown in FIG. 10. As the "hand designation", the name of a specific finger to be recognized by the finger position recognition unit 313 is registered. Here, the thumb and index are registered as specific fingers. The reference point JP10 of the tip of the thumb is registered, and the reference point JP20 of the tip of the index is registered. It is preferable that the other reference points described in FIG. 10 are similarly registered. The image coordinate points and the reference coordinate system positions of the fingers indicate the individual finger positions. However, the image coordinate points may be omitted.

[0040] In addition, in the above-mentioned FIG. 8, when steps S45 to S48 are executed and the pointing position in the pointing motion is specified, the pointing position is additionally registered as a recognition result of the hand and finger position.

[0041] The execution order of steps S20 to S40 described above can be changed arbitrarily. Moreover, the image used to recognize the worker's motion in step S30 and the image used to recognize the hand and finger position in step S40 may be images taken by different cameras. If the hand and finger position is photographed using a camera different from the camera that photographs the worker's motion, the hand and finger position can be recognized more accurately. Furthermore, the image used to recognize the worker's motion in step S30 and the image used to recognize the workpiece in step S20 may also be images taken by different cameras. If the workpiece is photographed using a camera different from the camera that photographs the worker's motion, the workpiece can be recognized more accurately.

[0042] 3, the work description list creation unit 314 creates a work description list WDL using the recognition results up to that point. The work description list WDL is time-series data that describes work in a robot-independent coordinate system that is not dependent on the type of robot.

[0043] FIG. 12 is an explanatory diagram showing the work description list WDL. In each record of the work description list WDL, a record number, an image frame number, an action name, a workpiece ID, a workpiece position and posture, an arm tip position and posture, and a gripping position are registered for each action included in the work. The "action name" is the type of each action. In the example of FIG. 12, five actions, approach, pick, depart, approach, and place, are registered in this order for the same workpiece WK1a. The approach action and the depart action are not included in the worker actions described in FIG. 7, but are necessary actions as action commands of the robot control program, and therefore are added by the worker list creation unit 314 as actions performed before and after the pick action and the place action.

[0044] The "arm tip position and posture" is the position and posture of the tip of the robot arm in each operation, and is calculated from the recognition result of the finger position shown in FIG. 11. For example, the "arm tip position and posture" can be determined as follows. For a pick operation, the position where the object and the fingertips of the hand come into contact is obtained as the gripping position from the recognition result of the finger position at the time of recognizing the pick operation, and coordinate transformation is performed with the reference coordinate system as the origin. Then, from this gripping position, the "arm tip position and posture" is calculated as a value indicating the tip position of the robot arm. At this time, it is preferable to determine the posture of the arm tip by taking into account the posture of the workpiece. The optimal arm tip position and posture may differ depending on the end effector used in the actual work. For example, the arm tip position and posture in a pick operation or a place operation using a gripper can be obtained as the center of gravity of multiple gripping positions. The position of the arm tip in the approach motion is set to a position that is a predetermined distance above the position of the arm tip in the previous or next pick or place motion, or a position where the finger position has moved a predetermined distance from the position where the pick or place motion was performed, or a position where the finger position has moved a predetermined time from the point where the pick or place motion was performed. The same is true for the position of the arm tip in the depart motion.

[0045] The "grasping position" is the finger position in each operation, and is calculated from the recognition result of the finger position shown in FIG. 11. In the example of FIG. 12, the position of the reference point JP10 of the tip of the thumb and the position of the reference point JP20 of the tip of the index finger are registered. Other reference points may be registered in the same manner, but it is preferable to register at least the positions of the reference point JP10 of the tip of the thumb and the reference point JP20 of the tip of the index finger. In addition, the "grasping position" is registered only when the workpiece is grasped by the fingers or the grip of the workpiece is released. In the example of FIG. 12, the "grasping position" is registered only during the pick operation and the place operation, and the "grasping position" is not registered during the approach operation or the depart operation.

[0046] All positions and postures registered in the work description list WDL are expressed in a reference coordinate system as a robot-independent coordinate system. Since this work description list WDL describes the work in a robot-independent coordinate system, a robot control program suitable for any type of robot can be easily created from this work description list WDL. In this way, the work description list WDL is a list in which the work is divided into units equivalent to one robot movement, and one movement is expressed by one line of data. It is preferable that the work description list WDL does not include a path plan. In other words, it is preferable that only waypoints that are the starting points of the robot movements extracted from the worker's movements are registered in the work description list WDL.

[0047] 3, the control program creation unit 315 receives input of a robot type. This robot type indicates the type of robot for which a robot control program is to be created, and is input by the worker TP.

[0048] In step S70, the second working area WA2 for the robot is photographed using the second camera 220. In step S80, the object recognition unit 311 recognizes the second workpiece WK2 present in the second working area WA2 from the image photographed by the second camera 220. At this time, the second workpiece WK2 is placed in the second supply area SA1 and is in the position before the movement operation.

[0049] In step S90, the control program creation unit 315 creates a robot control program according to the type of robot, using the work description list WDL created in step S50 and the position of the second work WK recognized in step S80. At this time, the position of the second work WK recognized in step S80 is used as the position of the work before the work. Also, the position of the work after the work registered in the work description list WDL is used as the position of the work after the work. However, if the second supply area SA2 shown in FIG. 1 is an area where the position of the second work WK2 is indefinite, steps S70 and S80 may be omitted and the robot control program may be created without using the position of the second work WK. In this case, the robot control program is written to pick the work recognized by the second camera 220 when the actual work is performed. Also, if the second supply area SA2 is an area where the second work WK2 to be picked is placed in a fixed position like a parts feeder, steps S70 and S80 may be omitted and the robot control program may be created without using the position of the second work WK.

[0050] In the robot control program, the operations registered in the work description list WDL are converted into commands and expressions according to the type of robot. In addition, since the position and posture are expressed in the robot coordinate system Σr in the robot control program RP, the position and posture expressed in the reference coordinate system Σc1 in the work description list WDL are converted to the robot coordinate system Σr by coordinate transformation. The transformation matrix that performs the coordinate transformation between the reference coordinate system Σc1 and the robot coordinate system Σr is known.

[0051] To create a robot control program, a correspondence table between commands in a robot control program language for various robots and work contents may be prepared in advance and registered in the memory 320. In this case, the control program creation unit 315 can execute rule-based processing, such as referring to this correspondence table, selecting a command for an operation registered in the work description list WDL, and providing the position and orientation registered in the work description list WDL as an argument to perform coordinate conversion.

[0052] In the work description list WDL shown in FIG. 12, gripping positions using multiple fingers are registered as "grip positions", so if the end effector actually used has multiple fingers that grip a workpiece, it is possible to describe the positions of those fingers in the robot control program. Also, if the end effector actually used does not have fingers and is, for example, a suction hand that picks up a workpiece, it is possible to describe the position and posture of the end effector using the "arm tip position and posture" without using the "grip position". As can be seen from these examples, in this embodiment, since the "arm tip position and posture" and the "grip position" are described in the work description list WDL, it is possible to create a robot control program suitable for the robot and end effector actually used.

[0053] As described above, in the above embodiment, the hand and finger positions are recognized when the worker's motion includes a specific finger motion accompanied by the movement of the finger joints, so that the processing load can be reduced compared to the case where the hand and finger positions are constantly recognized. Also, in the above embodiment, a work description list WDL that describes the work in a robot-independent coordinate system is created, and then a robot control program RP suitable for the type of robot is created from the work description list WDL, so that a control program for executing the work using any one of multiple types of robots can be easily created. However, the robot control program RP may be created from the recognition results of the worker's motion, the recognition results of the hand and finger positions, and the recognition results of the workpiece, without creating the work description list WDL.

[0054] In the above embodiment, an example of a pick-and-place operation has been described, but the present disclosure can be applied to other operations, such as a coating operation including a pointing motion, a screwdriver operation, a nail driving operation with a hammer, a workpiece insertion operation, a fitting operation, and an assembly operation.

[0055] Other embodiments: The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various forms without departing from the spirit of the present disclosure. For example, the present disclosure can be realized in the following aspects. The technical features in the above-mentioned embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined in order to solve some or all of the problems of the present disclosure, or to achieve some or all of the effects of the present disclosure. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0056] (1) According to a first aspect of the present disclosure, there is provided a computer program for causing a processor to execute a process of creating a control program for a robot. This computer program causes the processor to execute the following processes: (a) a process of recognizing one or more worker actions included in an operation in which a worker manipulates a workpiece using an arm and fingers from an image captured by an imaging device, (b) a process of recognizing a finger position in a specific finger action from an image captured by the imaging device when the worker action includes a specific finger action accompanied by a movement of the joints of the finger, (c) a process of recognizing a position of the workpiece after the operation from an image captured by the imaging device, and (d) a process of generating a control program for the robot using the worker action recognized in the process (a), the finger position recognized in the process (b), and the position of the workpiece recognized in the process (c).

[0057] (2) In the above computer program, the specific finger motion includes one or more of a grasping motion by the finger, a releasing motion by the finger, and a pointing motion by the finger, The process (b) may not include a process of recognizing the hand and finger positions when the worker motion does not include the specific hand and finger motion. According to this computer program, the process of recognizing the hand and finger positions is performed only when the worker's motion includes a specific hand and finger motion, so that the process of creating the robot control program can be executed at high speed.

[0058] (3) In the above computer program, the process (d) may include: (i) a process of creating a work description list using the worker movements, the finger positions, and the position of the workpiece, to describe the work in a robot-independent coordinate system that is independent of the type of robot; and (ii) a process of creating the control program using the work description list in accordance with the type of robot controlled by the control program. This computer program creates a work description list that describes tasks in a robot-independent coordinate system, and then creates a control program suitable for the type of robot from the work description list, making it easy to create a robot control program for performing tasks using any of a number of types of robots.

[0059] (4) In the above computer program, the imaging device may include a plurality of cameras, and the image used to recognize the worker's action in process (a) and the image used to recognize the hand and finger position in process (b) may be images captured by different cameras. According to this computer program, the positions of the hands and fingers are photographed using a camera separate from the camera that photographs the worker's movements, so that the positions of the hands and fingers can be recognized more accurately.

[0060] (5) In the above computer program, the imaging device may include multiple cameras, and the image used to recognize the worker's actions in process (a) and the image used to recognize the position of the workpiece in process (c) may be images taken by different cameras. According to this computer program, the workpiece is photographed using a camera separate from the camera that photographs the worker's movements, so the position of the workpiece can be recognized more accurately.

[0061] (6) In the above computer program, the image captured by the imaging device may include a plurality of image frames, and the process (a) may be a process of recognizing the worker's action using a first processing result obtained by inputting a first group of image frames extracted from the plurality of image frames in a first period into a first neural network, and a second processing result obtained by inputting a second group of image frames extracted from the plurality of image frames in a second period longer than the first period into a second neural network. This computer program allows for more accurate recognition of worker movements.

[0062] (7) According to a second aspect of the present disclosure, there is provided a method for creating a control program for a robot, the method including: (a) recognizing one or more worker actions included in an operation in which the worker manipulates a workpiece using an arm and fingers from an image captured by an imaging device, (b) recognizing a finger position in the specific finger action from an image captured by the imaging device when the worker action includes a specific finger action accompanied by a movement of the joints of the finger, (c) recognizing a position of the workpiece after the operation from an image captured by the imaging device, and (d) generating a control program for the robot using the worker action recognized in the step (a), the finger position recognized in the step (b), and the position of the workpiece recognized in the step (c). According to this method, the hand and finger positions are recognized when the worker's motion includes a specific hand and finger motion accompanied by movement of the finger joints, so that the processing load can be reduced compared to when the hand and finger positions are constantly recognized.

[0063] (8) According to a third aspect of the present disclosure, there is provided a system for executing a process for creating a control program for a robot. The system includes an information processing device having a processor, and an imaging device connected to the information processing device. The processor executes the following processes: (a) a process for recognizing one or more worker actions included in an operation in which a worker manipulates a workpiece using an arm and fingers from an image captured by the imaging device; (b) a process for recognizing a finger position in the specific finger action from an image captured by the imaging device when the worker action includes a specific finger action involving a movement of the joints of the finger; (c) a process for recognizing a position of the workpiece after the operation from an image captured by the imaging device; and (d) a process for generating a control program for the robot using the worker action recognized in the process (a), the finger position recognized in the process (b), and the position of the workpiece recognized in the process (c). According to this system, the hand and finger positions are recognized when the worker's motion includes a specific hand and finger motion accompanied by movement of the finger joints, so the processing load can be reduced compared to when the hand and finger positions are constantly recognized.

[0064] The present disclosure may be realized in various forms other than those described above, such as a robot system including a robot and a robot control device, a computer program for implementing the functions of the robot control device, a non-transitory storage medium on which the computer program is recorded, etc. [Explanation of symbols]

[0065] 100...robot, 210...first camera (imaging device), 220...second camera (imaging device), 230...third camera (imaging device), 300...information processing device, 310...processor, 311...object recognition unit, 312...motion recognition unit, 313...hand position recognition unit, 314...task description list creation unit, 315...control program creation unit, 320...memory, 330...interface circuit, 340...input device, 350...display unit

Claims

1. A computer that causes a processor to execute the process of creating a control program for the robot A program, (a) One or more worker movements involved in a task in which a worker manipulates a workpiece using his / her arms and fingers A process of recognizing the worker's action from an image captured by an imaging device; (b) when the worker motion includes a specific finger motion accompanied by a movement of the joints of the fingers, A process for recognizing a finger position in the specific finger motion from an image of the finger captured by the imaging device. Reason and (c) determining from an image of the workpiece captured by the imaging device the state of the workpiece after the operation; A process of recognizing the position of (d) the worker's motion recognized in the process (a) and the worker's motion recognized in the process (b). Using the hand and finger positions and the position of the work recognized in the process (c), A process of generating a control program; causing said processor to execute The specific finger motion includes a gripping motion by the fingers, a releasing motion by the fingers, and and a pointing gesture using a finger; In the process (b), when the worker motion does not include the specific hand motion, A computer program that does not perform location-aware processing.

2. 2. The computer program of claim 1, The process (d) is (i) Using the worker's motion, the finger position, and the work position, a robot type is determined. creating a task description list describing said tasks in a robot-independent coordinate system; , (ii) The operation record is generated according to the type of the robot controlled by the control program. creating the control program using the description list; A computer program comprising:

3. A computer program according to any one of claims 1 to 2, The imaging device includes a plurality of cameras; The image used to recognize the worker's action in the process (a), The image used to recognize the hand and finger position in process (b) is taken by a different camera. A computer program that is a projected image.

4. A computer program according to any one of claims 1 to 2, The imaging device includes a plurality of cameras; The image used to recognize the worker's action in the process (a), The image used to recognize the position of the workpiece in process (c) is captured by a different camera. A computer program that is an image taken with a

5. A computer program according to any one of claims 1 to 2, The image captured by the imaging device includes a plurality of image frames, The process (a) comprises: A first image frame group extracted from the plurality of image frames in a first period is treated as a first new a first processing result obtained by inputting the first processing result into the neural network; A second image frame is extracted from the plurality of image frames at a second period longer than the first period. The frame group is input to the second neural network, and the second processing result is obtained. A computer program for processing a user's actions.

6. A method for creating a control program for a robot, comprising the steps of: (a) One or more worker movements involved in a task in which a worker manipulates a workpiece using his / her arms and fingers recognizing the worker's action from an image captured by an imaging device; (b) when the worker motion includes a specific finger motion accompanied by a movement of the joints of the fingers, A step of recognizing a position of a finger in the specific finger motion from an image of the finger captured by the imaging device. With (c) determining from an image of the workpiece captured by the imaging device the state of the workpiece after the operation; Recognizing the position of (d) determining whether the worker's motion recognized in the step (a) is a motion of the worker that is recognized in the step (b) or not; Using the hand and finger positions and the position of the work recognized in the step (c), generating a control program; Including, The specific finger motion includes a gripping motion by the fingers, a releasing motion by the fingers, and and a pointing gesture using a finger; In the step (b), when the worker motion does not include the specific hand and finger motion, The method does not include a step of recognizing a position.

7. A system for executing a process for creating a control program for a robot, An information processing device having a processor; an imaging device connected to the information processing device; Equipped with The processor, (a) One or more worker movements involved in a task in which a worker manipulates a workpiece using his / her arms and fingers A process of recognizing the worker's action from an image captured by the imaging device; (b) when the worker motion includes a specific finger motion accompanied by a movement of the joints of the fingers, A process for recognizing a finger position in the specific finger motion from an image of the finger captured by the imaging device. Reason and (c) determining from an image of the workpiece captured by the imaging device the state of the workpiece after the operation; A process of recognizing the position of (d) the worker's motion recognized in the process (a) and the worker's motion recognized in the process (b). Using the hand and finger positions and the position of the work recognized in the process (c), A process of generating a control program; Run The specific finger motion includes a gripping motion by the fingers, a releasing motion by the fingers, and and a pointing gesture using a finger; In the process (b), when the worker motion does not include the specific hand motion, A system that does not perform location-aware processing.

Images