Computer program, generation method, and generation device

Abstract

To reduce a calculation amount necessary for generating a robot path.SOLUTION: A computer program having a computer perform processing to generate a robot path causes the computer to perform: range identifying processing which obtains a moving range where an operator's joints move during a period that specific work is performed from a plurality of images photographing the operator's arm and fingers in succession with an imaging device while the operator is performing specific work operating a work-piece by using the arm and fingers; space calculating processing calculating a movable space where a robot can move on the basis of the moving range; and path generation processing searching for a path where the robot moves within a movable space and generating a path in order to cause the robot to perform specific operation.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a computer program, a method, and a program generation device. [Background technology]

[0002] Various ideas have been devised to reduce the amount of calculation required to generate a program to control a robot. For example, in the technology described in Patent Document 1, when changing a trajectory indicating the path along which a robot moves, the entire trajectory of the robot is divided into a plurality of unit trajectories, which are obtained by dividing the entire trajectory into a plurality of sections, and the unit trajectories are divided into those that require correction and those that do not, based on the relationship between obstacles and the unit trajectories, and only the unit trajectories that require correction are corrected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-62690 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, depending on the relationship between an obstacle and a unit trajectory, corrections may be required for multiple unit trajectories, and the amount of calculations may not always be reduced. Therefore, a method that can efficiently reduce the amount of calculations has been desired. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a computer program that causes a computer to execute a process for generating a path for a robot. The computer program causes the computer to execute a range determination process for determining a movement range of the joints of a worker while the worker is performing a specific task of manipulating a workpiece using the worker's arm and fingers, from a plurality of images continuously captured by an imaging device of the worker's arm and fingers, a space calculation process for calculating a movable space in which the robot can move based on the movement range, and a path generation process for searching for a path for the robot to move within the movable space and generating the path so that the robot can perform the specific task.

[0006] According to another aspect of the present disclosure, there is provided a method for generating a path for a robot by a computer, the method including causing a computer to execute the following steps: determining a range of movement of joints of a worker while the worker is performing a specific task of manipulating a workpiece using the worker's arm and fingers from a plurality of images continuously captured by an imaging device of the worker's arm and fingers while the specific task is being performed; calculating a movable space in which the robot can move based on the range of movement; and searching for a path for the robot to move within the movable space and generating the path to have the robot perform the specific task.

[0007] According to yet another aspect of the present disclosure, there is provided a generation device for generating a path for a robot, the generation device including: a receiving unit that receives a plurality of images of a worker's arm and fingers captured by an imaging device while the worker is performing a specific task of manipulating a workpiece using the arm and fingers; a range identifying unit that determines, from the plurality of images, a movement range of the worker's joints during the specific task; a space calculating unit that calculates a movable space within which the robot can move based on the movement range; and a path generating unit that searches for a path for the robot to move within the movable space and generates the path so that the robot can perform the specific task. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing the overall configuration of a robot system. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of the information processing device. [Figure 3] 10 is a flowchart of a control program creation process. [Figure 4] FIG. 10 is a diagram showing an example of two image frames obtained by photographing a workpiece. [Figure 5] FIG. 10 is a diagram showing a recognition result of a workpiece by an object recognition unit. [Figure 6] FIG. 10 is a diagram showing an example of an image frame capturing a worker's action. [Figure 7] 10A and 10B are diagrams illustrating the results of recognition of a worker's action by the action recognition unit. [Figure 8] 10 is a flowchart showing the detailed procedure of step S40 of the control program creation process. [Figure 9] FIG. 10 is a diagram showing how a joint position is recognized. [Figure 10] FIG. 10 is a diagram showing reference points included in the range of the fingers of the upper limb. [Figure 11] FIG. 1 illustrates reference points included in the range of the upper limbs. [Figure 12] 10A and 10B are diagrams illustrating the recognition results of joint positions by a joint position recognition unit. [Figure 13] FIG. 10 is a diagram showing a work description list. [Figure 14] FIG. 1 is a diagram showing the position of a worker's arm changing over time. [Figure 15] FIG. 10 is a diagram for explaining calculation of a movable space. [Figure 16] FIG. 10 is a diagram for explaining calculation of a movable space. [Figure 17] FIG. 10 is a diagram showing a constraint list. [Figure 18] FIG. 10 is a diagram for explaining a search for a path for each joint of a robot. [Figure 19]FIG. 10 is a diagram showing another example of a set movable space. DETAILED DESCRIPTION OF THE INVENTION

[0009] A1. Implementation: FIG. 1 is a schematic diagram showing the overall configuration of a robot system 1000 according to this embodiment. In the robot system 1000, the robot is taught to perform a task based on a human demonstration. In this embodiment, the operation taught to the robot 100 is a pick-and-place operation to move a workpiece. A worker TP demonstrates the task that the robot 100 is to perform, and a control program is generated from image data obtained by capturing the worker TP's actions. The robot system 1000 includes the robot 100, a first camera 210, a second camera 220, a third camera 230, and an information processing device 300.

[0010] The robot 100 is a multi-axis robot having multiple joints and is equipped with a gripper that can hold a workpiece as an effector.

[0011] The first camera 210 and the third camera 230 capture images of the worker TP demonstrating the work. The second camera 220 captures images of the robot 100 performing the work.

[0012] The first camera 210, the second camera 220, and the third camera 230 have the function of recognizing a subject in three dimensions. For example, a stereo camera or an RGBD camera capable of simultaneously capturing a color image and a depth image can be used as these cameras. The first camera 210, the second camera 220, and the third camera 230 are also referred to as imaging devices. The first camera 210, the second camera 220, and the third camera 230 also have the function of capturing a subject as a video or multiple image frames. The first camera 210, the second camera 220, and the third camera 230 each output the captured image data to the information processing device 300.

[0013] In the robot system 1000 shown in FIG. 1, a first working area WA1 where the worker TP performs work and a second working area WA2 where the robot 100 performs work are defined. The first working area WA1 is a three-dimensional space on a working table TB1. For example, the first working area WA1 is defined as a space formed by the upper surface of the working table TB1 and the height direction. The upper surface of the working table TB1 is defined as a horizontal surface, and the height direction is defined as a direction perpendicular to the upper surface of the working table TB1. The height direction of the first working area WA1 is defined as a predetermined distance. The second working area WA2 is a three-dimensional space on a working table TB2. For example, the second working area WA2 is defined as a space formed by the upper surface of the working table TB2 and the height direction. The upper surface of the working table TB2 is defined as a horizontal surface, and the height direction is defined as a direction perpendicular to the upper surface of the working table TB2. The height direction of the second working area WA2 is defined as a predetermined distance.

[0014] The first camera 210 captures the work demonstration of the worker TP in the first working area WA1. The second camera 220 captures the operation of the robot 100 in the second working area WA2. The relative position of the first camera 210 to the working table TB1 and the relative position of the second camera 220 to the working table TB2 are set to be the same.

[0015] The third camera 230 captures the work demonstration of the worker TP in the first working area WA1. The third camera 230 is placed closer to the first working area WA1 than the first camera 210 so that it can capture the hands and fingers of the worker TP and the workpiece closer than the first camera 210.

[0016] A first supply area SA1 and a first target area TA1 are provided on the upper surface of the work table TB1. The first supply area SA1 is an area where the first workpiece WK1 is placed at the start of the demonstration. The first target area TA1 is an area where the first workpiece WK1 is placed by the operation of the worker TP. The worker TP moves the first workpiece WK1 from the first supply area SA1 to the first target area TA1. The shapes and positions of the first supply area SA1 and the first target area TA1 within the first work area WA1 can be set arbitrarily.

[0017] A second supply area SA2 and a second target area TA2 are provided on the upper surface of the work table TB2. The second supply area SA2 has the same shape and size as the first working area WA1. The second target area TA2 has the same shape and size as the first target area TA1. The second supply area SA2 is an area where the second workpiece WK2 is placed when the robot 100 starts working. The second target area TA2 is an area where the second workpiece WK2 is placed by the work of the robot 100. The robot 100 moves the second workpiece WK2 from the second supply area SA2 to the second target area TA2. Note that the first supply area SA1, the second supply area SA2, the first target area TA1, and the second target area TA2 do not necessarily have to be explicitly separated. The first workpiece WK1 to be worked on in the first working area WA1 and the second workpiece WK2 to be worked on in the second working area WA2 are the same type of object with the same design.

[0018] FIG. 1 illustrates a robot coordinate system Σr set for the robot 100, a first camera coordinate system Σc1 set for the first camera 210, a second camera coordinate system Σc2 set for the second camera 220, and a third camera coordinate system Σc3 set for the third camera 230. The robot coordinate system Σr, the first camera coordinate system Σc1, the second camera coordinate system Σc2, and the third camera coordinate system Σc3 are all Cartesian coordinate systems defined by three axes X, Y, and Z. The correspondence relationships between the robot coordinate system Σr, the first camera coordinate system Σc1, the second camera coordinate system Σc2, and the third camera coordinate system Σc3 are determined by calibration. In this embodiment, the first camera coordinate system Σc1 is used as a reference coordinate system. The position and posture of the worker TP are expressed in the reference coordinate system.

[0019] The information processing device 300 creates a control program for controlling the robot 100 based on image data obtained by capturing a demonstration of a task performed by the worker TP. The information processing device 300 is, for example, a personal computer.

[0020] 2 is a block diagram showing the functional configuration of the information processing device 300. The information processing device 300 includes a processor 310, a memory 320, an interface circuit 330, an input device 340, and a display unit 350.

[0021] The processor 310 realizes various functions by executing programs stored in the memory 320. The memory 320 stores programs and data used for various processes executed by the information processing device 300.

[0022] In this embodiment, the memory 320 stores robot characteristic data RD, workpiece attribute data WD, a task description list WDL, a constraint list CL, and a robot control program RP. The robot characteristic data RD is data including characteristics of the robot 100, such as the geometric structure, joint rotation angles, weight, and inertia value. The workpiece attribute data WD is data including attributes such as the type and shape of each of the first workpiece WK1 and the second workpiece WK2. The workpiece attribute data WD includes attributes such as type and shape for all types of parts that are the target of the task.

[0023] The work description list WDL is data representing the work content recognized from a video or multiple image frames of the worker TP's movements and the first workpiece WK1, and describes the work in a robot-independent coordinate system that is independent of the type of robot. The work description list WDL is also called an action list. The constraint list CL is data representing constraints on the robot's movements. The robot control program RP is composed of multiple commands for operating the robot 100. For example, the robot control program RP controls a pick-and-place operation that uses the robot 100 to move the second workpiece WK2 from the second supply area SA2 to the second target area TA2. The robot characteristic data RD and workpiece attribute data WD are prepared before the control program creation process described below. The work description list WDL, constraint list CL, and robot control program RP are created by the control program creation process.

[0024] An input device 340, a display unit 350, a first camera 210, a second camera 220, and a third camera 230 are connected to the interface circuit 330. The input device 340 is, for example, a keyboard or a mouse. The display unit 350 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0025] The information processing device 300 functionally includes an image receiving unit 311, an object recognition unit 312, an action recognition unit 313, a joint position recognition unit 314, a task description list creation unit 315, a movement space calculation unit 316, a constraint generation unit 317, and a control program creation unit 318. The functions of the image receiving unit 311, the object recognition unit 312, the action recognition unit 313, the joint position recognition unit 314, the task description list creation unit 315, the movement space calculation unit 316, the constraint generation unit 317, and the control program creation unit 318 are realized by the processor 310 executing a computer program stored in the memory 320. The joint position recognition unit 314 is also referred to as a range identification unit. The movement space calculation unit 316 is also referred to as a space calculation unit.

[0026] Image receiving unit 311 receives images captured by first camera 210, second camera 220, and third camera 230. Image receiving unit 311 is also referred to as a receiving unit.

[0027] The object recognition unit 312 recognizes the position and orientation of the first workpiece WK1 from images of the first working area WA1 captured by the first camera 210 and the third camera 230. Hereinafter, the position and orientation will be referred to as the position and orientation. The object recognition unit 312 also recognizes the position and orientation of the second workpiece WK2 from images of the second working area WA2 captured by the second camera 220. The object recognition unit 312 can recognize the respective positions and orientations of the first workpiece WK1 and the second workpiece WK2, for example, using a feature extraction model that has been trained by machine learning.

[0028] The action recognition unit 313 recognizes the action of the worker TP from the images of the first working area WA1 captured by the first camera 210 and the third camera 230. The action recognition unit 313 can recognize the action of the worker TP from the images captured by the first camera 210 and the third camera 230, for example, by using a feature extraction model that has been trained by machine learning.

[0029] The joint position recognition unit 314 recognizes the positions of multiple joints included in the upper limbs of the worker TP from the images captured by the first camera 210 and the third camera 230. In the embodiment, the upper limbs include the part of the worker TP from the shoulders to the fingers. The joint position recognition unit 314 can recognize the positions of multiple joints included in the upper limbs of the worker TP from the images using, for example, a feature extraction model that has been trained by machine learning.

[0030] The work description list creation unit 315 creates a work description list WDL using the recognition results from the object recognition unit 312, the action recognition unit 313, and the joint position recognition unit 314. The movement space calculation unit 316 calculates a movable space using the recognition results output by the joint position recognition unit 314. The movable space will be described later. The constraint condition generation unit 317 generates constraint conditions including the determined movable space. The constraint conditions will be described later. The control program creation unit 318 creates a control program for the robot 100 using the work description list WDL.

[0031] 3 is a flowchart of the control program creation process executed by the processor 310. The control program creation process starts when the worker TP inputs an instruction to start a work demonstration into the information processing device 300. As described above, the worker TP teaches the robot 100 a pick-and-place operation to move a workpiece. In the following description, the term "work" refers to the worker TP moving the first workpiece WK1 from the first supply area SA1 to the first target area TA1.

[0032] In step S10, the worker TP performs work in the first work area WA1, and under the control of the processor 310, the first camera 210 and the third camera 230 capture images of the first workpiece WK1 and the worker TP's movements in the first work area WA1. For example, the first camera 210 and the third camera 230 capture images of the first workpiece WK1 and the worker TP's movements at predetermined time intervals from the start to the end of the work. The images captured by the first camera 210 and the third camera 230 are also referred to as work images. The first camera 210 and the third camera 230 sequentially transmit the captured images to the information processing device 300. The image receiving unit 311 of the information processing device 300 receives the work images from the first camera 210 and the third camera 230. The image receiving unit 311 stores the work images in memory 320 in the order of their capture time. The work images are also stored in memory 320 sorted by sender. The work images include information on the capture time and sender.

[0033] In step S20, the object recognition unit 312 recognizes the first workpiece WK1 from the work images captured by the first camera 210 and the third camera 230.

[0034] Fig. 4 shows examples of image frames MF001 and MF600 capturing images of the first workpiece WK1 in the first working area WA1. The image frame MF001 shown in the upper part of Fig. 4 is an image taken before the worker TP moves the first workpiece WK1. The image frame MF600 shown in the lower part of Fig. 4 is an image taken after the worker TP moves the first workpiece WK1.

[0035] In the image frame MF001 before the movement operation, two first workpieces WK1a and WK1b are placed in the first supply area SA1, and no first workpieces WK1a and WK1b are placed in the first target area TA1. In this example, two types of first workpieces WK1a and WK1b are placed in the first supply area SA1. Note that only one type of part may be used as the first workpiece WK1, or two or more types of parts may be used.

[0036] The object recognition unit 312 references the work attribute data WD to recognize the type and position and orientation of the first workpieces WK1a and WK1b from the image frame MF001 before the movement operation. Based on the recognition results of the positions and orientations of the first workpieces WK1a and WK1b, the object recognition unit 312 displays an image frame MF001 on the display unit 350, in which frame lines surrounding each of the first workpieces WK1a and WK1b are drawn. The color and shape of these frame lines may vary depending on the type of workpiece recognized. In this case, the worker TP can distinguish the type of each workpiece by observing the frame lines drawn around each workpiece. An image coordinate system indicating the position within the image frame MF001 is set in the image frame MF001, and coordinate axes U and V representing the image coordinate system are drawn.

[0037] In the image frame MF600 after the movement operation, multiple first workpieces WK1a and WK1b are arranged in the first target area TA1, and no first workpieces WK1a and WK1b are arranged in the first supply area SA1. That is, the first workpieces WK1a and WK1b have been moved from the first supply area SA1 to the first target area TA1.

[0038] The object recognition unit 312 refers to the work attribute data WD and recognizes the type and position and orientation of the first workpieces WK1a and WK1b from the image frame MF600 after the movement operation. Based on the recognition results of the positions and orientations of the first workpieces WK1a and WK1b, the object recognition unit 312 displays the image frame MF600 on the display unit 350, in which frame lines surrounding each of the first workpieces WK1a and WK1b are drawn around the first workpieces WK1a and WK1b. An image coordinate system indicating positions within the image frame MF600 is set in the image frame MF600, and coordinate axes U and V representing the image coordinate system are drawn.

[0039] While FIG. 4 only illustrates image frame MF001 taken before the moving operation and image frame MF600 taken after the moving operation, the information processing device 300 is supplied with work images captured by the first camera 210 and the third camera 230 from before the moving operation to after the moving operation. When a change in the position and orientation of the workpiece occurs during the work, the object recognition unit 312 references the work attribute data WD and recognizes the type and position and orientation of the first workpieces WK1a and WK1b from the image frames. For example, the object recognition unit 312 compares at least two consecutive captured image frames, and when a change in the position and orientation of the workpiece occurs, recognizes the type and position and orientation of the first workpieces WK1a and WK1b from the image frames. Because object recognition is performed only when a change in the position and orientation of the workpiece occurs during the work, the processing load on the processor 310 can be reduced.

[0040] FIG. 5 is a diagram showing the recognition results for the first workpiece WK1 by the object recognition unit 312. Each record of this recognition result registers the image frame number, workpiece ID, workpiece type ID, image coordinate points, and the reference coordinate system position and orientation. The recognition results for the first workpiece WK1 are time-series data in which records are arranged in chronological order. In the example of FIG. 5, the recognition results for the two first workpieces WK1a and WK1b are registered for the image frame MF001 before the movement operation. The recognition results for the two first workpieces WK1a and WK1b are also registered for the image frame MF600 after the movement operation. The workpiece ID is an identifier that distinguishes each workpiece. The workpiece type ID is an identifier that indicates the type of workpiece. The image coordinate points are values ​​that express the position of the representative point of each workpiece in image coordinates (U, V). For example, the center of gravity of the workpiece or the upper left point of the frame surrounding the first workpieces WK1a and WK1b shown in FIG. 4 can be used as the representative point of the workpiece. The recognition result does not necessarily have to include image coordinate points.

[0041] The reference coordinate system position and orientation is a value that represents the position and orientation of the workpiece in a reference coordinate system, which is a robot-independent coordinate system that does not depend on the robot 100. Of the reference coordinate system position and orientation, the parameters X, Y, and Z that indicate coordinate positions indicate positions on three axes, respectively. Of the reference coordinate system position and orientation, the parameters θx, θy, and θz that indicate orientation or rotation indicate rotation angles around three axes, respectively. Note that, instead of rotation angles, any expression such as a rotation matrix or quaternion that represents rotation can be used to express the parameters that indicate orientation or rotation.

[0042] 3, the action recognition unit 313 recognizes the action of the worker TP from the work image. Hereinafter, the action of the worker TP will be referred to as the worker action.

[0043] FIG. 6 is a diagram showing an example of image frames capturing worker actions. FIG. 6 shows three image frames MF200, MF300, and MF400, which are part of multiple image frames captured in time series, stacked one on top of the other. Image frame MF200 shows the worker TP grasping the first workpiece WK1a in the first supply area SA1. The action recognition unit 313 sets a bounding box BB that surrounds the arm AM and the first workpiece WK1a in the image frame MF200 and draws a frame line representing the bounding box BB. The action recognition unit 313 similarly sets bounding boxes BB in the other image frames MF300 and MF400 and draws a frame line representing the bounding box BB.

[0044] The bounding box BB can be used for the following purposes, for example: (1) To perform contact detection on the image using the workpiece recognition results and the finger position recognition results. (2) To identify the gripping position on the image using the workpiece recognition results and the finger position recognition results. (3) Draw a bounding box BB on the image to show that the arm AM is correctly recognized.

[0045] The action recognition unit 313 recognizes the action of the worker TP by analyzing a plurality of consecutive image frames as shown in FIG.

[0046] FIG. 7 is a diagram showing the recognition results of the worker's actions by the action recognition unit 313. Each record of this recognition result registers the image frame number, individual ID, action number, action name, BB top left point position, and BB bottom right point position for each worker's action included in the work. BB is an abbreviation for bounding box. The recognition results of the worker's actions are time-series data in which records are arranged in chronological order. The individual ID is an identifier that distinguishes between arms AM. For example, different individual IDs are assigned to the right arm and the left arm.

[0047] The action number is an identifier that distinguishes between worker actions. The action name indicates the type of worker action in that image frame. In the example of Figure 7, a pick action is recognized in image frame MF200. A pick action is an action in which a workpiece is grasped with fingers. A pick action is also called a grasping action by fingers. A place action is recognized in image frame MF300. A place action is an action in which a grasped workpiece is placed in a certain position. The BB upper left point position and BB lower right point position are expressed as positions in the first camera coordinate system Σc1, which is the reference coordinate system.

[0048] Since a typical task involves multiple worker actions, multiple worker actions included in the task related to the work are recognized in step S30 shown in Fig. 3. However, a task can also be composed of one or more worker actions.

[0049] In step S40 of FIG. 3, the joint position recognition unit 314 recognizes the positions of multiple joints included in the upper limbs of the worker TP from the work image.

[0050] 8 is a flowchart showing the detailed procedure of step S40. In step S41, movement recognition unit 313 reads a plurality of consecutive image frames captured by first camera 210 and third camera 230. In step S42, movement recognition unit 313 recognizes the movement of the upper limbs by analyzing the plurality of consecutive image frames.

[0051] In step S43, the action recognition unit 313 determines whether the action of the upper limb is a specific action. A specific action is an action performed by the worker TP using his or her fingers for instruction. This action is also called a finger action. For example, one or more of a grasping action with fingers, a releasing action with fingers, and a pointing action with fingers are specified as the specific action. If, in step S43, the action of the upper limb corresponds to the specific action (step S43; YES), in step S44, the joint position recognition unit 314 recognizes the positions of pre-specified joints from among multiple joints included in the upper limb. For example, the joint position recognition unit 314 recognizes the shoulder joint, elbow joint, wrist joint, and finger joint. Hereinafter, the positions of the joints are referred to as joint positions. The joint position recognition unit 314 recognizes the joint positions from the image frame corresponding to the specific action. The recognition result of the joint positions will be described later. Thereafter, the processing of step S40 ends.

[0052] In step S43, if the finger movement does not correspond to the specific movement (step S43; NO), the processing of step S40 is terminated. In the embodiment, if the worker's movement does not include the specific movement, the processing to recognize the joint position is not performed. In this way, the processing to recognize the joint position is performed only when the worker's movement includes the specific movement, thereby reducing the processing load.

[0053] FIG. 9 is a diagram showing how joint positions are recognized. Here, in the image frame MF200 shown in FIG. 6, multiple reference points JP are set on the upper limbs of the worker TP. The joint position recognition unit 314 specifies the recognized joint positions on the upper limbs as reference points JP. The joint position recognition unit 314 also sets a link JL, which is a straight line connecting two adjacent reference points JP. The shoulder joint, elbow joint, and wrist joint each include two or more joints. In this embodiment, the joint position recognition unit 314 selects one representative joint for the shoulder joint and recognizes the position of the selected joint. The same applies to the elbow joint and the wrist joint. Furthermore, because the fingers are parts that grip a workpiece, the joint position recognition unit 314 selects two or more finger joints and recognizes the positions of the selected two or more joints.

[0054] FIG. 10 is a diagram showing reference points JP included in the range of the fingers of the upper limbs. FIG. 11 is a diagram showing reference points JP included in the range from the shoulder to the wrist of the upper limbs. Reference points JP included in the range of the fingers are not shown in FIG. 11. The following (1) to (8) are set as reference points JP. Some or all of the following reference points JP are recognized by the joint position recognition unit 314. Recognizing all of the reference points JP can improve the accuracy of the recognition results. However, from the perspective of reducing the processing load, it is preferable to exclude some of the reference points JP from the recognition targets. To reduce the resulting decrease in accuracy of the recognition results, it is preferable to recognize at least the tip of the thumb JP10, the tip of the index finger JP20, the wrist joint point JP60, the elbow joint point JP70, and the shoulder joint point JP80. (1) Tip of the thumb JP10 and joint points JP11-JP13 (2) Tip of index finger JP20 and joint points JP21-JP23 (3) Tip of the middle finger JP30 and joint points JP31-JP33 (4) Tip of ring finger JP40 and joint points JP41-JP43 (5) Tip of the little finger JP50 and joint points JP51-JP53 (6) Wrist joint point JP60 (7) Elbow joint point JP70 (8) Shoulder joint point JP80

[0055] FIG. 12 is a diagram showing the joint position recognition results obtained by the joint position recognition unit 314. Each record of the recognition results includes an image frame number, an individual ID, a joint position ID, a part name, an image coordinate point of the joint position, and a reference coordinate system position of the joint position. The joint position recognition results are time-series data in which records are arranged in chronological order. The individual ID is an identifier that distinguishes the arm AM. The joint position ID is an identifier that distinguishes the reference point JP shown in FIGS. 10 and 11. The part name is the name of the part of the upper limb that is the target of recognition by the joint position recognition unit 314. In FIG. 12, for example, the tip of the thumb JP10 is registered for "thumb," and the tip of the index finger JP20 is registered for "index." The other reference points described in FIGS. 10 and 11 are registered in a similar manner. The image coordinate points and reference coordinate system positions of the joint positions indicate individual joint positions. Note that the recognition results do not necessarily include image coordinate points.

[0056] In step S50 of FIG. 3, the work description list creation unit 315 creates a work description list WDL using the recognition result output by the object recognition unit 312 (see FIG. 5), the recognition result output by the action recognition unit 313 (see FIG. 7), and the recognition result output by the joint position recognition unit 314 (see FIG. 12).

[0057] FIG. 13 is a diagram showing the work description list WDL. Each record in the work description list WDL contains the record number, image frame number, action name, workpiece ID, workpiece position and orientation, arm tip position and orientation, and gripping position for each action included in the work. The action name indicates the type of each action. In the example of FIG. 12, five actions are registered in this order for the same first workpiece WK1a: approach action, pick action, depart action, approach action, and place action. Although the approach action and depart action are not included in the worker actions described in FIG. 7, they are necessary actions as action commands of the robot control program RP. Therefore, they are added to the work description list WDL by the work description list creation unit 315 as actions performed before and after the pick action, place action, etc. The approach action is an action that moves the tip of the arm of the robot 100 to a specified position. The depart action is an action that retracts the arm of the robot 100. The depart operation is an operation in which the workpiece is detached from the arm of the robot 100 and the arm is retreated to a specified position. In the depart operation, the arm is moved so that the tip of the arm rises directly upward or diagonally upward from the current position depending on the positions of obstacles around the arm. Alternatively, the arm may be moved so that the tip of the arm moves horizontally.

[0058] The arm tip position and orientation are the position and orientation of the arm tip of the robot 100 during each operation and are calculated from the joint position recognition results shown in FIG. 12. The arm tip position and orientation are determined, for example, as follows: For a pick operation, the task description list creation unit 315 determines the position where the fingertips of the hand contact the object as the gripping position based on the recognition results of the joint positions included in the range of the fingers when the pick operation is recognized, and performs coordinate transformation with the reference coordinate system as the origin. The task description list creation unit 315 calculates the arm tip position and orientation as a value indicating the tip position of the arm of the robot 100 from the gripping position. At this time, it is preferable to calculate the arm tip position and orientation taking into account the orientation of the workpiece. The optimal arm tip position and orientation may differ depending on the end effector used in the actual operation. For example, the arm tip position and orientation during a pick operation and a place operation using a gripper can be calculated as the center of gravity of multiple gripping positions. The arm tip position during an approach operation is set to a position elevated a predetermined distance from the arm tip positions during the previous or next pick operation or place operation. The same applies to the position of the arm tip during the departure operation.

[0059] The grip position indicates the position of the fingers in each action. The grip position is registered only when the fingers grip or release the workpiece. In the example of FIG. 13, the grip position is registered only during the pick action and the place action, and is not registered during the approach action and the depart action. The work description list creation unit 315 calculates the grip position from the recognition results for the fingers among the recognition results shown in FIG. 12. In the example of FIG. 13, the position of the tip JP10 of the thumb and the tip JP20 of the index finger are registered. Other reference points may be registered in the same way, but it is preferable to register at least the positions of the tip JP10 of the thumb and the tip JP20 of the index finger.

[0060] All positions and postures registered in the work description list WDL are expressed in a reference coordinate system, which is a robot-independent coordinate system. Therefore, a robot control program RP suitable for any type of robot can be easily created from this work description list WDL. The work description list WDL is a list in which work is divided into units equivalent to one robot movement, and one movement is expressed as one line of data.

[0061] In step S60 of FIG. 3, the movement space calculation unit 316 calculates a movable space that defines a space in which the robot 100 can move. This process is also called a space calculation process. The movable space is a space that is the target of path search for the robot 100. In the embodiment, path search for the robot 100 is performed only within the movable space. A path for the robot 100 is not searched outside the movable space.

[0062] The reason for calculating the movable space is as follows: The work description list WDL registers the starting point, intermediate points, and end point of the robot's movements extracted from the worker's movements. Since the work description list WDL does not include route information, it is necessary to generate a route for the robot 100 to move via intermediate points. The route of the robot 100 is generated by searching for a route in the space to be searched. In the embodiment, the search range is limited to a preset range in order to reduce the amount of calculation required for searching for a route.

[0063] FIG. 14 is a diagram showing the arm of worker TP, the position of which changes over time. FIG. 14 shows the arm of worker TP as viewed from above. Arm AM11 shows the position and posture of arm AM at time t1. Arm AM12 shows the position and posture of arm AM at time t2. Arm AM13 shows the position and posture of arm AM at time t3. Time t1 is the earliest time, and time t3 is the latest time. Time t2 is a time between time t1 and time t3. For example, time t1 is the time when the approach action starts, and time t2 is the time when the pick action starts.

[0064] Fig. 15 is a diagram for explaining calculation of the movable space. Fig. 15 also shows arms at time t1, time t2, and time t3. In Fig. 15, to facilitate understanding of the technology, arm AM11 at time t1, arm AM12 at time t2, and arm AM13 at time t3 are shown without overlapping.

[0065] The movement space calculation unit 316 divides the upper limb into the upper arm, the forearm, and the fingers, and defines the region containing each part as a cylindrical region indicated by a dashed line. Because FIG. 15 shows the worker TP photographed from above, the parts indicated by the dashed lines are rectangular, but each region is cylindrical in three-dimensional space. The upper arm is the part from the shoulder joint to the elbow joint. The forearm is the part from the elbow joint to the wrist joint. The fingers are the part from the wrist joint to the fingertips.

[0066] The movement space calculation unit 316 sets a cylindrical space having a central axis that is a line connecting two predetermined joints and a plane that intersects with this line at the positions of the two joints as the bottom. Specifically, the movement space calculation unit 316 sets the following space in which the upper arm is located: A line connecting joint point JP80 indicating the shoulder joint and joint point JP70 indicating the elbow joint is set as the central axis of the cylinder, and a plane that intersects with the central axis at the positions of joint point JP80 and joint point JP70 is set as the bottom. The movement space calculation unit 316 sets this cylindrical space as the space in which the upper arm is located. The space in which the upper arm is located is called a cylinder C1.

[0067] Furthermore, the movement space calculation unit 316 sets the following space in which the forearm is located. A straight line connecting a joint point JP70 indicating the elbow joint and a joint point JP60 indicating the wrist joint is set as the central axis of a cylinder, and a plane intersecting the central axis at the positions of the joint point JP70 and the joint point JP60 is set as the bottom. The wrist joint is a joint selected as a representative of the wrist joints. The movement space calculation unit 316 sets this cylindrical space as the space in which the forearm is located. The space in which the forearm is located is called a cylinder C2.

[0068] Furthermore, the movement space calculation unit 316 sets the following space in which the finger parts are located: A straight line connecting a joint point JP60 indicating the wrist joint and a tip JP20 indicating the tip of the index finger is set as the central axis of a cylinder, and a plane intersecting the central axis at each of the joint point JP60 and the tip JP20 is set as the bottom. The movement space calculation unit 316 sets this cylindrical space as the space in which the finger parts are located. The space in which the finger parts are located is called a cylinder C3.

[0069] The radii of the bases of the cylinders C1, C2, and C3 are set to predetermined radii. For example, they may be determined based on the shape and size of the arms of the robot 100. The reference coordinate system positions of the joints output by the joint position recognition unit 314 are used as the positions of the respective joints. The cylinders may interfere with each other, or may be set to have sizes that do not interfere with each other.

[0070] FIG. 16 is a diagram for explaining calculation of the movable space. FIG. 16 illustrates cylinders C1 and C2 at times t1, t2, and t3, respectively. The arms of the worker TP are not shown. The movement space calculation unit 316 calculates the movement range of the joint points JP70 and JP80 based on the time-series changes in the positions of the joint points JP70 and JP80. This process is also referred to as range identification processing. The movement space calculation unit 316 calculates the space R1 through which the cylinder C1 moves based on the cylinder C1 and the movement ranges of the joint points JP70 and JP80. In the illustrated example, the space R1 is set as a rectangular parallelepiped space. The movement space calculation unit 316 calculates the shape and size of the space R1 and the coordinates of a reference position for identifying the position of the space R1. For example, if the space R1 is a rectangular parallelepiped space, one corner of the rectangular parallelepiped is selected as the reference position. Furthermore, the movement space calculation unit 316 calculates the length, width, and height of the space R1. The space R1 represents the space through which the upper arm has moved between time t1 and time t3.

[0071] The movement space calculation unit 316 calculates the movement range of the joint points JP60 and JP70 based on the time-series changes in the positions of the joint points JP60 and JP70. Based on the cylinder C2 and the movement ranges of the joint points JP60 and JP70, it calculates the space R2 through which the cylinder C2 moves. The space R2 represents the space through which the forearm moves between time t1 and time t3. Furthermore, the movement space calculation unit 316 calculates the movement ranges through which the tip end JP20 and the joint point JP60 move based on the time-series changes in the positions of the tip end JP20 and the joint point JP60. Based on the cylinder C3 and the movement ranges of the tip end JP20 and the joint point JP60, the movement space calculation unit 316 calculates the space R3 through which the cylinder C3 moves. The space R3 represents the space through which the fingers move between time t1 and time t3. The spaces R2 and R3 are set as rectangular parallelepiped spaces, similar to the space R1. The movement space calculation unit 316 determines the space obtained by adding together the spaces R1, R2, and R3 as the movable space. The movable space is a section that is the target of path search for the robot 100.

[0072] In this way, the movement space calculation unit 316 sets the space occupied by each part constituting the upper limbs of the worker TP, and calculates the movable space based on information representing the set space and the path along which the joints have moved. The space occupied by each part constituting the upper limbs of the worker TP is, for example, the space occupied by the part constituting the body of the worker TP between two adjacent joints of the upper limb. The movement range is found from the actual movement of the arms and fingers of the worker TP, and the movable space is calculated from the movement range, so that the movable space can be set appropriately.

[0073] In step S70 of FIG. 3, the constraint generating unit 317 generates constraints including the determined movable space.

[0074] FIG. 17 is a diagram showing the constraint list CL. The constraint list CL describes constraints that restrict the robot 100 from performing a specific task within a movable space. Each record in the constraint list CL contains a record number, a space R1, a space R2, a space R3, a joint angle J1, a joint angle J2, and a joint angle J3. The record numbers correspond to the record numbers in the work description list WDL (see FIG. 13). For example, the constraint with record number 001 is the constraint for the record with record number 001 in the work description list WDL. The work description list WDL and the constraint list CL are collectively referred to as a constrained action list.

[0075] Spaces R1, R2, and R3 are information specifying the positions and ranges of the spaces R1, R2, and R3 that constitute the aforementioned movable space. Joint angle J1 is information specifying the rotatable angle of a predetermined joint of the robot 100. If joint angle J1 is set to 30-60 degrees, this indicates that the predetermined joint of the robot 100 can rotate within a range of 30 to 60 degrees. Generally, the degrees of freedom of the rotational angle of a robot's joint are greater than the rotational angle of a human joint. However, in this embodiment, the rotatable angle is specified in this manner to allow the robot 100 to mimic the movements of a human. For example, the joint of the robot 100 whose rotation angle is specified by joint angle J1 may be a joint corresponding to a human's shoulder joint. In this case, the angle specified by joint angle J1 may be set to match the range of motion of a human's shoulder joint.

[0076] The joint angle J2 is information that defines the rotatable angle of another joint of the robot 100. For example, the joint of the robot 100 whose rotation angle is defined by the joint angle J2 may be a joint that corresponds to a human elbow joint. In this case, the angle specified by the joint angle J2 may be set in accordance with the range of motion of a human elbow joint.

[0077] The joint angle J3 is information that defines the rotatable angle of another joint of the robot 100. For example, the joint of the robot 100 whose rotation angle is defined by the joint angle J3 may be a joint that corresponds to a human wrist joint. In this case, the angle specified by the joint angle J3 may be set in accordance with the range of motion of a human elbow joint.

[0078] In step S80 of Figure 3, the second workpiece WK2 present in the second working area WA2 is recognized. First, under the control of the processor 310, the second camera 220 captures an image of the second working area WA2 for the robot. The object recognition unit 312 recognizes the second workpiece WK2 present in the second working area WA2 from the image captured by the second camera 220. At this time, the second workpiece WK2 is placed in the first supply area SA1 and is in the position it was in before the movement operation.

[0079] In step S90, the robot control program RP is created. First, the control program creation unit 318 receives input of the robot type. The robot type indicates the type of robot to be controlled by the robot control program RP. For example, the worker TP inputs the robot type using an input device provided in the information processing device 300.

[0080] The control program creation unit 318 first searches for and generates a path for the robot 100 using an existing path search algorithm based on the position of the second workpiece WK2 recognized in step S80, the work description list WDL created in step S50, and the constraint list CL created in step S70. This process is also called a path generation process. The control program creation unit 318 is also called a path generation unit.

[0081] The position of the second workpiece WK2 recognized in step S80 is used as the position of the workpiece before the work. Furthermore, the position of the workpiece after the work, which is registered in the work description list WDL, is used. As described above, in the work description list WDL, the first camera coordinate system Σc1 is used as the reference coordinate system. The position and orientation expressed in the first camera coordinate system Σc1 is transformed into the robot coordinate system Σr by coordinate transformation. A known method is used as a method for performing coordinate transformation between the first camera coordinate system Σc1, which is the reference coordinate system, and the robot coordinate system Σr.

[0082] The control program creation unit 318 searches for a path so that the position and posture of the arm tip of the robot 100 transitions to the arm tip position and posture specified in each line of the work description list WDL. The path is searched for within the range of the movable space specified in the record of the constraint list CL corresponding to the record of the work description list WDL. In addition, a path is searched for so that the three joints of the robot 100, which are subject to constraints specified by the joint angles J1, J2, and J3, rotate within the range of the joint angles J1, J2, and J3, respectively.

[0083] The control program creation unit 318 stores path data representing the path of the robot 100 generated by the search in the memory 320. The control program creation unit 318 may output the data representing the path of the robot 100 generated by the search to a robot controller that controls the robot 100.

[0084] FIG. 18 is a diagram illustrating path search for each joint of the robot 100. The robot 100 includes joints J10, J20, and J30. The control program creation unit 318 searches for a path for each of the joints J10, J20, and J30. As shown in the figure, the path of the joint J30 can take various paths. The same applies to the joints J10 and J20, which are not shown in FIG. 18. For example, if the robot 100 has three joints, the control program creation unit 318 must search for a path in a three-dimensional space. Also, if the robot 100 has six joints, the control program creation unit 318 must search for a path in a six-dimensional space. For example, as shown in the figure, the path of the joint J30 can take various paths within the range of space R3.

[0085] In this embodiment, the control program creation unit 318 searches for a path for the robot 100 to move within the movable space, which is the combined space of the spaces R1, R2, and R3. Therefore, the space to be searched is limited compared to when the movable space is not set. Since the range of the search is limited to the movable space based on the path traveled by the worker's arm and finger joints, the amount of calculation required to generate the path can be reduced.

[0086] Furthermore, the control program creation unit 318 uses the work description list WDL and the constraint list CL to create a robot control program RP for operating the robot 100 along the generated path. The control program creation unit 318 stores the generated robot control program RP in memory 320. The control program creation unit 318 may output the robot control program RP to a robot controller that controls the robot 100.

[0087] Since the work description list WDL, which describes specific tasks in a robot-independent coordinate system, is used, a control program suitable for the type of robot can be easily created. Furthermore, when a control program is generated from the work description list WDL and the constraint list CL, i.e., an action list with constraints, the amount of calculation required to generate the control program can be reduced compared to when a control program is generated from an action list without constraints.

[0088] FIG. 19 is a diagram showing another example of a set movable space. FIG. 19 shows obstacles O1, O2, and O3 arranged around the first workpiece WK1a. Since it is naturally expected that the worker TP will work while avoiding the obstacles, in the method according to the embodiment, it is not necessary to identify the space occupied by the obstacles when calculating the movable space. Conventionally, it was necessary to consider the position, size, etc. of the obstacles when calculating a path, but in the method according to the embodiment, it is not necessary to consider the position, size, etc. of the obstacles.

[0089] The movable space is set based on the range of movement of the worker's joints, including at least one of the worker's shoulder joint, elbow joint, wrist joint, and finger joint. The movable space indicates the path that the worker's arm and fingers can move, so the robot 100 can be moved along a path that follows the path of movement of the worker's arm and fingers. Therefore, the movement of the robot 100 can be made to imitate the natural movement of a person.

[0090] B1. Other Embodiments The robot system 1000 of the embodiment includes the first camera 210 and the third camera 230 for capturing images of the worker TP and the workpiece. However, the robot system 1000 does not necessarily need to include the third camera 230.

[0091] Furthermore, the image used to recognize the worker's motion in step S30 of the control program creation process shown in Fig. 3 and the image used to recognize the hand and finger motion in step S40 may be images taken by different cameras. If the hand and finger motion is photographed using a camera different from the camera that photographs the worker's motion, the hand and finger motion can be recognized more accurately.

[0092] Furthermore, the image used to recognize the workpiece in step S20 and the image used to recognize the worker's movements in step S30 may be images taken by different cameras. If the workpiece is photographed using a camera different from the camera that photographs the worker's movements, the workpiece can be recognized more accurately.

[0093] In the embodiment, an example has been described in which the joint position recognition unit 314 recognizes the positions of the shoulder joint, elbow joint, wrist joint, and finger joint. However, the joint position recognition unit 314 may not recognize some of these joints. For example, the joint position recognition unit 314 may not recognize the wrist joint, which is a representative wrist joint.

[0094] In the embodiment, the robot 100 has one arm, but the robot 100 may be a dual-arm robot. In this case, the worker TP demonstrates the work using both arms. In this case, the paths of each of the robot's two arms are determined based on a captured image of the worker TP demonstrating the work. In this case, the movements of the robot 100 can also be made to imitate the natural movements of a person.

[0095] Although the above embodiment describes an example of a pick-and-place operation, the present disclosure can also be applied to other operations, such as a coating operation that includes a pointing motion, a screwdriver operation, a nailing operation with a hammer, a workpiece insertion operation, a fitting operation, an assembly operation, and the like.

[0096] Furthermore, the means for realizing the functions of the information processing device 300 is not limited to software, and some or all of the functions may be realized by dedicated hardware. For example, the dedicated hardware may be a circuit such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0097] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

[0098] C. Other forms: (1) According to one aspect of the present disclosure, there is provided a computer program that causes a computer to execute a process for generating a path for a robot. The computer program causes the computer to execute a range determination process for determining a range of movement of the joints of a worker while the worker is performing a specific task of manipulating a workpiece using the worker's arms and fingers, based on a plurality of images of the worker's arms and fingers captured continuously by an imaging device, a space calculation process for calculating a movable space in which the robot can move based on the range of movement, and a path generation process for searching for a path for the robot to move within the movable space and generating the path so that the robot can perform the specific task. According to the above embodiment, when searching for a path for the robot to move, the range of the search is limited to the movable space based on the path traveled by the joints of the worker's arms and fingers, thereby reducing the amount of calculation required to generate the path.

[0099] (2) The above computer program may execute the following processes: a process for recognizing a worker's movement from the image; a process for recognizing the positions of the worker's fingers from the image if the worker's movement includes finger movement; a process for recognizing the position of the workpiece from the image; a process for generating an action list that describes the specific work in a robot-independent coordinate system that is independent of the type of robot using the recognized worker movement, the positions of the fingers, and the position of the workpiece; and a process for generating a constrained action list by adding constraints to the action list that restrict the specific work from being performed within the movable space. According to the above-described embodiment, a control program suitable for the type of robot can be easily created from an action list that describes a specific task in a robot-independent coordinate system. Furthermore, when a control program is generated from an action list with constraints, the amount of calculation can be reduced compared to when a control program is generated from an action list without constraints.

[0100] (3) In the computer program, the range of movement may be determined using a result of recognizing the positions of the joints of the worker. According to the above embodiment, the movement range is determined from the actual movement of the worker's arm and fingers, and the movable space is calculated from the movement range. The movable space indicates the path that the worker's arm and fingers could have moved, so the robot can move along a path that follows the path of the worker's arm and fingers. This makes it possible to make the robot's movements imitate the natural movements of a human.

[0101] (4) In the above computer program, the joints may include at least one of a shoulder joint, an elbow joint, a wrist joint, and a finger joint. According to the above embodiment, the robot moves within a movable space based on the range of movement of the joints, including at least one of the worker's shoulder joint, elbow joint, wrist joint, and finger joint, so the robot's movements can be made to imitate the natural movements of a person.

[0102] (5) According to another aspect of the present disclosure, there is provided a method for generating a path for a robot by a computer, the method including causing a computer to execute the following steps: determining a range of movement of the joints of a worker while the worker is performing a specific task of manipulating a workpiece using the worker's arms and fingers from a plurality of images continuously captured by an imaging device of the worker's arms and fingers; calculating a movable space in which the robot can move based on the range of movement; and searching for a path for the robot to move within the movable space and generating the path so as to have the robot perform the specific task. According to the above embodiment, when searching for a path for the robot to move, the range of the search is limited to the movable space based on the path traveled by the joints of the worker's arms and fingers, thereby reducing the amount of calculation required to generate the path.

[0103] (6) According to yet another aspect of the present disclosure, there is provided a generation device for generating a path for a robot, the generation device including: a receiving unit that receives a plurality of images of a worker's arm and fingers captured by an imaging device while the worker is performing a specific task of manipulating a workpiece using the arm and fingers; a range identifying unit that determines, from the plurality of images, a range of movement of the worker's joints during the specific task; a space calculating unit that calculates a movable space within which the robot can move based on the range of movement; and a path generating unit that searches for a path for the robot to move within the movable space and generates the path so that the robot can perform the specific task. According to the above embodiment, when searching for a path for the robot to move, the range of the search is limited to the movable space based on the path traveled by the joints of the worker's arms and fingers, thereby reducing the amount of calculation required to generate the path. [Explanation of symbols]

[0104] 100...robot, 210...first camera, 220...second camera, 230...third camera, 300...information processing device, 310...processor, 311...image receiving unit, 312...object recognition unit, 313...motion recognition unit, 314...joint position recognition unit, 315...task description list creation unit, 316...movement space calculation unit, 317...constraint condition generation unit, 318...control program creation, 320...memory, 330...interface circuit, 340...input device, 350...display unit, 1000...robot system, AM, AM11, AM12...arm, BB...bounding box, C1, C2, C3...cylinder, CL...constraint condition list, J1, J2, J3...joint angle, J10, J20, J30...joint, JL...link, JP...reference Point, MF001...image frame, MF200...image frame, MF300...image frame, MF400...image frame, MF600...image frame, R1, R2, R3...space, RD...robot characteristic data, RP...robot control program, SA1...first supply area, SA2...second supply area, TA1...first target area, TA2...second target area, TB1, TB2...work table, TP...worker, WA1...first working area, WA2...second working area, WD...work attribute data, WDL...work description list, WK1, WK1a, WK1b...first workpiece, WK2...second workpiece, Σc1...first camera coordinate system, Σc2...second camera coordinate system, Σc3...third camera coordinate system, Σr...robot coordinate system, t1, t2, t3...time

Claims

1. A computer program that causes a computer to execute a process for generating a path for a robot, a range identification process for determining a range of movement of the joints of a worker while the worker is performing a specific task of manipulating a workpiece using the worker's arms and fingers, from a plurality of images of the worker's arms and fingers continuously captured by an imaging device; and a space calculation process for calculating a movable space in which the robot can move based on the movement range; a path generation process for searching for a path for the robot to move within the movable space and generating the path in order to have the robot perform the specific task; Execute Computer program.

2. 2. The computer program of claim 1, A process of recognizing a worker's action from the image; a process of recognizing positions of the worker's fingers from the image when the worker's motion includes a finger motion; A process of recognizing the position of the workpiece from the image; generating an action list describing the specific task in a robot-independent coordinate system that is independent of the type of robot, using the recognized worker motion, the positions of the fingers, and the position of the workpiece; a process of generating a constrained action list by adding a constraint condition that restricts the specific work from being performed within the movable space to the action list; Execute Computer program.

3. 3. A computer program according to claim 1 or 2, The movement range is determined using a result of recognizing the positions of the joints of the worker. Computer program.

4. 4. A computer program according to any one of claims 1 to 3, The joints include at least one of a shoulder joint, an elbow joint, a wrist joint, and a finger joint. Computer program.

5. A method for generating a path for a robot by a computer, comprising: On the computer, a step of determining, from a plurality of images continuously taken by an imaging device of the arm and the finger of a worker while the worker is performing a specific task of manipulating a workpiece using the arm and the finger, a movement range of the joints of the worker during the period when the specific task is being performed; calculating a movable space in which the robot can move based on the movement range; searching for a path for the robot to move within the movable space and generating the path in order to have the robot perform the specific task; Execute Generation method.

6. A generation device that generates a path for a robot, a receiving unit that receives a plurality of images of the worker's arms and fingers successively captured by an imaging device while the worker is performing a specific task of manipulating a workpiece using the arms and fingers; a range specifying unit that determines, from the plurality of images, a range of movement of the joints of the worker during the period in which the specific task was being performed; a space calculation unit that calculates a movable space in which the robot can move based on the movement range; a path generation unit that searches for a path for the robot to move within the movable space and generates the path in order to have the robot perform the specific task; A generating device comprising:

Images