Imaging method, appearance inspection method, control device, and imaging system

Abstract

To efficiently image a desired part of a workpiece, in a state in which a robot grips the workpiece.SOLUTION: An imaging method for imaging a workpiece includes: a step (b) of allowing a robot to adjust at least one of the position and the attitude of the gripped workpiece, so that the workpiece takes any one imaging attitude; a step (c) of allowing a camera to image a target part of the workpiece taking the any one imaging attitude; a step (d) of executing the step (b) and the step (c) in a new imaging attitude when the robot can image a non-imaged target part without re-gripping the workpiece; and a step (e) of controlling the robot so as to re-grip the workpiece in a new gripping part and gripping attitude, on the basis of imaging attitude information and the imaging attitude in which the step (c) has been executed when the workpiece cannot take other imaging attitude such that the robot can image the non-imaged target part while gripping the workpiece, and executing the step (b) and the step (c) in the new imaging attitude.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a photographing method, an appearance inspection method, a control device, and a photographing system.

Background Art

[0002] In the technology described in Patent Document 1, while the robot holds the workpiece, the workpiece is photographed to acquire video information used for inspection.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology described in Patent Document 1, since the portion where the robot holds the workpiece is hidden by the robot's gripping part, it is impossible to photograph that portion. For this reason, when photographing the workpiece while the robot holds the workpiece, a technology that can efficiently photograph a desired range has been desired.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] According to the first aspect of the present disclosure, a photographing method for photographing a workpiece is provided. This photographing method is a photographing method for photographing a workpiece, comprising: (a) causing a robot to grip the workpiece; (b) adjusting at least one of the position and orientation of the workpiece with respect to a camera by the robot so that the orientation of the gripped workpiece becomes any one of a plurality of predetermined photographing orientations capable of photographing a plurality of target parts to be photographed among the workpiece; (c) causing the camera to photograph the target parts of the workpiece gripped by the robot and set to any one of the photographing orientations; (d) when the workpiece can assume another photographing orientation that is a photographing orientation capable of photographing the unphotographed target parts without the robot regripping the workpiece and is different from the photographing orientation in which step (c) has been performed until then, performing steps (b) and (c) with the other photographing orientation as a new photographing orientation; and (e) when the workpiece cannot assume the other photographing orientation capable of photographing the unphotographed target parts even if the robot changes the position and orientation of the workpiece while still gripping the workpiece, controlling the robot to regrip the workpiece with a new gripping part and a gripping orientation specified based on the photographing orientation information associating the gripping part of the robot gripping the workpiece and the gripping orientation of the robot with respect to the workpiece with the photographing orientation capable of photographing the target parts, and the photographing orientation in which step (c) has been performed until then, and performing steps (b) and (c) with the other photographing orientation as the new photographing orientation.

[0007] According to a second aspect of the present disclosure, there is provided a control device that controls the operation of a robot. The control device is configured such that, after the camera has captured one or more of the target parts of the workpiece that can be captured in a plurality of predetermined imaging postures, with the robot holding the workpiece in a state where at least one of the position and posture of the workpiece with respect to the camera has been adjusted so that the posture of the workpiece becomes any one of the plurality of imaging postures for imaging the plurality of target parts to be imaged in the workpiece: (i) if the workpiece can assume another imaging posture that is different from the imaging postures that have been imaged so far and that enables the robot to image the un-imaged target parts without re-grasping the workpiece, the step of controlling the robot to change the position and posture of the workpiece with respect to the camera by setting the other imaging posture as a new imaging posture; and (ii) if the workpiece cannot assume the other imaging posture that enables the un-imaged target parts to be imaged even when the robot changes the position and posture of the workpiece while holding the workpiece, the step of controlling the robot to re-grasp the workpiece with a new gripping part and gripping posture that are specified based on the imaging posture information associating the gripping part and the gripping posture of the robot with respect to the workpiece and the imaging posture enabling the target parts to be imaged, and the imaging postures that have been imaged so far. At least one of these steps is executed.

[0008] According to a third aspect of the present disclosure, a photographing system having a camera and a control device is provided. In this photographing system, the control device is a robot control unit that controls the operation of a robot, and causes the robot to adjust at least one of the position and orientation of the workpiece so as to be in any one of a plurality of predetermined photographing postures while holding the workpiece. A robot control unit, and a camera control unit that controls the photographing operation of the camera so as to photograph the workpiece in the any one of the photographing postures while being held by the robot. When the robot can photograph an unphotographed target part without re-grasping the workpiece, the robot control unit and the camera control unit repeat the respective controls for changing the position and orientation of the workpiece by the robot and photographing the workpiece by the camera. Even if the robot changes the position and orientation of the workpiece while holding the workpiece, if the unphotographed target part cannot be photographed, the robot control unit determines the gripping part of the robot that holds the workpiece and the gripping posture of the robot with respect to the workpiece. The robot is controlled to re-grasp the workpiece with a new gripping part and a gripping posture specified based on the photographing posture information associated with the photographing posture capable of photographing the target part and the photographing posture in which photographing has been performed so far. The camera control unit controls the camera so as to photograph the target part of the workpiece.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0010] A. Embodiment: FIG. 1 is an explanatory diagram showing a schematic configuration of a robot system 1 according to the present embodiment. The robot system 1 photographs a workpiece held by the robot. The robot system 1 includes a robot 100 which is a six-axis robot, a robot controller 200, a first camera 410, a second camera 420, a depth sensor 430, a workbench 500, and an information processing device 600. The first camera 410 is also referred to as the "camera". The information processing device 600 is also referred to as the "control device".

[0011] In FIG. 1, a world coordinate system WC, a robot coordinate system RC, a first camera coordinate system CC1, a second camera coordinate system CC2, and a sensor coordinate system SC are set. The X-axis and Y-axis of the world coordinate system WC are parallel to the horizontal direction. The Z-axis of the world coordinate system is parallel to the vertical direction. The robot coordinate system RC is an orthogonal coordinate system with a predetermined position of the robot 100 as the coordinate origin. The first camera coordinate system CC1 is an orthogonal coordinate system with a predetermined position of the first camera 410 as the origin. The second camera coordinate system CC2 is an orthogonal coordinate system with a predetermined position of the second camera 420 as the origin. The sensor coordinate system SC is an orthogonal coordinate system with a predetermined position of the depth sensor 430 as the origin. The relative relationship between the second camera coordinate system CC2 and the sensor coordinate system SC is known, and the depth within the field of view of the depth sensor 430 is associated with the pixel position in the image captured by the second camera 420. Also, the first camera 410 and the second camera 420 are calibrated in advance, and the positions and postures in each of the world coordinate system WC, the robot coordinate system RC, the first camera coordinate system CC1, the second camera coordinate system CC2, and the sensor coordinate system SC can be mutually converted by coordinate transformation.

[0012] Robot 100 includes a base 110, a robot arm 120, a force sensor 130, and an end effector 150. The base 110 supports the robot arm 120. The base 110 is fixed to the workbench 500.

[0013] The robot arm 120 is connected by six joints J1 to J6. Each joint is provided with a servo motor, a speed reducer, and an angle sensor (not shown), respectively. The servo motor is supplied with current from the robot controller 200 and generates a rotational output for driving each joint. The speed reducer reduces the rotational input given from the corresponding servo motor. The angle sensor detects the rotational angle (the position of the axis) of the output shaft of the corresponding servo motor as the rotational angle of that joint. The angle sensor is, for example, an encoder, a potentiometer, or a resolver. The detected rotational angle is output to the robot controller 200. The force sensor 130 is attached to the tip of the robot arm 120. The force sensor 130 detects the force applied to the end effector 150. The detection value of the force sensor 130 is output to the robot controller 200.

[0014] The end effector 150 is attached to the tip of the robot arm 120 via the force sensor 130. In the present embodiment, the end effector 150 is an electric gripper of a two-finger type for gripping the workpiece WK.

[0015] The robot controller 200 is disposed inside the base 110. The robot controller 200 controls the position of the control point of the robot arm 120 by driving the joints J1 to J6 of the robot arm 120. The control point of the robot arm 120 is installed, for example, at a predetermined position at the tip of the robot arm 120. The control point may be referred to as TCP (Tool Point Center). The position of the control point in the robot coordinate system RC can be represented by the position in the X-axis direction, the position in the Y-axis direction, and the position in the Z-axis direction. Further, the posture of the control point in the robot coordinate system RC can be represented by the angular position of rotation about the X-axis, the angular position of rotation about the Y-axis, and the angular position of rotation about the Z-axis.

[0016] On the workbench 500, a first tray 510 and a second tray 520 are installed. A plurality of workpieces WK are accommodated in the first tray 510. In the present embodiment, the plurality of workpieces WK are randomly stacked in the first tray 510. The second tray 520 is used as a space for temporarily placing the workpiece WK grasped by the robot 100.

[0017] The first camera 410 is fixed to the workbench 500. As the first camera 410, for example, a monocular camera can be used. The first camera 410 photographs the workpiece WK held at a position where it is grasped by the robot 100 and enters the field of view of the first camera 410 from below according to the control of the information processing apparatus 600. The image acquired by the first camera 410 is output to the information processing apparatus 600 by wireless communication.

[0018] The second camera 420 is installed above the first tray 510. As the second camera 420, for example, a monocular RGB camera can be used. The second camera 420 photographs a subject from above the first tray 510 according to the control of the information processing device 600. Here, the work WK in the first tray 510 is the subject. The image acquired by the second camera 420 is output to the information processing device 600 by wireless communication. Also, as the second camera 420, a stereo RGB camera may be used. In this case, the depth can be estimated using the photographed image.

[0019] The depth sensor 430 is installed above the first tray 510. The depth sensor 430 is, for example, a phase shift type sensor, and includes a projector that projects a pattern onto a subject and a camera that photographs the subject in a state where the pattern is projected. Here, the work WK in the first tray is the subject. The depth sensor 430 acquires a projection image according to the control of the information processing device 600. The projection image is an image obtained by photographing the subject in a state where the pattern is projected. The projection image acquired by the depth sensor 430 is output to the information processing device 600 by wireless communication.

[0020] FIG. 2 is a block diagram showing the functions of the information processing device 600. The information processing device 600 includes a memory 610, an input interface circuit 620, an input device 630, a display device 640, and a CPU (Central Processing Unit) 650 as a processor. The memory 610 stores various programs and various data. The input device 630 and the display device 640 are connected to the input interface circuit 620. Also, the robot controller 200, the first camera 410, the second camera 420, and the depth sensor 430 are connected to the input interface circuit 620. The CPU 650 realizes various functions of the information processing device 600 by executing the programs stored in the memory 610.

[0021] The CPU 650 functions as an object recognition unit 651, a robot control unit 652, a camera control unit 653, and an inspection unit 654 by executing a program stored in the memory 610.

[0022] The object recognition unit 651 obtains the depth within the field of view of the depth sensor 430 using the projected image supplied from the depth sensor 430 and the pattern projected onto the subject. The object recognition unit 651 executes recognition processing of the position and orientation of the work WK using the image acquired by the second camera 420 and the depth obtained from the projected image acquired by the depth sensor 430.

[0023] The robot control unit 652 controls the operation of the robot. In the present embodiment, the robot control unit 652 adjusts at least one of the position and orientation of the work WK to the robot 100 so that the work WK is in any one of a plurality of predetermined shooting postures while being gripped.

[0024] The camera control unit 653 controls the shooting operations of the second camera 420 and the depth sensor 430 so as to shoot a plurality of works WK accommodated in the first tray 510. Further, the camera control unit 653 controls the shooting operation of the first camera 410 so as to shoot the work WK in the state of being gripped by the robot 100.

[0025] The inspection unit 654 executes an appearance inspection using the image of the work WK captured by the first camera 410. The appearance inspection is an inspection for checking appearance defects such as foreign matter attached to the surface of the work WK and deformation of the work WK.

[0026] FIG. 3 is a flowchart showing a series of processes executed by the information processing apparatus 600 for the appearance inspection of the work WK.

[0027] In step S10, the process of recognizing the work is executed.

[0028] Specifically, first, the information processing apparatus 600 instructs the second camera 420 to take a picture. In response, the second camera 420 takes pictures of the plurality of workpieces WK in the first tray 510 and outputs the acquired images to the information processing apparatus 600. Also, the information processing apparatus 600 instructs the depth sensor 430 to take a picture. In response, the depth sensor 430 takes pictures of the plurality of workpieces WK in the first tray 510 and outputs the acquired projection images to the information processing apparatus 600.

[0029] The information processing apparatus 600 executes a process of recognizing one or more workpieces WK in the first tray 510 using the image acquired by the second camera 420 and the projection image acquired by the depth sensor 430. Further, the information processing apparatus 600 specifies candidates for workpieces WK that can be grasped by the robot 100 using the result of the recognition process. The plurality of workpieces WK are randomly stacked in the first tray and are not arranged and stored in the first tray. For this reason, for example, among the plurality of randomly stacked workpieces WK, a workpiece WK that is not underlaid by all or part of other workpieces WK is specified as a candidate for a workpiece WK that can be grasped. Further, for each candidate for a workpiece WK that can be grasped, the coordinates and posture of the workpiece WK are calculated. The process of step S10 is executed by the CPU 650 as the object recognition unit 651.

[0030] In step S20, a process of grasping and moving the workpiece WK is executed.

[0031] First, the information processing apparatus 600 specifies the object to be grasped from the candidates for workpieces WK that can be grasped. For example, among the candidates for workpieces WK that can be grasped, the workpiece WK located at the highest position in the first tray 510 is specified as the object to be grasped.

[0032] Subsequently, the information processing apparatus 600 calculates the part (grasping part) where the robot 100 grasps the workpiece WK that is the object of grasping and the posture of the robot 100 with respect to the workpiece WK that is the object of grasping (grasping posture). The grasping posture indicates the advancing direction of the grasping claws of the gripper when the gripper as the end effector 150 of the robot 100 grasps the workpiece WK.

[0033] FIG. 4 is an explanatory diagram of the grasping posture. As shown in FIG. 4(a), up / down, front / back, and left / right are set. In FIGS. 4(b) to 4(f), the illustration of up / down, front / back, and left / right is omitted, but it is the same. In the present embodiment, as shown in FIG. 4(a), the grasping posture when the advancing direction of the grasping claws CL of the gripper is from the upper left to the lower right of the workpiece WK is referred to as "upper left". As shown in FIG. 4(b), the grasping posture when the advancing direction of the grasping claws CL of the gripper is from the top to the bottom of the workpiece WK is referred to as "upper vertical". As shown in FIG. 4(c), the grasping posture when the advancing direction of the grasping claws CL of the gripper is from the left to the right of the workpiece WK is referred to as "left horizontal". As shown in FIG. 4(d), the grasping posture when the advancing direction of the grasping claws CL of the gripper is from the upper right to the lower left of the workpiece WK is referred to as "upper right". As shown in FIG. 4(e), the grasping posture when the advancing direction of the grasping claws CL of the gripper is from the right to the left of the workpiece WK is referred to as "right horizontal". As shown in FIG. 4(f), the grasping posture when the advancing direction of the grasping claws CL of the gripper is from the bottom to the top of the workpiece WK is referred to as "lower vertical". Note that FIG. 4 shows only a part of the plurality of grasping postures in which the robot 100 can grasp the workpiece WK. Also, in FIG. 4, an example where the grasping part is a side surface is shown, but the part where the robot 100 grasps the workpiece WK that is the object of grasping is not limited to the side surface and includes other parts. In the example shown in FIG. 4, a convex claw part is provided on the upper surface of the workpiece WK. For example, the grasping part may be a claw part.

[0034] Also, when a plurality of candidates are calculated for the gripping posture of the workpiece WK to be gripped, for example, the one with the smallest value of the code for identifying the gripping posture is selected as the gripping posture. As shown in FIG. 4, when candidates (a) to (e) are calculated, (a) is selected.

[0035] The information processing device 600 outputs an instruction to grip the workpiece WK to the robot controller 200, together with information on the coordinates indicating the position of the workpiece WK to be gripped in the world coordinate system WC and information indicating the gripping part and the gripping posture. In response to this, the robot controller 200 controls the angles of the respective joints of the robot 100 to cause the robot 100 to grip the workpiece WK at the specified coordinates in the specified gripping part and gripping posture. When the robot 100 grips the workpiece WK, the robot controller 200 notifies the information processing device 600 to that effect.

[0036] The information processing device 600 outputs an instruction to move the workpiece WK to the robot controller 200, together with information on the coordinates indicating the destination of the workpiece WK. As the destination of the workpiece WK, a predetermined position included in the imaging range of the first camera 410 is specified. In response to this, the robot controller 200 controls the angles of the respective joints of the robot 100 to move the workpiece WK to the specified position by the robot 100. Therefore, the workpiece WK is arranged in the imaging range of the first camera 410 while being gripped by the robot 100. When the robot 100 moves the workpiece WK to the specified destination, the robot controller 200 notifies the information processing device 600 to that effect.

[0037] The information processing apparatus 600 identifies the inspection posture planned for the target workpiece WK based on the shooting posture information D1. The inspection posture represents the posture of the workpiece WK in which the first camera 410 can shoot one target part to be shot among the workpieces WK. The inspection posture is determined by the gripping part and the gripping posture of the robot 100 on the workpiece WK and the posture of the workpiece WK with respect to the first camera 410. In the present embodiment, a plurality of shooting postures in which a plurality of target parts can be respectively shot are set in advance.

[0038] FIG. 5 is an explanatory diagram showing the shooting posture information D1. In the shooting posture information D1, for each type of the workpiece WK, the inspection posture to be taken for the appearance inspection is defined. Here, it is assumed that the shapes of workpieces WK with different types are similar to each other. For example, when the type of the workpiece WK is "001", it is necessary to shoot the workpiece WK in postures 1 to 3 as the inspection posture.

[0039] The information processing apparatus 600 identifies, based on the inspection feasibility information D2, among the inspection postures planned for the target workpiece WK defined in the shooting posture information D1, the inspection postures that the workpiece WK can take with the current gripping part and the gripping posture.

[0040] FIG. 6 is an explanatory diagram showing the inspection feasibility information D2. In the inspection feasibility information D2, for the workpiece WK of the target type, for each gripped gripping part and gripping posture, the inspection postures that can be taken and the inspection postures that cannot be taken are defined.

[0041] For example, when the type of the workpiece WK is "001", and the gripping part is "side" and the gripping posture is "left horizontal", the workpiece WK can take postures 2 to 4. However, in this case, the workpiece WK cannot take posture 1. In FIG. 6, the inspection feasibility information D2 in the case where the type of the workpiece WK is "001" is illustrated, but the inspection feasibility information D2 is defined in advance for each type.

[0042] When it is necessary to change the posture of the workpiece WK to a planned inspection posture, the information processing apparatus 600 outputs an instruction to change the posture of the workpiece WK to the robot controller 200 together with information indicating the specified inspection posture. In response to this, the robot controller 200 controls the angles of the respective joints of the robot 100 to change the posture of the workpiece WK by the robot 100. Therefore, the workpiece WK is set to the specified inspection posture within the imaging range of the first camera 410 while being gripped by the robot 100. On the other hand, when it is not necessary to change the posture of the workpiece WK, the information processing apparatus 600 does not output an instruction to change the posture of the workpiece WK to the robot controller 200. The process of step S20 is executed by the CPU 650 as the object recognition unit 651 and the robot control unit 652.

[0043] In step S30, the inspection process is executed.

[0044] First, the information processing apparatus 600 instructs the first camera 410 to take a picture. In response to this, the first camera 410 takes a picture of the workpiece WK held by the robot 100 and outputs the acquired image to the information processing apparatus 600. The information processing apparatus 600 performs an appearance inspection of the workpiece WK held by the robot 100 using the image supplied from the first camera 410. As a method of appearance inspection using an image, for example, a known method is used in which the presence or absence of defects in the target workpiece WK is determined by comparing the feature amount of the image of the target workpiece WK with the feature amount of a reference image obtained by photographing a good product of the same type. The process of step S30 is executed by the CPU 650 as the camera control unit 653 and the inspection unit 654.

[0045] In step S40, the information processing apparatus 600 determines whether it is necessary to photograph the target workpiece WK in another inspection posture based on the imaging posture information D1 (see FIG. 5).

[0046] Here, the case where it is necessary to photograph the target work WK in another inspection posture means the case where it is necessary to photograph parts other than the target part photographed in the current inspection posture. Also, the case where it is not necessary to photograph the target work WK in another inspection posture means the case where all target parts have been photographed in the current or past postures and the inspection for all inspection items has been completed using the photographed images. Information regarding the association between inspection items and inspection postures is supposed to be stored in the memory 610.

[0047] As shown in FIG. 3, in step S40, when it is necessary to photograph the target work WK in another inspection posture (step S40; YES), the information processing apparatus 600 executes the process of step S50. When it is not necessary to photograph the target work WK in another inspection posture (step S40; NO), the information processing apparatus 600 executes step S80. The process of step S40 is executed by the CPU 650 as the inspection unit 654.

[0048] In step S50, the information processing apparatus 600 determines whether the robot 100 needs to re-grasp the target work WK based on the inspection feasibility information D2 (see FIG. 6). Re-grasping means that the robot 100 releases the work WK once and then grasps it again.

[0049] The case where it is necessary to re-grasp the target work WK is, for example, the case where the part of the work WK grasped by the robot 100 is hidden by a part of the gripper which is the end effector 150 and the part cannot be photographed. The case where it is not necessary to re-grasp the target work WK is the case where at least one of the position and posture of the work WK can be changed while the robot 100 holds the work WK and the images necessary for the inspection can be acquired.

[0050] For example, when the robot 100 grips the workpiece WK with the gripping part being the "side surface" and the gripping posture being "upper left", as defined in the inspection feasibility information D2 (see FIG. 6), the workpiece WK cannot take postures 1, 3, and 4. After the appearance inspection in posture 2 is performed, the robot 100 needs to re-grip the target workpiece WK.

[0051] As shown in FIG. 3, in step S50, when the robot 100 needs to re-grip the target workpiece WK (step S50; YES), the information processing device 600 executes step S60. When the robot 100 does not need to re-grip the target workpiece WK (step S50; NO), the information processing device 600 executes step S70. The process of step S50 is executed by the CPU 650 as the inspection unit 654.

[0052] In step S60, the process of the robot 100 re-gripping the workpiece WK is executed. First, the information processing device 600 identifies a new gripping part and a gripping posture as a new imaging posture based on the imaging posture information D1, the inspection feasibility information D2, and the inspection postures that have been imaged so far.

[0053] For example, assume that the inspection postures scheduled for the workpiece WK are postures 1 to 3. Also, assume that the robot 100 was able to image the workpiece WK in posture 2 when it first gripped the workpiece WK. In this case, it is necessary to image the workpiece WK in postures 1 and 3. As shown in FIG. 6, when the gripping part is the "side surface" and the gripping posture is "vertically downward", the workpiece can be imaged in postures 1 and 3. The information processing device 600 can easily identify the gripping part and the gripping posture with which the robot 100 should grip the workpiece WK by using the inspection feasibility information D2.

[0054] When there are multiple candidates for the gripping part and the gripping posture, for example, the one with the smallest value of the record number (not shown in FIG. 6) in the inspection feasibility information D2 is selected.

[0055] Also, when there is no gripping part and gripping posture that can take all of the plurality of inspection postures that need to be photographed, the gripping part and gripping posture that can take the largest number of inspection postures among the plurality of inspection postures that need to be photographed are selected. Therefore, the number of times the robot 100 re-grips the work WK can be reduced, and the desired range can be efficiently photographed.

[0056] The information processing device 600 outputs an instruction to re-grip the work WK together with information indicating a new gripping part and gripping posture to the robot controller 200. In response to this, the robot controller 200 causes the robot 100 to place the work WK on the second tray 520 by controlling the angles of the respective joints of the robot 100. Further, the robot controller 200 causes the robot 100 to grip the work WK in the second tray 520 in the designated gripping part and gripping posture by controlling the angles of the respective joints of the robot 100. The process of step S60 is executed by the CPU 650 as the robot control unit 652. After step S60 is executed, the process of step S30 is executed again.

[0057] In step S70, a process of changing the posture of the work WK held by the robot 100 is executed. First, the information processing device 600 identifies other inspection postures that the work WK can assume in the current gripping part and gripping posture using the photographing posture information D1 and the inspection availability information D2.

[0058] For example, assume that the inspection postures scheduled for the work WK are postures 1 to 3. Also, assume that the robot 100 was able to photograph the work WK in posture 2 when it first grasped the work WK. In this case, it is necessary to photograph the work WK in postures 1 and 3. Also, assume that the work WK can take posture 1 by rotating the work WK around an arbitrary rotation axis in the current gripping part and gripping posture. In this case, the information processing device 600 calculates the rotation axis and rotation angle for rotating the work WK so that the work WK takes posture 1. The information processing device 600 outputs an instruction to rotate the work WK to the robot controller 200 together with information specifying the rotation axis and rotation angle. In response to this, the robot controller 200 controls the angles of the respective joints of the robot 100 to rotate the work WK by the specified rotation angle around the specified rotation axis for the robot 100. The process of step S70 is executed by the CPU 650 acting as the robot control unit 652. After step S70 is executed, the process of step S30 is executed again.

[0059] In step S80, when the end condition for ending the process is satisfied (step S80; YES), the process shown in FIG. 3 ends. The end condition is, for example, that the appearance inspection of all the works WK accommodated in the first tray 510 has been completed. On the other hand, when the end condition is not satisfied (step S80; NO), the process of step S10 is executed again. The process of step S80 is executed by the CPU 650 acting as the inspection unit 654.

[0060] According to this embodiment, when the robot 100 can photograph the work WK in another photographing posture while holding the work WK, the robot 100 adjusts the posture of the work WK and then the work WK is photographed again. Since the work WK is photographed in as many photographing postures as possible with a single gripping, the desired range can be efficiently photographed. In addition, for unphotographed postures, since new gripping parts and gripping postures are specified based on the inspection feasibility information D2, it becomes easy to specify the gripping parts and gripping postures where the robot 100 should grip the work.

[0061] B. Other embodiments: B1. Other embodiment 1: In the embodiment, an example was described in which, at the time of re-gripping, the gripping part and the gripping posture that can take the most inspection postures among a plurality of inspection postures that need to be photographed are selected. However, at the time of re-gripping, it is not necessary to select the gripping part and the gripping posture that can take the most inspection postures among a plurality of inspection postures that need to be photographed.

[0062] For example, assume that the current gripping part is "side" and the gripping posture is "upper left". At the time of re-gripping, assume that the gripping part that can take the most inspection postures among a plurality of inspection postures that need to be photographed is "side" and the gripping posture is "lower vertical". In this case, when the work WK is once placed in the second tray 520, it is necessary to place the work WK in the second tray 520 with the claw part of the work facing down. If a jig or the like for reversing the up and down of the work is prepared in the second tray 520, the robot 100 only needs to place the work WK in the current posture on the jig, but if there is no such jig, it is necessary to change the holding of the work WK several times. In such a case, when the work WK is once placed in the second tray 520, a new gripping part and gripping posture that do not require reversing the up and down of the work WK may be selected.

[0063] B2. Other embodiment 2: The inspection feasibility information D2 may be set based on the following viewpoints. For example, the gripping part and gripping posture where the desired imaging range of the workpiece WK is not shielded by the gripping of the robot 100 may be set in the inspection feasibility information D2. In this case, in the inspection feasibility information D2, based on whether the target part is shielded by the gripping of the robot 100 or not, it is set whether it is the gripping part and gripping posture corresponding to the imaging posture capable of imaging the target part. By setting the inspection feasibility information D2, therefore, in the gripping part and gripping posture specified using the inspection feasibility information D2, the desired imaging range is not shielded by the gripping of the robot 100. Also, by using the inspection feasibility information D2, the gripping part and gripping posture where the robot 100 should grip the workpiece WK can be easily specified in order to image the workpiece WK in the non-imaged posture.

[0064] B3. Other Embodiment 3: Also, the inspection feasibility information D2 may be set based on the following viewpoints. The gripping part and gripping posture may be set in the inspection feasibility information D2 so that the angle θ formed between the rotation axis direction of the gripper equipped on the robot 100 and the imaging direction of the first camera 410 for imaging the workpiece WK is equal to or greater than a predetermined angle. The angle θ is preferably, for example, 90 degrees or more.

[0065] FIG. 7 is an explanatory diagram of the angle formed between the rotation axis direction of the gripper and the imaging direction of the first camera 410. In Example 1 of FIG. 7, the angle θ is 90 degrees or more. On the other hand, in Example 2 of FIG. 7, the angle θ is 90 degrees or less. In FIG. 7, the illustration of the robot arm 120 is omitted, but in the gripping posture like Example 2, the robot arm 120 may interfere with the first camera 410, or a part of the imaging range of the first camera 410 may be shielded by the robot arm 120. On the other hand, in the gripping posture like Example 1, such a problem does not occur.

[0066] In the inspection feasibility information D2, based on whether or not the angle formed by the rotational axis direction of the gripper equipped on the robot 100 and the imaging direction of the first camera 410 for imaging the workpiece WK is 90 degrees or more, it is set whether or not it is a gripping part and a gripping posture corresponding to an imaging posture capable of imaging the target part. Therefore, in the gripping part and the gripping posture specified using the inspection feasibility information D2, interference with the first camera 410 by the robot arm 120 and shielding of a part of the imaging range by the robot arm 120 do not occur. Also, in order to image the workpiece WK in an un-imaged posture, by using the inspection feasibility information D2, the gripping part and the gripping posture by which the robot 100 should grip the workpiece WK can be easily specified.

[0067] B4. Other Embodiment 4: Also, the inspection feasibility information D2 may be set based on the following viewpoints. The gripping part and the gripping posture in which the robot 100 gripping the workpiece WK can reach the imaging range of the first camera 410 may be set in the inspection feasibility information D2. The robot 100 moves the workpiece WK by rotational drive of six axes. For this reason, depending on the gripping part and the gripping posture of the workpiece WK, there may be a case where the imaging range of the first camera 410 cannot be reached. Therefore, in the inspection feasibility information D2, based on whether or not the workpiece WK gripped by the robot 100 can reach the imaging range of the first camera 410, it is set whether or not it is the gripping part and the gripping posture corresponding to an imaging posture capable of imaging the target part. For example, based on data collected by simulation using a computer, determination for each gripping part and gripping posture may be made as to whether or not the imaging range of the first camera 410 can be reached. In order to image the workpiece WK in an un-imaged posture, by using the inspection feasibility information D2, the gripping part and the gripping posture by which the robot 100 should grip the workpiece WK can be easily specified.

[0068] B5. Other Embodiment 5: The inspection feasibility information D2 may include all or part of the gripping parts and gripping postures set by the methods described in the above Other Embodiments 2 to Other Embodiment 4.

[0069] B6. Other Embodiment 6: In the embodiment, an example was described in which the work WK to be gripped by the robot 100 is identified based on the image acquired by the second camera 420 and the projection image by the depth sensor 430 for the plurality of works WK accommodated in the first tray 510, and an arbitrary gripping posture is selected for the identified work WK. However, even when gripping any work WK for the first time, for example, using the estimated position and posture of the work WK, the imaging posture information D1, and the inspection permission information D2, a gripping posture that can efficiently perform the appearance inspection may be selected. For example, a gripping part and a gripping posture that can take the most inspection postures among the plurality of inspection postures that need to be imaged may be selected. Alternatively, among the plurality of inspection postures that need to be imaged, the inspection posture with the smallest value of the code for identifying the inspection posture is selected, and a gripping part and a gripping posture that can take that inspection posture may be selected.

[0070] B7. Other Embodiment 7: In the embodiment, an example of performing an appearance inspection using the images obtained by individually imaging the work WK was described. However, the imaging method described in the embodiment can be applied to other uses. For example, by saving the images obtained by imaging the work WK as logs, the imaging method described in the embodiment can be applied to the use of log collection.

[0071] B8. Other Embodiment 8: In the embodiment, an example was shown in which the work WK has a rectangular parallelepiped shape (see FIG. 4), but the shape of the work WK is not limited to this. For example, it may be a polyhedron having polygonal faces, or a prism having a polygon as the base. Further, the work WK may be a sphere provided with convex or concave portions that allow the gripper to grip the work WK.

[0072] B9. Other Embodiment 9: In the embodiment, an example where the end effector 150 is a two-finger type gripper (see FIG. 1) has been described, but the end effector 150 may be a three-finger type gripper. Alternatively, the end effector 150 may be a suction hand using a vacuum suction method or a suction hand using a magnetic adsorption method.

[0073] B10. Other Embodiment 10: The means for realizing the functions of the information processing apparatus 600 is not limited to software, and part or all of it may be realized by dedicated hardware. For example, as the dedicated hardware, a circuit typified by an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used.

[0074] C. Other Forms: The present disclosure is not limited to the above-described embodiments, and can be realized in various configurations without departing from the gist thereof. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be appropriately replaced and combined in order to solve some or all of the above-described problems or to achieve some or all of the above-described effects. Further, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0075] (1) According to the first aspect of the present disclosure, a photographing method for photographing a workpiece is provided. This photographing method is a photographing method for photographing a workpiece, and includes: (a) a step of causing a robot to grip the workpiece; (b) a step of adjusting at least one of the position and orientation of the workpiece with respect to a camera by the robot so that the orientation of the gripped workpiece becomes any one of a plurality of predetermined photographing orientations capable of photographing a plurality of target parts to be photographed among the workpiece; (c) a step of causing the camera to photograph the target part of the workpiece held by the robot and set in any one of the photographing orientations; (d) when the workpiece can take another photographing orientation that is a photographing orientation capable of photographing an unphotographed target part without the robot gripping the workpiece again and is different from the photographing orientation in which step (c) has been executed until then, a step of executing steps (b) and (c) with the other photographing orientation as a new photographing orientation; and (e) when the workpiece cannot take the other photographing orientation that is a photographing orientation capable of photographing an unphotographed target part even if the robot changes the position and orientation of the workpiece while still gripping the workpiece, the robot is controlled to grip the workpiece again with a new gripping part and a gripping orientation specified based on the photographing orientation information associating the gripping part of the robot gripping the workpiece and the gripping orientation of the robot with respect to the workpiece with the photographing orientation capable of photographing the target part, and the photographing orientation in which step (c) has been executed until then, and steps (b) and (c) are executed with the other photographing orientation as the new photographing orientation. According to this embodiment, when the robot can photograph the workpiece in another photographing posture while holding the workpiece, the position and posture of the workpiece are adjusted for the robot and then the workpiece is photographed again. Since the workpiece is photographed in as many photographing postures as possible with a single gripping, the desired range can be efficiently photographed. Also, when the robot cannot photograph an unphotographed target part while holding the workpiece, the robot is controlled to re-grip the workpiece at a new gripping part and gripping posture specified based on the photographing posture information. The gripping part and gripping posture where the robot should grip the workpiece can be easily specified, and the workpiece can be efficiently photographed. (2) In the photographing method of the above embodiment, in step (e), based on the photographing posture information, the gripping part and gripping posture in the photographing posture that can photograph the most of the unphotographed target parts among the unphotographed target parts may be specified. According to this embodiment, the number of times the robot re-grips the workpiece can be reduced, and the desired range can be efficiently photographed. (3) In the photographing method of the above embodiment, (f) a step executed before step (a), causing a second camera different from the camera to photograph a plurality of the workpieces in a state where the robot is not holding them, and based on a plurality of images of the workpieces obtained by the second camera, estimating the position and posture of at least the workpiece specified as the target to be gripped by the robot among the plurality of workpieces, and based on the estimated position and posture of the specified workpiece, specifying at least one gripping part and gripping posture of the workpiece by which the robot can grip the workpiece, and further including a step of causing the robot to grip the specified workpiece in the gripping part and gripping posture. (4) In the photographing method of the above embodiment, in step (f), based on the estimated position and posture of the specified workpiece and the photographing posture information, the gripping part and gripping posture in the photographing posture that can photograph the workpiece in the most of the predetermined plurality of photographing postures may be specified. (5) In the imaging method of the above-described embodiment, in the imaging posture information, whether or not the gripping part and the gripping posture corresponding to the imaging posture capable of imaging the target part are set based on whether or not the target part is shielded by the gripping of the robot may be determined. According to this embodiment, since the gripping part and the gripping posture in which the target part is not shielded by the gripping of the robot are set in the imaging posture information, in order to image the workpiece in the un-imaged posture, the gripping part and the gripping posture by which the robot should grip the workpiece can be easily specified. (6) In the imaging method of the above-described embodiment, in the imaging posture information, whether or not the gripping part and the gripping posture corresponding to the imaging posture capable of imaging the target part are set based on whether or not the angle formed by the rotation axis direction of the gripper equipped on the robot and the imaging direction of the camera for imaging the workpiece is 90 degrees or more may be determined. According to this embodiment, since the gripping part and the gripping posture that can suppress the interference of the robot arm with the camera are set in the imaging posture information, in order to image the workpiece in the un-imaged posture, the gripping part and the gripping posture by which the robot should grip the workpiece can be easily specified. (7) In the imaging method of the above-described embodiment, in the imaging posture information, whether or not the gripping part and the gripping posture corresponding to the imaging posture capable of imaging the target part are set based on whether or not the workpiece gripped by the robot can reach the imaging range of the camera may be determined. According to the above embodiment, since the gripping part and the gripping posture that can reach the imaging range of the camera are set in the imaging posture information, in order to image the workpiece in the un-imaged posture, the gripping part and the gripping posture by which the robot should grip the workpiece can be easily specified. (8) In the appearance inspection method using the imaging method of the above-described embodiment, the appearance of the workpiece may be inspected using an image obtained by the camera imaging the workpiece held by the robot in any of the imaging postures. (9) According to a second aspect of the present disclosure, a control device for controlling the operation of a robot is provided. After the robot holding the workpiece adjusts at least one of the position and posture of the workpiece with respect to the camera so that the posture of the workpiece becomes any one of a plurality of predetermined photographing postures capable of photographing a plurality of target parts to be photographed among the workpiece, and any one of the target parts is photographed one or more times by the camera, (i) when the workpiece can take another photographing posture different from the photographing postures that have been photographed so far, which is a photographing posture capable of photographing the unphotographed target parts without the robot re-grasping the workpiece, the robot is controlled to change the position and posture of the workpiece with respect to the camera so as to use the other photographing posture as a new photographing posture; and (ii) when the workpiece cannot take the other photographing posture capable of photographing the unphotographed target parts even if the robot changes the position and posture of the workpiece while holding the workpiece, the robot is controlled to re-grasp the workpiece with a new gripping part and a gripping posture specified based on the photographing posture information associating the gripping part of the robot for gripping the workpiece and the gripping posture of the robot with respect to the workpiece with the photographing posture capable of photographing the target part and the photographing postures that have been photographed so far. At least one of the above steps is executed. According to this aspect, when the robot can photograph the workpiece in another photographing posture while holding the workpiece, the position and posture of the workpiece are adjusted by the robot and then the workpiece is photographed again. Since the workpiece is photographed in as many photographing postures as possible with a single gripping, the desired range can be efficiently photographed. Further, when the robot cannot photograph the unphotographed target parts while holding the workpiece, the robot is controlled to re-grasp the workpiece with a new gripping part and a gripping posture specified based on the photographing posture information. The gripping part and the gripping posture by which the robot should grip the workpiece can be easily specified, and the workpiece can be efficiently photographed. (10) According to the third aspect of the present disclosure, a photographing system having a camera and a control device is provided. In this photographing system, the control device is a robot control unit that controls the operation of the robot, and at least one of the position and posture of the workpiece is adjusted by the robot so as to be in any one of a plurality of predetermined photographing postures while the workpiece is being held. A robot control unit that causes the robot to adjust, and a camera control unit that controls the photographing operation of the camera so as to photograph the workpiece in the any one of the photographing postures while being held by the robot. When the robot can photograph an unphotographed target part without re-grasping the workpiece, regarding the change in the position and posture of the workpiece by the robot and the photographing of the workpiece by the camera, each control by the robot control unit and the camera control unit is repeated. Even if the robot changes the position and posture of the workpiece while holding the workpiece, if the unphotographed target part cannot be photographed, the robot control unit determines the gripping part of the robot that holds the workpiece and the gripping posture of the robot with respect to the workpiece, and the photographing posture that can photograph the target part. The robot is controlled to re-grasp the workpiece with a new gripping part and gripping posture specified based on the photographing posture information and the photographing posture that has been photographed so far, and the camera control unit controls the camera so as to photograph the target part of the workpiece. According to this aspect, when the robot can photograph the workpiece in another photographing posture while holding the workpiece, the position and posture of the workpiece are adjusted by the robot and then the workpiece is photographed again. Since the workpiece is photographed in as many photographing postures as possible with a single gripping, the desired range can be efficiently photographed. Further, when the robot cannot photograph an unphotographed target part while holding the workpiece, the robot is controlled to re-grasp the workpiece with a new gripping part and gripping posture specified based on the photographing posture information. The gripping part and gripping posture in which the robot should grip the workpiece can be easily specified, and the workpiece can be photographed efficiently.

Explanation of Reference Numerals

[0076] 1... Robot system, 100... Robot, 110... Base, 120... Robot arm, 130... Force sensor, 150... End effector, 200... Robot controller, 410... First camera, 420... Second camera, 430... Depth sensor, 500... Workbench, 510... First tray, 520... Second tray, 600... Information processing device, 610... Memory, 620... Input interface circuit, 630... Input device, 640... Display device, 650... CPU, 651... Object recognition unit, 652... Robot control unit, 653... Camera control unit, 654... Inspection unit, CC1... First camera coordinate system, CC2... Second camera coordinate system, CL... Gripping claw, D1... Photographing posture information, D2... Inspection permission information, J1~J6... Joints, RC... Robot coordinate system, SC... Sensor coordinate system, WC... World coordinate system, WK... Work

Claims

1. A photographing method for photographing a workpiece, comprising: (a) causing a robot to grip the workpiece; (b) adjusting at least one of the position and orientation of the workpiece with respect to a camera by the robot so that the orientation of the gripped workpiece becomes any one of a plurality of predetermined photographing orientations capable of photographing a plurality of target parts to be photographed among the workpiece; (c) causing the camera to photograph the target part of the workpiece held by the robot and set to any one of the photographing orientations; (d) when the workpiece can assume another photographing orientation that can photograph the unphotographed target parts without the robot gripping the workpiece again and is different from the photographing orientation in which step (c) has been executed so far, executing steps (b) and (c) using the other photographing orientation as a new photographing orientation; (e) when the workpiece cannot assume the other photographing orientation that can photograph the unphotographed target parts even if the robot changes the position and orientation of the workpiece while gripping the workpiece, controlling the robot to regrip the workpiece with a new gripping part and gripping orientation specified based on the photographing orientation information associating the gripping part of the robot for gripping the workpiece and the gripping orientation of the robot with respect to the workpiece with the photographing orientation capable of photographing the target part, and the photographing orientation in which step (c) has been executed so far, and executing steps (b) and (c) using the other photographing orientation as the new photographing orientation; A photographing method comprising the above steps.

2. The photographing method according to claim 1, wherein in step (e), the gripping part and the gripping orientation in the photographing orientation capable of photographing the largest number of the unphotographed target parts are specified based on the photographing orientation information. A photographing method.

3. The photographing method according to claim 1, wherein (f) a step executed before step (a), causing a second camera different from the camera to photograph a plurality of the workpieces in a state where the robot is not gripping them. Based on the images of the plurality of workpieces acquired by the second camera, estimate the position and orientation of at least the workpiece identified as the object to be gripped by the robot among the plurality of workpieces, Based on the estimated position and orientation of the identified workpiece, identify at least one gripping part and gripping posture of the workpiece by which the robot can grip the workpiece, Cause the robot to grip the identified workpiece in the gripping part and gripping posture, Further include the step of, Imaging method.

4. The imaging method according to claim 3, wherein In the step (f), For the identified workpiece, based on the estimated position and orientation and the imaging posture information, the gripping part and gripping posture in which the workpiece can be imaged in the most imaging postures among the plurality of predetermined imaging postures are identified, Imaging method.

5. The imaging method according to claim 1, wherein In the imaging posture information, based on whether or not the target part is shielded by the gripping of the robot, it is set whether or not it is the gripping part and gripping posture corresponding to the imaging posture in which the target part can be imaged, Imaging method.

6. The imaging method according to claim 1, wherein In the imaging posture information, based on whether or not the angle formed by the rotation axis direction of the gripper equipped on the robot and the imaging direction of the camera for imaging the workpiece is 90 degrees or more, it is set whether or not it is the gripping part and gripping posture corresponding to the imaging posture in which the target part can be imaged, Imaging method.

7. The imaging method according to claim 1, wherein In the imaging posture information, based on whether or not the workpiece gripped by the robot can reach the imaging range of the camera, it is set whether or not it is the gripping part and gripping posture corresponding to the imaging posture in which the target part can be imaged, Imaging method.

8. An appearance inspection method using the imaging method according to any one of claims 1 to 7, wherein An appearance inspection of the workpiece is performed using an image obtained by the camera imaging the workpiece held by the robot in any of the imaging postures, Appearance inspection method.

9. A control device for controlling the operation of a robot, The robot that holds the workpiece adjusts at least one of the position and orientation of the workpiece with respect to the camera so that the posture of the workpiece becomes any one of a plurality of predetermined imaging postures capable of imaging a plurality of target parts to be imaged among the workpiece. After any one of the target parts has been imaged one or more times by the camera, (i) If the workpiece can assume another imaging posture that is different from the imaging postures that have been imaged so far and that enables the robot to image the un-imaged target parts without re-grasping the workpiece, Controlling the robot to change the position and orientation of the workpiece with respect to the camera, using the other imaging posture as a new imaging posture; (ii) If the workpiece cannot assume the other imaging posture that enables the robot to image the un-imaged target parts even when the robot changes the position and orientation of the workpiece while holding the workpiece, Controlling the robot to re-grasp the workpiece with a new gripping part and gripping posture specified based on the imaging posture information associating the gripping part of the robot that holds the workpiece and the gripping posture of the robot with respect to the workpiece with the imaging posture capable of imaging the target part, and the imaging postures that have been imaged so far; Executing at least one of the above; A control device.

10. An imaging system having a camera and a control device, The control device includes: A robot control unit that controls the operation of the robot, and adjusts at least one of the position and orientation of the workpiece to the robot so that the workpiece assumes any one of a plurality of predetermined imaging postures while being held; A camera control unit that controls the imaging operation of the camera to image the workpiece that has been placed in any one of the imaging postures while being held by the robot; And includes: When the robot can image the un-imaged target parts without re-grasping the workpiece, For the change in the position and orientation of the workpiece by the robot and the imaging of the workpiece by the camera, the respective controls by the robot control unit and the camera control unit are repeated. Even if the robot changes the position and orientation of the workpiece while holding the workpiece, if the unphotographed target part cannot be photographed, the robot control unit controls the robot to re-grasp the workpiece with a new grasping part and a grasping posture specified based on the imaging posture information associating the grasping part where the robot grasps the workpiece and the grasping posture of the robot with respect to the workpiece with the imaging posture in which the target part can be imaged, and the imaging posture in which imaging has been performed so far, and the camera control unit controls the camera to image the target part of the workpiece. Imaging system.

Images