Robot system and method for controlling robot system

Abstract

The robot system (100) acquires a reference image in a state in which the robot arm (11) is disposed in one reference posture during the calibration operation. Then, on the basis of the acquired reference image, the robot system (100) sets a plurality of calibration postures of the robot arm (11) such that the recognition component is imaged by the imaging unit (21, 22) in a state in which the recognition component is included in the field of view of the imaging unit (21, 22), and performs calibration of the robot arm (11) for each of the set plurality of calibration postures. A correction image for calibration obtained by imaging an identification member by an imaging unit (21, 22) is acquired.

Description

Technical Field

[0001] The present disclosure relates to a robot system and a control method of the robot system. Background Art

[0002] In the past, a robot system that controls the movement of a robot arm based on a captured image captured by a capturing unit is known. For example, Japanese Patent Gazette No. 2022-028672 discloses a system having a visual system camera and a robot arm. In the system disclosed in Japanese Patent Gazette No. 2022-028672, a calibration action, i.e., hand-eye calibration, is performed in order to control the movement of the robot arm within a scene captured by a camera. In the hand-eye calibration, a calibration object mounted on the robot arm is moved so that images of a plurality of differently configured calibration objects are captured by the camera. Then, by determining the features of the calibration object within the captured image, the conversion from the robot coordinate space to the camera coordinate space is calculated.

[0003] In the system described in Japanese Patent Gazette No. 2022-028672, a pre-correction process is performed in such a way that the correction object is arranged within the field of view of the camera in the hand-eye correction. In the pre-correction process, a three-point correction is performed within the field of view of the camera, and a rough conversion from the robot coordinate space to the image coordinate space is calculated. Specifically, in the pre-correction process, first, the characteristic pattern of the correction object is arranged in the central area of ​​the field of view of the camera. Then, the correction object is moved while adjusting the step size as the moving distance, so as to perform a three-point correction using the feature points of the correction object that have been identified. Based on the rough conversion from the robot coordinate space to the camera coordinate space and the size of the field of view of the camera, the spatial position that maintains the features of the correction object within the field of view of the camera is calculated, thereby setting the range of motion of the robot in the hand-eye correction.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2022-028672

[0005] However, in the robot system described in the above-mentioned Japanese Patent Publication No. 2022-028672, in order to perform hand-eye calibration to establish a correspondence between the robot coordinate space as the position information of the robot arm and the camera image coordinate system as the position information of the captured image, a pre-calibration process is performed to calculate a rough conversion by performing three-point calibration while changing the position of the robot arm. Therefore, in the calibration action, i.e., hand-eye calibration, in order to set multiple postures of the robot arm, it is necessary to move the calibration object to the three-point position in advance and obtain an image, and it is necessary to perform a rough conversion operation process to establish a correspondence between the position information of the robot arm and the position information of the captured image in advance based on the image of the three-point position that has been obtained, so it is considered that the time and processing required for the control processing in the calibration action increase. Therefore, it is desirable to suppress the increase in the time and processing required for the control processing in the calibration action of establishing a correspondence between the position information of the robot arm and the position information of the captured image. Summary of the invention

[0006] The present disclosure is completed to solve the problems as mentioned above. One purpose of the present disclosure is to provide a robot system and a control method of the robot system that can suppress the increase in the time and processing volume required for control processing in the calibration action of establishing a correspondence between the position information of the robot arm and the position information of the captured image.

[0007] In order to achieve the above-mentioned purpose, the robot system of the first aspect of the present invention comprises: a robot arm for mounting a hand that operates on a workpiece; a photographing unit that photographs the workpiece; and a control unit that performs a calibration action to establish a correspondence between position information of the robot arm and position information of the photographed image in order to move the robot arm based on a photographed image photographed by the photographing unit. The control unit obtains a photographed image, i.e., a reference image, obtained by photographing a calibration identification component by the photographing unit when the robot arm is configured in a posture, and based on the obtained reference image, sets a plurality of calibration postures of the robot arm in a manner in which the identification component is photographed by the photographing unit in a state in which the identification component is included in the field of view of the photographing unit during the calibration action, and obtains a calibration photographed image, i.e., a correction image, obtained by photographing the identification component by the photographing unit for each of the plurality of set calibration postures.

[0008] In the robot system of the first aspect of the present disclosure, as described above, the control unit obtains a reference image obtained by the imaging unit when the robot arm is configured in a posture. Then, the control unit sets a plurality of postures for calibration of the robot arm in a manner that the imaging unit images the recognition component in a state where the recognition component is included in the field of view of the imaging unit in the calibration operation based on the acquired reference image. Then, the control unit obtains a calibration image obtained by the imaging unit imaging the recognition component for each of the plurality of postures for calibration that have been set. Thus, the plurality of postures for calibration of the robot arm are set based on the reference image, so that the time required for the control process to set the posture for calibration can be shortened and the amount of control processing can be reduced compared to the case where the posture for calibration is set by roughly converting the plurality of imaging images in advance as in three-point calibration. As a result, in the calibration operation of establishing a correspondence between the position information of the robot arm and the position information of the imaging image, the increase in the time required for the control process and the amount of processing can be suppressed.

[0009] In addition, in order to achieve the above-mentioned purpose, the control method of the robot system of the second aspect of the present disclosure comprises the following steps: performing a calibration action, in which a correspondence is established between the position information of the robot arm installed for the hand working on the workpiece and the position information of the captured image captured by the capturing unit for capturing the workpiece; and after performing the calibration action, controlling the action of the robot arm based on the captured image captured by the capturing unit, the step of performing the calibration action comprises the following steps: obtaining a captured image, i.e., a reference image, obtained by capturing a calibration identification component by the capturing unit in a state where the robot arm is configured in a posture; based on the obtained reference image, setting a plurality of calibration postures of the robot arm in a manner in which the identification component is captured by the capturing unit in a state where the identification component is included in the field of view of the capturing unit; and obtaining a calibration captured image, i.e., a correction image, obtained by capturing the identification component by the capturing unit for each of the plurality of calibration postures that have been set.

[0010] In the control method of the robot system of the second aspect of the present disclosure, as described above, a reference image is obtained by capturing a calibration recognition component by a capturing unit in a state where the robot arm is configured in a single posture. Then, based on the reference image that has been acquired, a plurality of calibration postures of the robot arm are set in a manner that the recognition component is captured by the capturing unit in a state where the recognition component is included in the field of view of the capturing unit. Then, for each of the plurality of calibration postures that have been set, a calibration captured image, that is, a correction image, is acquired by capturing the recognition component by the capturing unit. Thus, the plurality of calibration postures of the robot arm are set based on a reference image that is a captured image, so that compared with the case where the calibration posture is set by roughly converting the plurality of captured images in advance as in three-point calibration, the time required for the control process to set the calibration posture can be shortened, and the amount of control processing can be reduced. As a result, a control method for the robot system can be provided that can suppress the increase in the time required for the control process and the amount of processing in the calibration operation of establishing a correspondence between the position information of the robot arm and the position information of the captured image.

[0011] According to the present disclosure, as described above, it is possible to provide a robot system and a control method of the robot system that can suppress an increase in the time and processing amount required for control processing in a calibration operation for establishing a correspondence between position information of a robot arm and position information of a captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 : is a schematic diagram showing the overall structure of a robot system according to one embodiment of the present disclosure.

[0013] Figure 2 FIG. 1 is a diagram showing an example of images captured by each of two imaging units.

[0014] Figure 3 This is a diagram for explaining a plurality of correction images for calibration.

[0015] Figure 4 It is a diagram for explaining the fields of view of two imaging units and the arrangement of recognition components.

[0016] Figure 5 is a diagram showing an example of the reference image 51 a.

[0017] Figure 6 is a diagram showing an example of the reference image 52 a.

[0018] Figure 7 This is a diagram for explaining how to obtain four calibration postures on a reference plane.

[0019] Figure 8FIG. 1 is a diagram showing an example of a corrected image in four calibration postures on a reference plane.

[0020] Fig. 9 This is a diagram for explaining how to obtain four calibration postures on the depth side compared with the reference plane.

[0021] Fig.10 FIG. 1 is a diagram showing an example of corrected images in four calibration postures on the deeper side than the reference plane.

[0022] Fig.11 This is a diagram for explaining how to obtain four calibration postures that are closer to the front side than the reference plane.

[0023] Fig.12 FIG. 1 is a diagram showing an example of corrected images in four calibration postures that are closer to the front side than the reference plane.

[0024] Fig.13 Detailed description is a flowchart showing a control method of a robot system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments in which the present disclosure is embodied will be described based on the drawings.

[0026] (Structure of the robot system)

[0027] Reference Figures 1 to 12 , the structure of a robot system 100 according to an embodiment of the present disclosure is described.

[0028] like Figure 1 As shown, the robot system 100 is a system in which the robot arm 11 performs an operation on the workpiece W captured by the imaging unit 21 and the imaging unit 22. For example, the robot arm 11 performs an operation of transferring the workpiece W disposed near the robot arm 11 to a predetermined position. Figure 1 2 shows an example in which the workpiece W is a corrugated cardboard box.

[0029] The robot system 100 includes a robot 10, an imaging unit 21, an imaging unit 22, and a visual controller 30. In addition, the imaging unit 21 is an example of a "photographing unit" and a "first imaging unit" in the claims. In addition, the imaging unit 22 is an example of a "photographing unit" and a "second imaging unit" in the claims. In addition, the visual controller 30 is an example of a "control unit" in the claims.

[0030] The robot 10 includes a robot arm 11 and a robot controller 12. The robot arm 11 is a vertical multi-joint robot arm having a plurality of joints. A hand H for working on a workpiece W is mounted on the robot arm 11. For example, the hand H holds the workpiece W. The hand H is mounted on the front end of the robot arm 11. The robot arm 11 is controlled by the robot controller 12 to move the plurality of joints. For example, the robot arm 11 is moved by being driven by a driving unit such as a servo motor provided at the plurality of joints. In addition, the robot controller 12 is an example of a "control unit" in the claims.

[0031] The robot controller 12 controls the motion of the robot arm 11 . In addition, the robot controller 12 controls the motion of the hand H attached to the robot arm 11 . The robot controller 12 includes, for example, a processor such as a CPU (Central Processing Unit) and a memory for storing information. The robot controller 12 communicates with the vision controller 30 .

[0032] The photographing unit 21 and the photographing unit 22 photograph the workpiece W. The photographing unit 21 and the photographing unit 22 include photographing elements such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The photographing unit 21 is arranged at a first position different from the robot arm 11. The photographing unit 22 is arranged separately from the photographing unit 21 at a second position different from the robot arm 11. The first position and the second position are each, for example, a ground surface, a wall surface, or a base portion of the robot arm 11 separated from the robot arm 11. That is, the photographing unit 21 and the photographing unit 22 are fixedly arranged at a position for photographing the workpiece W relatively independently of the movement of the robot arm 11. Moreover, the photographing unit 21 and the photographing unit 22 are respectively arranged to photograph a common workpiece W together. The photographing unit 21 and the photographing unit 22 are stereo cameras for obtaining the three-dimensional position of the workpiece W. Therefore, in order to identify the position of the workpiece W by triangulation, the photographing unit 21 and the photographing unit 22 are fixed in a state separated from each other by a predetermined distance in the left-right direction. The imaging unit 21 and the imaging unit 22 are connected in parallel to the vision controller 30 .

[0033] The visual controller 30 is an image processing device that processes images captured by the imaging unit 21 and the imaging unit 22. The visual controller 30 is, for example, a personal computer including a processor such as a CPU and a GPU (Graphics Processing Unit), and a memory for storing information, a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0034] like Figure 2 As shown, the visual controller 30 acquires the captured image 41 of the imaging unit 21 and the captured image 42 of the imaging unit 22 by controlling the imaging unit 21 and the imaging unit 22. Then, the visual controller 30 detects the workpiece W included in each of the captured image 41 and the captured image 42 by performing image processing on the acquired captured image 41 and the captured image 42. In addition, the captured image 41 is an example of the "first captured image" and the "captured image" in the claims. In addition, the captured image 42 is an example of the "second captured image" and the "captured image" in the claims.

[0035] (Control of the robot arm based on captured images)

[0036] In the robot system 100, the robot arm 11 grasps the workpiece W based on the captured images 41 and 42, which are images of the workpiece W captured by the imaging unit 21 and the imaging unit 22, through control processing performed by the robot controller 12 and the vision controller 30. The vision controller 30 outputs a signal for controlling the operation of the robot arm 11 to the robot controller 12 based on the captured images 41 captured by the imaging unit 21 and the captured images 42 captured by the imaging unit 22. Specifically, in order to control the operation of the robot arm 11, the vision controller 30 acquires the three-dimensional position of the workpiece W by performing image processing on the captured images 41 captured by the imaging unit 21 and the captured images 42 captured by the imaging unit 22. Then, the vision controller 30 outputs a signal indicating the acquired three-dimensional position of the workpiece W to the robot controller 12. The robot controller 12 controls the operation of the robot arm 11 in a manner to operate the workpiece W based on the signal from the vision controller 30. That is, the robot controller 12 recognizes the three-dimensional position of the workpiece W based on the signal from the vision controller 30. Then, based on the recognized position of the workpiece W, the robot controller 12 operates the robot arm 11 and the hand H so that the hand H attached to the robot arm 11 grasps the workpiece W.

[0037] <Calibration Action>

[0038] Here, in the robot system 100, when the robot arm 11 grasps the workpiece W based on the captured images 41 and 42, it is necessary to identify the relative relationship between the coordinate system of the imaging unit 21 and 22 and the coordinate system of the robot arm 11. Therefore, the robot controller 12 and the visual controller 30 communicate with each other and perform a calibration operation in advance to establish a correspondence between the position information of the robot arm 11 and the position information of the captured images 41 and 42 in order to operate the robot arm 11 based on the captured images 41 and 42 captured by the imaging unit 21 and 22. That is, in the present embodiment, the robot controller 12 and the visual controller 30 perform control processing while communicating with each other, and act as a control unit that performs a calibration operation. The calibration operation is performed, for example, when the robot system 100 is set up or when the imaging unit 21 and 22 are replaced.

[0039] like Figure 3 As shown, in the calibration operation, while the posture of the robot arm 11 is changed by the robot controller 12, the recognition component 101 for calibration attached to the hand H at the front end of the robot arm 11 is photographed multiple times by the photographing units 21 and 22, so that a plurality of photographed images 41 for calibration, that is, a plurality of correction images 51 and a plurality of photographed images 42 for calibration, that is, a plurality of correction images 52, are obtained by the vision controller 30. Then, the vision controller 30 recognizes the position and angle of the recognition component 101 for calibration included in the plurality of correction images 51 and the plurality of correction images 52 by image processing, thereby performing a calculation process for establishing a correspondence between the position information of the robot arm 11 and the position information of the photographed images 41 and the photographed images 42. The recognition component 101 is a plate-shaped component on which a plurality of recognition points 101a are arranged in a grid.

[0040] Specifically, if Figure 4 As shown, in the calibration operation, the robot controller 12 moves the robot arm 11 to a plurality of calibration postures, thereby disposing the recognition component 101 mounted on the hand H at a position that is included in both the field of view 21a of the photographed portion 21 and the field of view 22a of the imaging portion 22. The visual controller 30 controls the imaging portion 21 and the imaging portion 22 to photograph the recognition component 101 disposed at a position that is included in both the field of view 21a of the photographed portion 21 and the field of view 22a of the imaging portion 22 in each of the plurality of calibration postures. In the present embodiment, the robot controller 12 automatically sets a plurality of calibration postures of the robot arm 11. In addition, the posture of the robot arm 11 mentioned here refers to the configuration position and configuration angle of the robot arm 11.

[0041] like Figure 5As shown, in the calibration action, in order to obtain a plurality of postures for calibration, first, a reference image 51a serving as a reference is obtained. In the present embodiment, the visual controller 30 obtains the reference image 51a obtained by photographing the recognition component 101 by the photographing unit 21 in a state where the robot arm 11 is configured in a reference posture. The reference image 51a is a captured image 41 obtained by photographing the recognition component 101 of the robot arm 11 configured in the reference posture by the photographing unit 21. In the reference posture, the recognition component 101 is configured in a position that is approximately the center of the field of view 21a of the photographing unit 21 and is approximately the center of the field of view 22a of the photographing unit 22. Therefore, the reference image 51a is captured in a manner in which the recognition component 101 is configured in the approximate center of the image. In addition, the reference image 51a is an example of a "reference image" and a "first reference image" in the claims.

[0042] like Figure 6 As shown, similarly to the reference image 51a, in order to obtain a plurality of calibration postures, a reference image 52a is obtained by photographing the recognition component 101 of the hand H arranged on the robot arm 11 by the imaging unit 22 in a state where the robot arm 11 is arranged in one reference posture. The reference image 52a photographed by the imaging unit 22 is also photographed in a manner such that the recognition component 101 is arranged approximately at the center of the image, similarly to the reference image 51a. In addition, the reference image 52a is an example of the "reference image" and the "second reference image" in the claims.

[0043] In addition, in the reference posture, the recognition component 101 is arranged so that the surface on which the recognition points 101a of the recognition component 101 are arranged faces both the imaging unit 21 and the imaging unit 22. The imaging unit 21 and the imaging unit 22 are arranged separately from each other, so in the state of the robot arm 11 arranged in the reference posture, the recognition component 101 is arranged so that the surface on which the recognition points 101a are arranged faces the middle point where each of the imaging unit 21 and the imaging unit 22 is arranged. Therefore, in both the reference image 51a and the reference image 52a, the recognition component 101 is photographed in a state in which the surface on which the plurality of recognition points 101a are arranged faces substantially the front.

[0044] In addition, the control amount for configuring the robot arm 11 to the reference posture is set based on the input performed by the operator. For example, the visual controller 30 displays the images from each of the imaging unit 21 and the imaging unit 22, that is, the captured images 41 and the captured images 42, on a display device not shown in the figure, such as a display connected to the visual controller 30, and overlaps and displays a cross-shaped image for identifying the center position on each of the captured images 41 and the captured images 42. Then, the operator configures the recognition component 101 at the center position of both the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22 while visually confirming the cross-shaped image for identifying the center position on the display device not shown. For example, based on the input operation of the operating unit such as the keyboard and mouse not shown connected to the visual controller 30 or the robot controller 12, the robot controller 12 changes the posture of the robot arm 11 to adjust the configuration of the recognition component 101, thereby setting the reference posture.

[0045] Then, in the present embodiment, the robot controller 12 and the vision controller 30 set a plurality of calibration postures of the robot arm 11 based on the acquired reference images 51a and 52a so that the recognition component 101 is captured by each of the imaging unit 21 and the imaging unit 22 in a state where the recognition component 101 is included in both the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22 during the calibration operation. The robot controller 12 and the vision controller 30 set a plurality of calibration postures of the robot arm 11 based on the distance D of the recognition position of the actual recognition component 101 and the distance D of the recognition position of the recognition component 101 on the image of the reference image 51a during the calibration operation. The distance D of the recognition position is the distance between the plurality of recognition points 101a of the recognition component 101. For example, the distance D of the recognition position is the distance between the four corners of the plurality of recognition points 101a arranged in a grid.

[0046] In detail, the visual controller 30 detects the frame line 101b included in the recognition component 101 in each of the obtained reference images 51a and reference images 52a. Then, the visual controller 30 recognizes each of the multiple recognition points 101a arranged in a grid shape inside the frame line 101b. Then, the visual controller 30 obtains the distance D of the recognition position of the recognition component 101 on the image of the reference image 51a and the reference image 52a by obtaining the distances between the recognition points 101a at the four corners among the multiple recognition points 101a. The distance D of the recognition position of the recognition component 101 on the image is, for example, a numerical value expressed in the number of pixels. In addition, the distance D of the recognition position of the recognition component 101 on the image is obtained, for example, along the horizontal and vertical directions of the image. Then, the visual controller 30 pre-stores the distances between the recognition points 101a at the four corners in the actual recognition component 101, that is, the distance D of the actual recognition position.

[0047] The vision controller 30 compares the distance D of the actual recognition position of the recognition component 101 with the distance D of the recognition position of the recognition component 101 on the reference image 51a and the reference image 52a, and calculates the limit of the movement amount of the recognition component 101 so that the recognition component 101 is included in the field of view 21a and the field of view 22a regardless of how much the robot arm 11 moves the recognition component 101, so that the recognition component 101 is included in both the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22. Then, the vision controller 30 outputs the calculated movement amount to the robot controller 12. The robot controller 12 obtains the movement amount output from the vision controller 30, and sets a calibration posture based on the obtained movement amount. In this embodiment, 12 calibration postures for obtaining the calibration images 51 and 52 are set based on the obtained reference images 51a and 51b by the robot controller 12 and the vision controller 30.

[0048] like Figure 7 As shown, the robot controller 12 and the vision controller 30 set four postures of the robot arm 11 for calibration based on the acquired reference image 51a and reference image 52a so that the recognition component 101 is moved to four positions P2, position P3, position P4, and position P5 in the prescribed reference plane S1. The prescribed reference plane S1 includes the position P1, which is the position of the recognition component 101 when the reference image 51a and the reference image 52a are captured. Moreover, the reference plane S1 is a plane along the surface where the recognition points 101a of the recognition component 101 are arranged. When the recognition component 101 is moved from position P1 to position P2, position P3, position P4, and position P5, the vision controller 30 outputs a signal indicating the amount of movement for moving the recognition component 101 along the surface where the recognition point 101a of the recognition component 101 is located to the robot controller 12.

[0049] like Figure 8 As shown, the robot controller 12 moves the recognition component 101 to four positions P2, P3, P4, and P5 in the predetermined reference plane S1 based on the signal indicating the movement amount obtained from the vision controller 30, thereby setting four postures of the robot arm 11 for calibration in such a manner that the recognition component 101 is arranged at each of the four corners of the image in the calibration image 51 and the calibration image 52. Figure 8In FIG. 5 , only the correction image 51 is shown, and the correction image 52 is omitted. The position P2, the position P3, the position P4, and the position P5 are calculated by the vision controller 30 so that the recognition component 101 is included in both the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22. That is, the vision controller 30 calculates the value of the limit movement amount of the actual recognition component 101 relative to the moving distance of the imaging unit 21 and the imaging unit 22 so that the recognition component 101 is arranged in the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22 in each of the reference image 51a captured by the imaging unit 21 and the reference image 52a captured by the imaging unit 22. Then, the vision controller 30 obtains the position P2, the position P3, the position P4, and the position P5 based on the smaller limit movement amount of the recognition component 101 in the reference image 51a and the reference image 52a. The robot controller 12 changes the posture of the robot arm 11 so that the recognition component 101 is arranged at the position P2, the position P3, the position P4, and the position P5. Whenever the recognition component 101 is arranged at each of the positions P2, the position P3, the position P4, and the position P5, the robot controller 12 outputs a signal indicating the shooting of the correction image 51 and the correction image 52 to the vision controller 30, and outputs a signal indicating the posture for respectively arranging the recognition component 101 at the position P2, the position P3, the position P4, and the position P5. In addition, when a part or all of the recognition component 101 is not included in the correction image 51 and the correction image 52 obtained by shooting the recognition component 101 at the position P2, the position P3, the position P4, and the position P5, the vision controller 30 corrects the position P2, the position P3, the position P4, and the position P5. In this case, for example, the moving distance from the center, that is, the position P1, is gradually reduced according to the calculation result. That is, the value of the calculated limit movement amount is gradually reduced toward the center of the image. Then, the movement amounts corresponding to the corrected positions P2 , P3 , P4 , and P5 are output to the robot controller 12 again, and the calibration images 51 and 52 are captured again.

[0050] Then, after obtaining the correction images 51 and 52 obtained by photographing the recognition component 101 configured at the positions P2, P3, P4 and P5, the robot controller 12 further sets the eight postures of the robot arm 11 in such a manner that the recognition component 101 is moved to the positions between each of the four positions P2, P3, P4 and P5 in the reference plane S1 and a point on one side of the direction orthogonal to the reference plane S1, i.e., the depth side, and between each of the four positions P2, P3, P4 and P5 in the reference plane S1 and a point on the other side of the direction orthogonal to the reference plane S1, i.e., the near front side.

[0051] For example, Fig. 9 As shown, the robot controller 12 obtains four positions P12, P13, P14, and P15 for obtaining four postures for calibration on one side in the direction perpendicular to the reference plane S1, that is, the depth side. After the robot controller 12 moves the position from the position P1 to the depth side, it obtains the uppermost position of the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22 as the position Pa, and obtains the lowermost position as the position Pb. Then, the robot controller 12 obtains the midpoint between the position Pa and the position P2 as the position P12, the midpoint between the position Pa and the position P3 as the position P13, the midpoint between the position Pb and the position P4 as the position P14, and the midpoint between the position Pb and the position P5 as the position P15.

[0052] Then, if Fig.10 As shown, the robot controller 12 outputs a signal for instructing the imaging of the calibration image 51 and the calibration image 52 to the visual controller 30 each time the recognition component 101 is arranged at each of the positions P12, P13, P14, and P15, and outputs a signal indicating the posture for arranging the recognition component 101 at each of the positions P12, P13, P14, and P15. In addition, the robot controller 12 sets the posture for calibration in such a manner that the relative angle with respect to the imaging unit 21 and the imaging unit 22 is changed from the state of the reference posture at the positions P12, P13, P14, and P15. That is, at the positions P2, P3, P4, and P5, the posture for calibration is set in such a manner that the recognition component 101 moves parallel to the reference plane S1 from the posture at the position P1, but at the positions P12, P13, P14, and P15, the surface where the recognition point 101a of the recognition component 101 is located is arranged to face a direction different from the reference plane S1. For example, if the directions orthogonal to each other in the reference plane S1 are defined as the X direction and the Y direction, and the direction orthogonal to the reference plane S1 is defined as the Z direction, then at positions P12, P13, P14, and P15, the recognition component 101 is configured to be rotated by a predetermined angle from the reference posture in at least one of the X direction, the Y direction, and the Z direction. The angular directions at positions P12, P13, P14, and P15 are set in advance.

[0053] In addition, if Fig.11As shown, similarly to the depth side, the robot controller 12 obtains four positions P22, P23, P24, and P25 for obtaining four postures for calibration on the other side of the direction orthogonal to the reference plane S1, that is, the near side. After the robot controller 12 moves the position from the position P1 to the near side, it obtains the position that is the uppermost end of the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22 as the position Pc, and obtains the position that is the lowermost end as the position Pd. Then, the robot controller 12 obtains the midpoint between the position Pc and the position P2 as the position P22, obtains the midpoint between the position Pc and the position P3 as the position P23, obtains the midpoint between the position Pd and the position P4 as the position P24, and obtains the midpoint between the position Pd and the position P5 as the position P25.

[0054] Then, if Fig.12 As shown, the robot controller 12 outputs a signal instructing the imaging of the correction image 51 and the correction image 52 to the vision controller 30 each time the recognition component 101 is arranged at each of the positions P22, P23, P24, and P25, and outputs a signal indicating the posture for arranging the recognition component 101 at each of the positions P22, P23, P24, and P25. As in the positions P12, P13, P14, and P15, at the positions P22, P23, P24, and P25, the surface where the recognition point 101a of the recognition component 101 is located is arranged to face a direction different from the reference plane S1. At the positions P22, P23, P24, and P25, as in the positions P12, P13, P14, and P15, the recognition component 101 is arranged in a state rotated by a predetermined angle in at least one of the X direction, the Y direction, and the Z direction from the state of the reference posture. In addition, the angle directions at positions P22, P23, P24, and P25 are also preset. Figure 8 , Fig.10 as well as Fig.12 In FIG. 5 , only the example of the corrected image 51 is shown, but the same is true for the corrected image 52. Figure 8 , Fig.10 as well as Fig.12 In the calibration image 51 , the robot arm 11 is omitted from illustration.

[0055] In addition, the distances in the depth direction of the position Pa and the position Pb from the reference plane S1 are preset. In addition, the distances in the depth direction can also be changed based on the input operation performed by the operator. Similarly, the distances in the near-front direction of the position Pc and the position Pd from the reference plane S1 are also preset and can also be changed by the operator. In addition, the angular directions at the positions P12, P13, P14, and P15, and the angular directions at the positions P22, P23, P24, and P25 can also be changed based on the input operation performed by the operator.

[0056] The robot controller 12 sets 12 calibration postures set based on the reference posture and 13 postures of the total of the reference postures as calibration postures. Then, the visual controller 30 uses each of the imaging unit 21 and the imaging unit 22 to capture each of the 13 calibration postures that have been set, thereby obtaining 13 calibration images 51 and 13 calibration images 52. Then, based on the position of the recognition component 101 in each of the 13 calibration images 51 and the 13 calibration images 52 that have been obtained, and the posture of the robot arm 11 at the time of capturing each of the 13 calibration images 51 and the 13 calibration images 52, the visual controller 30 performs a calculation process to establish a correspondence between the position information of the robot arm 11 and the position information of the imaging unit 21 and the imaging unit 22, thereby performing a calibration operation. In detail, the visual controller 30 detects the coordinates of the recognition point 101a of the recognition component 101 in each of the 13 calibration images 51 and the 13 calibration images 52 that have been obtained. Then, the visual controller 30 establishes a correspondence between the coordinates of the identification point 101a in each of the 13 correction images 51 and the 13 correction images 52 and the 13 calibration postures, thereby performing a calculation process for establishing a correspondence between the position information of the robot arm 11, that is, the coordinate system of the robot 10, and the respective coordinate systems of the imaging unit 21 and the imaging unit 22.

[0057] (Control method of robot system)

[0058] Reference Fig.13 , a control method of the robot system 100 will be described. The control method of the robot system 100 is performed by the robot controller 12 and the vision controller 30 executing control processing.

[0059] like Fig.13As shown, first, in step S101, the calibration operation is started to obtain the reference image 51a and the reference image 52a. Specifically, based on the operation performed by the operator, the robot arm 11 is set to a reference posture. Then, in a state where the robot arm 11 is configured to a reference posture, the recognition member 101 for calibration is photographed by using the imaging unit 21 and the imaging unit 22, thereby obtaining a reference image 51a and a reference image 52a.

[0060] Next, in step S102, based on the acquired reference images 51a and 52a, a plurality of calibration postures of the robot arm 11 are set so that the recognition component 101 is captured by the imaging unit 21 and the imaging unit 22 respectively in a state where the recognition component 101 is included in both the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22. Specifically, based on the distance D between the actual recognition positions of the recognition component 101 and the distance D between the recognition positions of the recognition component 101 on the reference images 51a and 52a, 13 calibration postures are set so that the recognition component 101 is arranged at 13 positions, namely, position P1, position P2, position P3, position P4, position P5, position P12, position P13, position P14, position P15, and position P22, position P23, position P24, and position P25.

[0061] Next, in step S103, for each of the plurality of calibration postures that have been set, a plurality of calibration correction images 51 and correction images 52 captured by the imaging unit 21 and the imaging unit 22 are acquired. Specifically, for each of the 13 calibration postures, the recognition component 101 is captured to acquire 13 correction images 51 and 13 correction images 52.

[0062] Next, in step S104, based on the position of the identification component 101 in each of the 13 correction images 51 and the 13 correction images 52 obtained, and the posture of the robot arm 11 at the time of shooting each of the 13 correction images 51 and the 13 correction images 52, a calibration action is performed to establish a correspondence between the position information of the robot arm 11 and the position information of the captured images 41 of the imaging unit 21 and the captured images 42 of the imaging unit 22.

[0063] In this way, in steps S101 to S104, a calibration action is performed to establish a correspondence between the position information of the robot arm 11 installed for the hand H that operates the workpiece W and the position information of the captured images captured by the imaging units 21 and 22 of the workpiece W, that is, the captured images 41 and 42.

[0064] Then, in step S105 , after the calibration operation of steps S101 to S104 is performed, the operation of the robot arm 11 is controlled based on the captured image 41 captured by the imaging unit 21 and the captured image 42 captured by the imaging unit 22 .

[0065] [Effects of Embodiment]

[0066] In this embodiment, the following effects can be obtained.

[0067] In the present embodiment, the robot controller 12 and the vision controller 30 obtain a reference image 51a, which is a captured image 41 obtained by the imaging unit 21 capturing the recognition member 101 for calibration when the robot arm 11 is configured in a reference posture. In addition, the robot controller 12 and the vision controller 30 obtain a reference image 52a, which is a captured image 42 obtained by the imaging unit 22 capturing the recognition member 101 for calibration when the robot arm 11 is configured in a reference posture during the calibration operation. Then, based on the acquired reference images 51a and 52a, the robot controller 12 and the vision controller 30 set a plurality of calibration postures of the robot arm 11 in such a manner that the recognition member 101 is captured by the imaging unit 21 and the imaging unit 22 in a state where the recognition member 101 is included in the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22 during the calibration operation. Then, the robot controller 12 and the vision controller 30 obtain the calibration image 41 obtained by the imaging unit 21 capturing the recognition component 101, that is, the calibration image 51, and the calibration image 42 obtained by the imaging unit 22 capturing the recognition component 101, that is, the calibration image 52, for each of the plurality of calibration postures that have been set. Thus, the plurality of calibration postures of the robot arm 11 are set based on one imaging image 41, that is, the reference image 51a, and one imaging image 42, that is, the reference image 52a. Therefore, compared with the case where the calibration posture is set by roughly converting the plurality of imaging images in advance as in the three-point calibration, the time required for the control process to set the calibration posture can be shortened, and the amount of control processing can be reduced. As a result, in the calibration operation of establishing a correspondence between the position information of the robot arm 11 and the position information of the imaging images 41 and 42, the increase in the time required for the control process and the amount of processing can be suppressed.

[0068] In addition, in the present embodiment, the robot controller 12 and the vision controller 30 set a plurality of calibration postures of the robot arm 11 based on the distance D between the actual recognition position of the recognition component 101 and the distance D between the recognition position of the recognition component 101 on the reference image 51a and the reference image 52a. Thus, by comparing the distance D between the actual recognition position of the recognition component 101 and the distance D between the recognition position of the recognition component 101 on the reference image 51a and the reference image 52a, it is possible to easily obtain a limit value of the relative movement distance of the actual recognition component 101 with respect to the imaging unit 21 and the imaging unit 22 so that the recognition component 101 is arranged within the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22. Therefore, by setting the calibration posture based on the distance D of the actual recognition position of the recognition component 101 and the distance D of the recognition position of the recognition component 101 on the reference image 51a and the reference image 52a, the calibration posture can be easily set in such a manner that the recognition component 101 is included in the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22.

[0069] In addition, in the present embodiment, the recognition member 101 includes a plurality of recognition points 101a, and the robot controller 12 and the vision controller 30 obtain the distance D of the recognition position of the actual recognition member 101 based on the actual distances between the plurality of recognition points 101a, and obtain the distance D of the recognition position of the recognition member 101 on the image based on the distances between the plurality of recognition points 101a on the reference image 51a and the reference image 52a in which the plurality of recognition points 101a are captured, thereby setting a plurality of calibration postures of the robot arm 11. Thus, by obtaining the distances between the plurality of recognition points 101a of the recognition member 101, the distance D of the recognition position of the actual recognition member 101 and the distance D of the recognition position of the recognition member 101 on the reference image 51a and the reference image 52a can be obtained more easily. Therefore, the calibration posture can be set more easily so that the recognition position of the recognition member 101 is included in the field of view 21a of the imaging unit 21 and the field of view 22a of the imaging unit 22.

[0070] In addition, in the present embodiment, the recognition member 101 includes a plate-like member in which a plurality of recognition points 101a are arranged in a grid, and the robot controller 12 and the vision controller 30 obtain the distance D of the recognition position of the actual recognition member 101 based on the actual distances between the recognition points 101a at the four corners among the plurality of recognition points 101a arranged in a grid, and obtain the distance D of the recognition position of the recognition member 101 on the image based on the distances between the recognition points 101a at the four corners on the reference image 51a and the reference image 52a in which the plurality of recognition points 101a arranged in a grid are photographed, thereby setting a plurality of calibration postures of the robot arm 11. Thus, by obtaining the distances between the recognition points 101a at the four corners among the plurality of recognition points 101a arranged in a grid, the distance D of the recognition position of the actual recognition member 101 and the distance D of the recognition position of the recognition member 101 on the reference image 51a and the reference image 52a can be obtained more easily. Therefore, the posture for calibration can be set more easily so that the recognition position of the recognition member 101 is included in the field of view 21 a of the imaging unit 21 and the field of view 22 a of the imaging unit 22 .

[0071] In addition, in the present embodiment, the robot system 100 includes a photographing unit 21 that is fixed at a first position different from the robot arm 11 and photographs the workpiece W, and a photographing unit 22 that is separately fixed from the photographing unit 21 at a second position different from the robot arm 11 and photographs the workpiece W together with the photographing unit 21. The robot controller 12 and the visual controller 30 perform a calibration action to establish a correspondence between the position information of the robot arm 11 and the position information of each of the photographed images 41 and 42 in order to move the robot arm 11 based on the photographed image taken by the photographing unit 21, i.e., the photographed image 41, and the photographed image taken by the photographing unit 22, i.e., the photographed image 42. Then, during the calibration action, the robot controller 12 and the visual controller 30 obtain a captured image 41, i.e., a reference image 51a, obtained by photographing the recognition component 101 arranged on the hand H by the photographing unit 21 when the robot arm 11 is configured in a reference posture, and a captured image 42, i.e., a reference image 52a, obtained by photographing the recognition component 101 arranged on the hand H by the photographing unit 22 when the robot arm 11 is configured in a reference posture, and based on the obtained reference image 51a and reference image 52a, a plurality of calibration postures of the robot arm 11 are set in a manner in which the recognition component 101 is photographed by each of the photographing units 21 and 22 in a state where the recognition component 101 is included in the fields of view 21a and 22a of both the photographing units 21 and 22 during the calibration action, and a correction image 51 and a correction image 52 captured by each of the photographing units 21 and 22 are obtained for each of the plurality of calibration postures that have been set. Thus, it is possible to set a plurality of calibration postures of the robot arm 11 in such a manner that the recognition part 101 is photographed by each of the photographing units 21 and 22 in a state where the recognition part 101 is included in the fields of view 21a and 22a of both the photographing units 21 and 22. Therefore, even when the movement of the robot arm 11 is three-dimensionally controlled by using two different photographing units 21 and 22, in the calibration action of establishing a correspondence between the position information of the robot arm 11 and the position information of each of the photographed images 41 and 42, it is possible to suppress an increase in the time and processing amount required for the control processing.

[0072] In addition, in the present embodiment, the robot controller 12 and the vision controller 30 obtain reference images 51a and 52a captured in a manner that the recognition component 101 is arranged at approximately the center of the image, and based on the obtained reference images 51a and 52a, the robot controller 12 and the vision controller 30 move the recognition component 101 to four positions P2, P3, P4, and P5 within a predetermined reference plane S1 including the position P1 of the recognition component 101 when the reference images 51a and 52a were captured, thereby setting the robot controller 12 and the vision controller 30 in a manner that the recognition component 101 is arranged at each of the four corners of the image in the correction images 51 and 52. The four postures of the robot arm 11 are set in such a manner that the recognition member 101 moves to positions P12, P13, P14, and P15 between each of the four positions P2, P3, P4, and P5 in the reference plane S1 and a point on one side in a direction perpendicular to the reference plane S1, and each of positions P22, P23, P24, and P25 between each of the four positions P2, P3, P4, and P5 in the reference plane S1 and a point on the other side in a direction perpendicular to the reference plane S1. Thus, 13 calibration postures can be set in such a manner that the recognition member 101 moves three-dimensionally relative to the imaging units 21 and 22. Therefore, by moving the recognition member 101 three-dimensionally, the position information of the robot arm 11 and the position information of the captured images 41 and 42 can be associated with high accuracy in the calibration operation.

[0073] [Modifications]

[0074] The embodiments disclosed this time should be considered in all respects as illustrative and not restrictive. The scope of the present disclosure is indicated by the claims rather than the description of the embodiments described above, and includes all changes (modifications) within the meaning and scope equivalent to the claims.

[0075] For example, in the above embodiment, the following example is shown, that is, the recognition component 101 arranged on the robot arm 11 is photographed by the photographing unit 21 and the photographing unit 22 fixedly arranged at positions different from the robot arm 11, that is, the first position and the second position, so as to perform the calibration operation, but the present disclosure is not limited to this. In the present disclosure, the recognition component arranged at the position where the object for operation, that is, the workpiece is arranged, can also be photographed by the photographing unit arranged on the robot arm to perform the calibration operation. That is, the workpiece can also be operated while photographing it by the photographing unit arranged on the robot arm. In this case, as in the above embodiment, a reference image is obtained, and a plurality of calibration postures are set based on the distance of the recognition position of the recognition component on the image of the reference image and the distance of the recognition position of the actual recognition component.

[0076] In addition, the following example is shown in the above embodiment, that is, by using the recognition component 101 in which a plurality of recognition points 101a are arranged in a grid shape, the distances between the recognition points 101a arranged at the four corners among the plurality of recognition points 101a are obtained, thereby obtaining the distance D of the recognition position of the recognition component 101, but the present disclosure is not limited to this. In the present disclosure, the distance of the recognition position of the recognition component may not be the distance between the recognition points. For example, a cross-shaped symbol or a rectangular symbol of a recognizable size may be arranged on the recognition component. In addition, only four recognition points may be set instead of setting the recognition points in a grid shape.

[0077] In addition, in the above embodiment, the following example is shown, that is, the action of the robot arm 11 is controlled based on the captured images 41 and 42 captured by the two imaging units 21 and 22, but the present disclosure is not limited to this. In the present disclosure, the action of the robot arm can also be controlled by using one imaging unit.

[0078] In addition, an example of setting 13 calibration postures is shown in the above embodiment, but the present disclosure is not limited to this. In the present disclosure, more than 13 calibration postures may be set. In addition, less than 13 calibration postures may be set. In addition, the calibration posture may be a posture in which the recognition component is configured at a position different from the example of the above embodiment. In addition, the angle of the recognition component in the 13 calibration postures may also be an angle different from the example of the above embodiment.

[0079] In addition, in the above embodiment, an example of the robot arm 11 performing the operation of gripping the workpiece W is shown, but the present disclosure is not limited to this. In the present disclosure, the robot arm may also perform the operation of manufacturing and assembling the workpiece.

[0080] In addition, in the above-mentioned embodiment, an example is shown in which the calibration action is performed by the control processing of two hardwares, the robot controller 12 and the visual controller 30, but the present disclosure is not limited to this. In the present disclosure, the control of the calibration action can also be performed by a control device as a control unit. In addition, the calculation processing of establishing the correspondence between the position information of the robot and the position information of the captured image can also be performed by the robot controller that controls the action of the robot arm, rather than by the visual controller that obtains the captured image. In addition, the control of detecting and identifying components from the calibration image can also be performed by the robot controller.

[0081] The functions of the elements disclosed in this specification can be performed using circuits or processing circuits including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), existing circuits, and / or combinations thereof that are configured or programmed to perform the disclosed functions. The processor includes transistors and other circuits, so it can be regarded as a processing circuit or circuit. In the present disclosure, a circuit, unit, or mechanism is hardware that performs the listed functions, or hardware that is programmed to perform the listed functions. The hardware may be the hardware disclosed in this specification, or it may be other known hardware that is programmed or configured to perform the listed functions. In the case where the hardware is a processor that is considered to be a type of circuit, the circuit, mechanism, or unit is a combination of hardware and software, and the software is used to constitute the hardware and / or processor.

[0082] [form]

[0083] It will be understood by those skilled in the art that the above-mentioned exemplary embodiments are specific examples of the following aspects.

[0084] (Item 1)

[0085] A robot system, comprising:

[0086] A robotic arm, for mounting hands that work on the workpiece;

[0087] A photographing unit, for photographing the workpiece; and

[0088] a control unit that performs a calibration operation for establishing a correspondence between position information of the robot arm and position information of the captured image in order to cause the robot arm to move based on the captured image captured by the capturing unit;

[0089] The control unit is composed of:

[0090] obtaining a reference image, which is a captured image obtained by capturing a calibration recognition component by the capturing unit in a state where the robot arm is arranged in one posture, and

[0091] Based on the acquired reference image, a plurality of calibration postures of the robot arm are set in such a manner that the recognition member is photographed by the imaging unit in a state where the recognition member is included in the field of view of the imaging unit during the calibration operation.

[0092] The captured image for calibration, that is, the correction image, is obtained by capturing the recognition member by the capturing unit in each of the plurality of calibration postures that have been set.

[0093] (Item 2)

[0094] A robot system according to item 1, wherein:

[0095] The control unit sets the plurality of calibration postures of the robot arm based on an actual distance between the recognition positions of the recognition member and a distance between the recognition positions of the recognition member on the reference image.

[0096] (Item 3)

[0097] A robot system according to item 2, wherein:

[0098] The identification component includes a plurality of identification points.

[0099] The control unit obtains the actual distance between the identification positions of the identification component based on the actual distance between the multiple identification points, and obtains the distance between the identification positions of the identification component on the image based on the distance between the multiple identification points on the image of the reference image that captured the multiple identification points, thereby setting the multiple calibration postures of the robot arm.

[0100] (Item 4)

[0101] A robot system according to item 3, wherein:

[0102] The identification member includes a plate-shaped member on which the plurality of identification points are arranged in a grid pattern.

[0103] The control unit obtains the actual distance of the recognition position of the recognition component based on the actual distances between the recognition points at the four corners among the multiple recognition points arranged in a grid, and obtains the distance of the recognition position of the recognition component on the image based on the distances between the recognition points at the four corners on the image of the reference image of the multiple recognition points arranged in a grid, thereby setting the multiple calibration postures of the robot arm.

[0104] (Item 5)

[0105] The robot system according to any one of items 1 to 4, wherein:

[0106] The photographing unit includes: a first photographing unit fixed at a first position different from the robot arm, photographing the workpiece; and a second photographing unit, separately from the first photographing unit, fixed at a second position different from the robot arm, photographing the workpiece together with the first photographing unit.

[0107] The control unit is composed of:

[0108] In order to move the robot arm based on the captured image captured by the first imaging unit, i.e., the first captured image, and the captured image captured by the second imaging unit, i.e., the second captured image, the calibration operation is performed to establish a correspondence between the position information of the robot arm and the position information of each of the first captured image and the second captured image.

[0109] In the calibration operation, a first captured image, i.e., a first reference image, is obtained by capturing the recognition component disposed on the hand by the first capturing unit when the robot arm is configured in the one posture, and a second captured image, i.e., a second reference image, is obtained by capturing the recognition component disposed on the hand by the second capturing unit when the robot arm is configured in the one posture, and

[0110] Based on the first reference image and the second reference image that have been obtained, the plurality of calibration postures of the robot arm are set in such a manner that the recognition component is photographed by each of the first imaging unit and the second imaging unit in a state where the recognition component is included in the fields of view of both the first imaging unit and the second imaging unit during the calibration operation.

[0111] The calibration image captured by each of the first imaging unit and the second imaging unit is acquired in each of the plurality of calibration postures that have been set.

[0112] (Item 6)

[0113] The robot system according to any one of items 1 to 5, wherein:

[0114] The control unit is composed of:

[0115] The reference image is captured with the recognition component disposed approximately at the center of the image, and

[0116] Based on the acquired reference image, the recognition component is moved to four positions within a predetermined reference plane including the position of the recognition component when the reference image was captured, and four postures of the robot arm are set in such a manner that the recognition component is arranged at each of the four corners of the image in the correction image.

[0117] The eight postures of the robot arm are set in such a manner that the identification component moves to a position between each of the four positions in the reference plane and a point on one side of a direction orthogonal to the reference plane, and to a position between each of the four positions in the reference plane and a point on the other side of a direction orthogonal to the reference plane.

[0118] (Item 7)

[0119] A control method for a robot system, comprising the following steps:

[0120] performing a calibration operation in which position information of a robot arm mounted on a hand for working on a workpiece and position information of an image captured by an imaging unit for capturing the workpiece are associated with each other; and

[0121] After the calibration operation is performed, the operation of the robot arm is controlled based on the image captured by the imaging unit.

[0122] The steps of performing the above calibration action include the following steps:

[0123] acquiring a reference image, which is a captured image obtained by capturing a calibration recognition component by the capturing unit in a state where the robot arm is arranged in one posture;

[0124] Based on the acquired reference image, a plurality of calibration postures of the robot arm are set in such a manner that the recognition member is captured by the imaging unit in a state where the recognition member is included in a field of view of the imaging unit; and

[0125] The captured image for calibration, that is, the correction image, is obtained by capturing the recognition member by the capturing unit in each of the plurality of calibration postures that have been set.

Claims

1. A robot system, characterized in that: have: A robotic arm, for mounting hands that work on the workpiece; a photographing unit that photographs the workpiece; and a control unit that performs a calibration operation for establishing a correspondence between position information of the robot arm and position information of the captured image in order to cause the robot arm to move based on the captured image captured by the capturing unit; The control unit is composed of: acquiring a reference image, which is a captured image obtained by capturing a recognition component for calibration by the capturing unit in a state where the robot arm is arranged in one posture, and Based on the acquired reference image, a plurality of calibration postures of the robot arm are set in such a manner that the recognition member is photographed by the imaging unit in a state where the recognition member is included in the field of view of the imaging unit during the calibration operation. The captured image for calibration, that is, the correction image, is acquired by capturing the recognition member by the capturing unit for each of the plurality of calibration postures that have been set.

2. The robot system according to claim 1, characterized in that: The control unit sets the plurality of calibration postures of the robot arm based on an actual distance between recognition positions of the recognition member and a distance between the recognition positions of the recognition member on the reference image.

3. The robot system according to claim 2, characterized in that: The identification component includes a plurality of identification points, The control unit obtains the actual distance of the recognition position of the recognition component based on the actual distance between the multiple recognition points, and obtains the distance of the recognition position of the recognition component on the image based on the distance between the multiple recognition points on the image of the reference image that captured the multiple recognition points, thereby setting the multiple calibration postures of the robot arm.

4. The robot system according to claim 3, characterized in that: The identification member includes a plate-shaped member on which the plurality of identification points are arranged in a grid pattern. The control unit obtains the actual distance of the recognition position of the recognition component based on the actual distances between the recognition points at the four corners among the multiple recognition points arranged in a grid, and obtains the distance of the recognition position of the recognition component on the image based on the distances between the recognition points at the four corners on the image of the reference image of the multiple recognition points arranged in a grid, thereby setting the multiple calibration postures of the robot arm.

5. The robot system according to claim 1, characterized in that: The photographing unit includes: a first photographing unit, fixed at a first position different from the robot arm, photographing the workpiece; and a second photographing unit, separately from the first photographing unit, fixed at a second position different from the robot arm, photographing the workpiece together with the first photographing unit. The control unit is composed of: In order to move the robot arm based on the captured image taken by the first imaging unit, i.e., the first captured image, and the captured image taken by the second imaging unit, i.e., the second captured image, the calibration operation is performed to establish a correspondence between the position information of the robot arm and the position information of each of the first captured image and the second captured image. In the calibration operation, a first captured image, i.e., a first reference image, is obtained by capturing the recognition component arranged on the hand by the first capturing unit when the robot arm is arranged in the one posture, and a second captured image, i.e., a second reference image, is obtained by capturing the recognition component arranged on the hand by the second capturing unit when the robot arm is arranged in the one posture, and The plurality of calibration postures of the robot arm are set based on the first reference image and the second reference image that have been obtained, in such a manner that the recognition component is photographed by each of the first photographing unit and the second photographing unit in a state where the recognition component is included in the fields of view of both the first photographing unit and the second photographing unit during the calibration operation, The calibrated image captured by each of the first imaging unit and the second imaging unit is acquired in each of the plurality of calibration postures that have been set.

6. The robot system according to claim 1, characterized in that: The control unit is composed of: acquiring the reference image captured in such a manner that the recognition component is arranged substantially at the center of the image, and Based on the acquired reference image, the recognition component is moved to four positions within a predetermined reference plane including the position of the recognition component when the reference image was captured, thereby setting four postures of the robot arm in a manner such that the recognition component is arranged at each of the four corners of the image in the correction image. The eight postures of the robot arm are set in such a manner that the identification component moves to a position between each of the four positions in the reference plane and a point on one side of a direction orthogonal to the reference plane, and to a position between each of the four positions in the reference plane and a point on the other side of a direction orthogonal to the reference plane.

7. A control method for a robot system, characterized in that: The following steps are required: performing a calibration operation in which position information of a robot arm mounted on a hand for working on a workpiece and position information of an image captured by an imaging unit for capturing the workpiece are associated with each other; and After performing the calibration operation, the operation of the robot arm is controlled based on the image captured by the imaging unit. The steps of performing the calibration operation include the following steps: acquiring a reference image, which is a captured image obtained by capturing a recognition component for calibration by the capturing unit in a state where the robot arm is arranged in one posture; Based on the acquired reference image, a plurality of calibration postures of the robot arm are set in such a manner that the recognition member is captured by the imaging unit in a state where the recognition member is included in the field of view of the imaging unit; and The captured image for calibration, that is, the correction image, is acquired by capturing the recognition member by the capturing unit for each of the plurality of calibration postures that have been set.

Images