An imaging device that calculates three-dimensional position based on images captured by a visual sensor

Abstract

This imaging device comprises a focus position detection unit for detecting a focus position when a visual sensor is in focus. The imaging device comprises a parameter setting unit for setting a parameter for calculating a three-dimensional position corresponding to a specific position in an image captured by the visual sensor. The imaging device comprises a characteristic position calculation unit for calculating the three-dimensional position of a characteristic portion using the parameter set by the parameter setting unit. The parameter setting unit sets the parameter on the basis of the focus position.

Description

[Technical Field]

[0001] The present invention relates to an imaging device that calculates a three-dimensional position based on an image captured by a visual sensor. [Background technology]

[0002] Conventionally, devices have been known that detect the three-dimensional position of an object by processing images obtained by capturing an image of the object using a visual sensor. For example, a device is known that captures two-dimensional images of an object from two directions and calculates the three-dimensional position of a specific part (for example, JP 2016-706475 A). Alternatively, a visual sensor called a stereo camera is known that simultaneously captures images using two two-dimensional cameras and calculates the three-dimensional position of a feature point based on the parallax between the two images.

[0003] Such a device for calculating three-dimensional position can be attached to a robot for moving a work tool to perform a predetermined task. In a robot system for transporting a workpiece, a camera captures an image of the workpiece placed in a predetermined position. The three-dimensional position of the workpiece is detected based on the image captured by the camera. The position and posture of the robot are changed so that the workpiece can be grasped according to the position of the workpiece. This type of control makes it possible to detect the exact position of the workpiece and perform the task reliably.

[0004] When calculating the three-dimensional position of an object based on a two-dimensional image captured by a visual sensor, a calculation model is used to convert the position in the image into a three-dimensional position. The calculation model includes predetermined parameters such as coefficients and constants. By using the calculation model, the three-dimensional position can be calculated from the position in the image captured by the camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-70674 Summary of the Invention [Problem to be solved by the invention]

[0006] The parameters in the computational model for calculating the three-dimensional position of an object depend on the camera installation conditions, lens characteristics, individual lens differences, etc. The parameters can be determined in advance through calculations or experiments, etc. For example, after placing the camera in a predetermined position, an image of the object can actually be captured and the parameters can be calculated in advance.

[0007] In conventional camera body and lens combinations, the camera is fixed in a predetermined position. The position of the camera lens is fixed, and parameters are calculated in advance. However, there are cases where the object being imaged by the camera has individual differences. Or, the position of the object when imaged by the camera may deviate from the desired position. As a result, the image may be blurred when the workpiece is imaged.

[0008] To address this issue, a camera focusing control method is being considered. For example, with a camera equipped with an autofocus function, it is conceivable to adjust the focus depending on the position where the workpiece is placed or on individual differences in the workpiece. However, when focusing is performed, the position of the lens at which the focus is achieved changes, and the parameters in the calculation model also change. Because the parameters are set to correspond to one focus position, there is a problem in that it is not possible to focus on positions other than the predetermined focus position. Alternatively, it has been necessary to use the parameters without considering the focus position.

[0009] Furthermore, in order to maintain a constant focus position, it is possible to align the relative position of the camera with respect to the workpiece to a predetermined position. However, the position of at least one of the workpiece and the camera may change. For example, the position and orientation of a workpiece placed on a workbench may change depending on the situation in which the workpiece is being transported. In this case, the robot can change its position and orientation according to the position and orientation of the workpiece. However, the robot may interfere with obstacles such as fences placed around the robot system. Or, the robot's stroke may be limited. For this reason, it may be difficult to align the relative position of the camera with respect to the workpiece to a predetermined position. [Means for solving the problem]

[0010] An imaging device according to one aspect of the present disclosure includes a visual sensor that captures an image of an object and a focus position detection unit that detects the focus position when the visual sensor is in focus. The imaging device also includes a parameter setting unit that sets parameters for calculating a three-dimensional position corresponding to a specific position in an image captured by the visual sensor. The imaging device also includes a storage unit that stores setting information for setting the parameters corresponding to the focus position. The imaging device also includes a feature detection unit that detects a predetermined feature portion in the image of the object, and a feature position calculation unit that calculates the three-dimensional position of the feature portion using the parameters set by the parameter setting unit. The visual sensor is configured to capture a two-dimensional image. The feature position calculation unit uses a calculation model to calculate the three-dimensional position of a feature from a specific position of the feature in the two-dimensional image. In the calculation model, the distance from the visual sensor to the workpiece is predetermined. The parameter setting unit determines, based on the focus position and setting information, At least one in the computational model Set the parameters. [Effects of the Invention]

[0011] According to an aspect of the present disclosure, it is possible to provide an imaging device that accurately detects the three-dimensional position of a characteristic part when the focus position changes. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a first robot system according to an embodiment. [Figure 2] FIG. 1 is a block diagram of a first robot system according to an embodiment. [Figure 3] FIG. 2 is a plan view of a workpiece according to the embodiment. [Figure 4] 1 is a schematic diagram illustrating a camera focus position and a camera field of view. FIG. [Figure 5] This is an example of an image when the focus position is changed. [Figure 6] 1 is a schematic diagram of the image sensor, lens, and features of a workpiece when the lens position is moved to adjust focus. FIG. [Figure 7] 10 is a flowchart illustrating control of the first robot system. [Figure 8] FIG. 10 is a first schematic diagram illustrating another control of the first robot system. [Figure 9] FIG. 10 is a second schematic diagram illustrating another control of the first robot system. [Figure 10] FIG. 2 is a schematic diagram of a second robot system according to an embodiment. [Figure 11] FIG. 2 is a schematic diagram of a transport device according to an embodiment. [Figure 12] FIG. 2 is a block diagram of a transport device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] An imaging device according to an embodiment will be described with reference to Figures 1 to 12. The imaging device according to the present embodiment functions as a three-dimensional position acquisition device that calculates the three-dimensional position of a specific position in an image based on an image captured by a visual sensor.

[0014] Fig. 1 is a schematic diagram of a first robot system equipped with an imaging device according to the present embodiment. Fig. 2 is a block diagram of the first robot system according to the present embodiment. The robot system according to the present embodiment detects the position of a workpiece as an object and transports the workpiece.

[0015] 1 and 2, the first robot system 3 includes a hand 5 as a work tool for gripping a workpiece 38, and a robot 1 for moving the hand 5. The robot system 3 includes a control device 2 for controlling the robot system 3. The robot system 3 also includes a stand 95 on which the workpiece 38 is placed.

[0016] The hand 5 in this embodiment is a work tool that grips and releases the workpiece 38. The work tool attached to the robot 1 is not limited to this form, and any work tool can be used depending on the work to be performed by the robot system 3. For example, a work tool for performing welding or the like can be used as an end effector.

[0017] The robot 1 of this embodiment is an articulated robot including multiple joints 18. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a swivel base 13. The swivel base 13 is supported by a base 14. The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 to which a hand 5 is fixed. The components of the robot 1 are formed to rotate around a predetermined drive axis. The robot 1 is not limited to this configuration, and any robot that can move a work tool can be used.

[0018] The robot 1 of this embodiment includes a robot driving device 21 having a driving motor that drives components such as the upper arm 11. The hand 5 includes a hand driving device 22 that drives the hand 5. The hand driving device 22 of this embodiment drives the hand 5 by air pressure. The hand driving device 22 includes an air pump and a solenoid valve for supplying compressed air to a cylinder.

[0019] The control device 2 includes a control device main body 40 and a teaching operation panel 26 for an operator to operate the control device main body 40. The control device main body 40 includes an arithmetic processing device (computer) having a CPU (Central Processing Unit) as a processor. The arithmetic processing device has RAM (Random Access Memory) and ROM (Read Only Memory), etc., connected to the CPU via a bus. The robot 1 is driven based on operation commands from the control device 2. The robot 1 automatically transports a workpiece 38 based on an operation program 61. The robot driving device 21 and hand driving device 22 are controlled by the control device 2.

[0020] The control device main body 40 includes a storage unit 42 that stores any information related to the robot system 3. The storage unit 42 can be configured with a non-transitory storage medium capable of storing information. For example, the storage unit 42 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium.

[0021] An operation program 61 created in advance for operating the robot 1 is input to the control device 2. Alternatively, an operator can set teaching points for the robot 1 by operating the teaching operation panel 26 to drive the robot 1. The control device 2 can generate the operation program 61 based on the teaching points.

[0022] The operation program 61 is stored in the memory unit 42. The operation control unit 43 sends operation commands to the robot driving unit 44 for driving the robot 1 based on the operation program 61. The robot driving unit 44 includes an electrical circuit that drives the drive motor, and supplies electricity to the robot driving device 21 based on the operation commands. The operation control unit 43 also sends operation commands to the hand driving unit 45 for driving the hand driving device 22. The hand driving unit 45 includes an electrical circuit that drives an air pump or the like, and supplies electricity to the air pump or the like based on the operation commands.

[0023] The operation control unit 43 corresponds to a processor that operates in accordance with the operation program 61. The processor is configured to be able to read information stored in the storage unit 42. The processor reads the operation program 61 and performs the control defined in the operation program 61, thereby functioning as the operation control unit 43.

[0024] The robot 1 includes a state detector for detecting the position and posture of the robot 1. The state detector in this embodiment includes a position detector 23 attached to the drive motor of each drive shaft of the robot drive device 21. The position detector 23 can be configured, for example, by an encoder that detects the rotational position of the output shaft of the drive motor. The position and posture of the robot 1 are detected by the output of the position detector 23.

[0025] In the robot system 3, a reference coordinate system 71 is set which remains stationary even when the position and posture of the robot 1 change. In the example shown in FIG. 1, the origin of the reference coordinate system 71 is located on the base 14 of the robot 1. The reference coordinate system 71 is also referred to as a world coordinate system. In the reference coordinate system 71, the position of the origin is fixed, and furthermore, the orientation of the coordinate axes is fixed. The reference coordinate system 71 has X-axis, Y-axis, and Z-axis which are orthogonal to each other as the coordinate axes. Furthermore, the W-axis is set as the coordinate axis around the X-axis. The P-axis is set as the coordinate axis around the Y-axis. The R-axis is set as the coordinate axis around the Z-axis.

[0026] The teaching operation panel 26 is connected to the control device main body 40 via a communication device. The teaching operation panel 26 includes an input unit 27 for inputting information about the robot 1 and the hand 5. The input unit 27 is composed of input members such as a keyboard and a dial. The teaching operation panel 26 includes a display unit 28 for displaying information about the robot 1 and the hand 5. The display unit 28 can be composed of any display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel. Note that when the teaching operation panel is equipped with a touch panel type display panel, the display panel functions as both the input unit and the display unit.

[0027] A tool coordinate system is set in the robot system 3, with its origin set at an arbitrary position on the work tool. The position and orientation of the tool coordinate system change along with the work tool. In this embodiment, the origin of the tool coordinate system is set at the tool tip point of the hand 5. The position of the robot 1 corresponds to the position of the tool tip point (the position of the origin of the tool coordinate system). Furthermore, the orientation of the robot 1 corresponds to the orientation of the tool coordinate system relative to the reference coordinate system 71.

[0028] The robot system 3 in this embodiment is equipped with an imaging device that detects the position of the workpiece 38. The imaging device detects the position of the workpiece 38 on the base 95 before the hand 5 grasps the workpiece 38. The imaging device is equipped with a camera 6 as a visual sensor that captures an image of the workpiece 38. The camera 6 in this embodiment is a two-dimensional camera that captures a two-dimensional image. The camera 6 is supported by the robot 1. The camera 6 is fixed to the hand 5 via a support member.

[0029] The camera 6 can capture an image in the field of view 6a. The camera 6 has a focus adjustment mechanism 24 for adjusting the focus. The focus adjustment mechanism 24 in this embodiment has a function of automatically adjusting the focus. In other words, the camera 6 has an autofocus function. The camera 6 is configured to automatically focus on the workpiece 38 and capture an image of the workpiece 38 when the position and posture of the robot 1 are changed. As the focus adjustment mechanism, a mechanism that adjusts the focus by any control such as a contrast detection method or a phase difference method can be used.

[0030] Alternatively, a camera equipped with a liquid lens can be used as the visual sensor. In this case, a mechanism for changing the shape of the liquid lens can be used as the focus adjustment mechanism. For example, a mechanism for changing the voltage applied to the liquid lens or a mechanism for moving a holding member of the liquid lens to change the water pressure applied to the liquid lens can be used.

[0031] In the robot system 3, a camera coordinate system 72 is set as a sensor coordinate system for the camera 6. The position and orientation of the camera coordinate system 72 change together with the camera 6. The origin of the camera coordinate system 72 is set at a predetermined position of the camera 6, such as the lens center or optical center of the camera 6. The camera coordinate system 72 has an X-axis, a Y-axis, and a Z-axis that are orthogonal to one another. In the present embodiment, the camera coordinate system 72 is set so that the Z-axis is parallel to the optical axis of the lens of the camera 6.

[0032] The imaging device of this embodiment includes a moving device that moves one of the workpiece 38 as an object and the camera 6 to change the position of one relative to the other. In the first robot system 3, the robot 1 functions as the moving device. When the position and posture of the robot 1 change, the position and posture of the camera 6 also change.

[0033] The imaging device includes an image processing device that processes images captured by the visual sensor. In the robot system 3 of this embodiment, the control device main body 40 functions as the image processing device. The control device main body 40 includes an image processing unit 51 that processes images captured by the camera 6. The image processing unit 51 includes an imaging control unit 58 that sends a command to the camera 6 to capture an image.

[0034] The image processing unit 51 includes a focus position detection unit 52 that detects the focus position when the camera 6 is focused. The image processing unit 51 includes a parameter setting unit 53 that sets parameters for calculating a three-dimensional position corresponding to a specific position in the image captured by the camera 6. The image processing unit 51 includes a feature detection unit 54 that detects a predetermined feature portion in the image of the workpiece 38. The image processing unit 51 includes a feature position calculation unit 55 that calculates the three-dimensional position of the feature portion using the parameters set in the parameter setting unit 53. The image processing unit 51 includes a distance calculation unit 56 that calculates the distance from the camera 6 to the workpiece 38. The image processing unit 51 includes an operation command generation unit 59 that generates operation commands for the robot 1 and the hand 5 based on the results of image processing.

[0035] The image processing unit 51 corresponds to a processor that operates in accordance with the operation program 61. In particular, each of the units, namely, the focus position detection unit 52, the parameter setting unit 53, the feature detection unit 54, the feature position calculation unit 55, the distance calculation unit 56, the imaging control unit 58, and the operation command generation unit 59, corresponds to a processor that operates in accordance with the operation program 61. The processor reads the operation program 61 and performs the control defined in the operation program 61, thereby functioning as each unit.

[0036] 1, in the first robot system 3 of this embodiment, the workpiece 38 is placed on the surface of the pedestal 95 by a predetermined method. For example, a worker or another robot system places the workpiece 38 on the surface of the pedestal 95. Then, the robot 1 changes its position and posture to grasp the workpiece 38 placed on the top surface of the pedestal 95 with the hand 5. The robot system 3 conveys the workpiece 38 to a predetermined position by changing the position and posture of the robot 1.

[0037] When the workpiece 38 is placed on the surface of the base 95, the position of the workpiece 38 on the base 95 may be shifted. In the example shown in Fig. 1, a position P38a is shown that is determined in a teaching operation when teaching the position and posture of the robot 1. The position P38a is a position where the workpiece 38 is preferably placed, and is a reference position for placing the workpiece 38.

[0038] However, when the workpiece 38 is actually placed on the upper surface of the stand 95, the workpiece 38 may be placed at position P38b, which is shifted from the reference position P38a. Alternatively, there may be a dimensional error in the workpiece 38. In the robot system 3, the camera 6 captures an image of the workpiece 38. The image processing unit 51 then calculates the three-dimensional position of the workpiece 38 based on the image of the workpiece 38. For example, the image processing unit 51 detects the three-dimensional positions of characteristic parts of the workpiece 38. The image processing unit 51 then calculates the position of the workpiece 38 based on the three-dimensional positions of the characteristic parts of the workpiece 38. The position of the workpiece 38 can be calculated using the reference coordinate system 71. The image processing unit 51 controls the position and posture of the robot 1 so that it corresponds to the position of the workpiece 38. The hand 5 then grasps the workpiece 38 and transports it to a desired, predetermined position.

[0039] FIG. 3 shows a plan view of workpiece 38 in this embodiment. With reference to FIGS. 1 and 3, workpiece 38 has plate-shaped portion 38a and plate-shaped portion 38b formed on the upper side of plate-shaped portion 38a. Each of plate-shaped portions 38a, 38b has a rectangular parallelepiped shape. Plate-shaped portion 38b has edge portion 38c on the outer periphery of its upper surface. Edge portion 38c is a portion corresponding to a corner formed on plate-shaped portion 38b. In this embodiment, edge portion 38c, which has a quadrangular shape when viewed from above, is a characteristic portion of workpiece 38.

[0040] 1, in this embodiment, camera 6 is placed vertically above workpiece 38. At this time, the distance from the surface on which the characteristic portions of workpiece 38 are formed to camera 6 is determined in advance. In this example, the position and attitude of robot 1 are controlled so that the position of the upper surface of plate-shaped portion 38b is at a predetermined Z-axis value in camera coordinate system 72. In addition, the attitude of camera 6 is adjusted so that the optical axis of camera 6 is approximately perpendicular to the surface of plate-shaped portion 38b on which the characteristic portions of workpiece 38 are formed.

[0041] The camera 6 adjusts its focus and captures an image of the workpiece 38. The feature detection unit 54 of the image processing unit 51 detects the edge 38c as a characteristic part of the workpiece 38 by performing pattern matching. A reference image for detecting the position of the edge 38c is created in advance and stored in the storage unit 42. The feature detection unit 54 uses the reference image to detect the edge 38c, which is a characteristic part, in the image captured by the camera 6.

[0042] The feature position calculation unit 55 calculates the position of the workpiece in three-dimensional space based on the positions of the feature parts in the image captured by the camera. The position of the workpiece can be calculated as the position of any set point set on the workpiece. The position of the workpiece 38 can be acquired in the reference coordinate system 71.

[0043] The operation command generation unit 59 calculates the position and posture of the robot 1 based on the position of the workpiece 38 calculated by the characteristic position calculation unit 55. Then, the operation command generation unit 59 sends the position and posture of the robot 1 for gripping the workpiece 38 to the operation control unit 43. Based on the operation command received from the operation command generation unit 59, the operation control unit 43 drives the robot 1 and the hand 5 to grip the workpiece 38.

[0044] In the control of gripping a workpiece in this embodiment, the feature detection unit 54 detects feature portions, and the feature position calculation unit 55 accurately calculates the three-dimensional position of the workpiece based on the positions of the feature portions. This allows the robot system 3 to grip the workpiece 38 more reliably. Even if the position of the workpiece 38 on the base 95 (the position of the workpiece 38 in the reference coordinate system 71) differs from the reference position or if there is a dimensional error in the workpiece, the robot system 3 can grip the workpiece 38 reliably.

[0045] Referring to FIG. 1, if the position of the workpiece 38 shifts or due to individual differences in the workpiece, the focal position at which the camera is focused may shift. Here, the focal position can be any variable corresponding to the position of the lens driven by the focus adjustment mechanism 24. For example, the focus adjustment mechanism 24 may include a drive motor for driving the lens to adjust the focus. In this case, the rotational position of the drive motor's output shaft when the focus is adjusted can be used as the focal position. Alternatively, the lens may be provided with a focus ring for adjusting the focus. The position of the focus ring can be determined as the focal position. Alternatively, if the camera has multiple lenses, the position of a specific lens can be determined as the focal position. Furthermore, in the case of a camera with a liquid lens, the magnitude of the voltage applied to the liquid lens can be used as the focal position. Alternatively, the rotational position of the output shaft of a motor included in a drive mechanism of a holding member that changes the pressure applied to the liquid lens can be used as the focal position.

[0046] Figure 4 shows a schematic diagram explaining the field of view when the focus position is changed. The field of view of the camera 6 corresponds to the angle of view or imaging range. Figure 4 shows focus positions A and B. When focusing at position A, the imaging range of the camera 6 becomes field of view A. When focusing at position B, the imaging range of the camera 6 becomes field of view B. In this way, the size of the field of view changes when the focus position changes. If the position of the feature part of the workpiece remains the same, and the focus position changes, the position of the feature part in the image will change.

[0047] Figure 5 shows examples of images captured by a camera when the focus position is changed. Image 66 is an image captured when the focus is set to one position, which corresponds to position A in Figure 4, for example. Image 67 is an image captured when the focus is set to another position, which corresponds to position B in Figure 4, for example. An image coordinate system 73 is set for images 66 and 67.

[0048] Images 66 and 67 each contain images 68a and 68b of holes as feature parts. By changing the focus position, as indicated by arrow 101, the position of image 68a of the hole in image 66 becomes the position of image 68b of the hole in image 67. When the focus position is changed, the positions of the feature points in the images change. In the mathematical model, changing the focus position is synonymous with changing the focal length.

[0049] Figure 6 is a schematic diagram explaining the positions on the pixel sensor where feature points of the object are captured. A lens 37 is placed between the surface of the workpiece and the image sensor that generates the camera image. Focal lengths f1 and f2 correspond to the distance from the image sensor to the center of lens 37. Figure 6 shows focusing position A and focusing position B. Position A corresponds to lens 37 being placed at a position with focal length f1. Position B corresponds to lens 37 being placed at a position with focal length f2. Distances z1 and z2 from lens 37 to the workpiece change depending on focal lengths f1 and f2.

[0050] At position A, a feature 69 located at a distance X1 from the optical axis on the workpiece surface is detected by the image sensor at a distance u1 from the optical axis. At position B, a feature 69 located at a distance X2, the same distance as X1, is detected by the image sensor at a distance u2. For example, for position B, the relationship X2 / z2 = u2 / f2 holds. Because focal length f2 is greater than focal length f1, distance u2 on the image sensor is greater than distance u1. Thus, even if the position on the workpiece surface remains the same, the position of the feature in the camera image changes depending on the focus position.

[0051] The parameter setting unit 53 calculates parameters in a calculation model that calculates the position of a feature part of a workpiece from an image captured by a camera, based on the focus position. The feature position calculation unit 55 calculates a three-dimensional position from a specific position in the image using the calculation model. The three-dimensional position of the feature part on the surface of the workpiece is calculated using parameters set corresponding to the focus position.

[0052] Next, we will explain the calculation model for calculating three-dimensional positions in space from images captured by a camera. The position in a camera image corresponding to an arbitrary position in space is generally expressed by the following equation (1) using the pinhole camera model.

[0053]

number

[0054] The coordinate values ​​(X, Y, Z) of a three-dimensional position are expressed, for example, in a reference coordinate system 71. The coordinate values ​​(u, v) of a position on an image are expressed, for example, in an image coordinate system 73. The matrix of external parameters is a transformation matrix for converting a three-dimensional position in space into coordinate values ​​in a camera coordinate system 72. The matrix of internal parameters is a matrix for converting coordinate values ​​in the camera coordinate system 72 into coordinate values ​​in an image coordinate system 73 in the image. Here, the Z-axis value of the three-dimensional position, or the z-axis coordinate value in the camera coordinate system 72, is predetermined in accordance with the distance from the camera to the workpiece.

[0055] The above formula (1) is an ideal example where there is no lens distortion, etc. In practice, changes in parameters due to lens distortion, etc., must be taken into consideration. First, the calculation of the part of formula (1) relating the three-dimensional position in space and the matrix of external parameters can be expressed by the following formula (2).

[0056]

number

[0057] Using equation (2), coordinate values ​​(X, Y, Z) expressed in the reference coordinate system 71 can be converted into coordinate values ​​(x, y, z) expressed in the camera coordinate system 72. Next, in order to take into account the distortion of the camera lens, variables x' and y' are defined as shown in the following equations (3) and (4). Furthermore, variables x'' and y'' that take distortion into account are calculated as shown in equations (5) and (6). Here, the relationship between variables x', y', and r is as shown in equation (7).

[0058]

number

[0059] In equations (5) and (6), coefficients k1 to k6 are coefficients relating to lens distortion in the radial direction, and coefficients p1 and p2 are coefficients relating to lens distortion in the circumferential direction. Using variables x'' and y'' that take lens distortion into account, coordinate values ​​(u, v) on the image in image coordinate system 73 can be calculated as shown in the following equations (8) and (9). Equations (8) and (9) correspond to the part of equation (1) above that uses a matrix of internal parameters.

[0060]

number

[0061] The above explanation describes a method for calculating a position in an image from a three-dimensional position in space, but in this embodiment, based on the above relational expressions, a three-dimensional position (X, Y, Z) in space is calculated based on the coordinate values ​​(u, v) of a position on the image and the distance z from the camera 6 to the workpiece 38 in the camera coordinate system 72. The distance z from the camera 6 to the workpiece 38 can be determined in advance and stored in the storage unit 42. The feature position calculation unit 55 calculates a three-dimensional position (X, Y, Z) in space from the coordinate values ​​(u, v) of a specific position on the image based on a calculation model.

[0062] Here, the calculation model for calculating the three-dimensional position from the position in the image by referring to equations (2) to (9) includes the product f x ,f y , image center c x ,c y , and distortion-related coefficients k1 to k6, p1, and p2 are required. These parameters change depending on the focal position when the camera is focused.

[0063] In this embodiment, setting information 63 for setting parameters corresponding to the focus position is determined in advance. The setting information 63 is stored in the storage unit 42. The parameter setting unit 53 sets these parameters using the focus position and the setting information 63. Table 1 shows a table of parameter values ​​corresponding to the focus position pp as the setting information 63.

[0064] [Table 1]

[0065] Here, as an example of a parameter, the product f x In this case, the setting information 63 has parameter values ​​previously determined for a plurality of discrete focus positions pp. The parameter setting unit 53 sets parameters in the calculation model based on the parameter values ​​determined for each focus position. For example, if the focus position pp when the image is captured by the camera 6 is 1.4, the parameter setting unit 53 sets the product f x The value can be set to 2.8 by interpolating the value of . The parameters can be set by any method using a table containing discrete parameter values. For example, the median value of two parameters corresponding to two focus positions pp may be used, or the value of the parameter corresponding to the closest focus position pp may be used.

[0066] By using setting information including parameter values ​​for discrete focus positions, the parameter setting unit can set parameters according to any focus position. The parameter setting unit can set parameters through simple calculations. Alternatively, even if it is difficult to set the function described below, the parameter setting unit can set parameters according to the focus position.

[0067] As the setting information, a function for calculating parameters according to the focus position pp can be determined. The parameters can be calculated using a mathematical formula including the focus position pp. For example, as shown in equation (10), a function f(pp) for calculating the product f of the focal length and the effective pixel size for the focus position pp can be determined in advance. Alternatively, as shown in equation (11), a function k(pp) for calculating the distortion coefficient k for the focus position pp can be determined in advance.

[0068]

number

[0069] Such a function can be, for example, a high-order equation with the focus position pp as a variable. The parameter setting unit 53 can set each parameter, such as a parameter related to distortion, using the function. The feature position calculation unit 55 can calculate the three-dimensional position of the feature part based on the parameters set by the parameter setting unit 53.

[0070] A control flowchart in this embodiment is shown in Fig. 7. With reference to Figs. 1, 2, and 7, the operator determines setting information for calculating parameters of the calculation model in advance. Then, the operator stores setting information 63 in storage unit 42.

[0071] In step 80, the operation control unit 43 moves the camera 6 to an imaging position for imaging the workpiece 38. In this embodiment, the camera 6 is placed directly above the reference position P38a of the workpiece 38. The attitude of the camera 6 is also adjusted so that the direction of the Z axis of the camera coordinate system 72 is parallel to the vertical direction. Here, the distance from the surface of the plate-shaped portion 38b of the workpiece 38, on which the characteristic portion is formed, to the camera 6 is determined in advance.

[0072] Next, in step 81, the focus adjustment mechanism 24 of the camera 6 adjusts the focus of the camera 6. In this embodiment, the focus adjustment mechanism 24 has an autofocus function, so it automatically adjusts the focus. In step 82, the imaging control unit 58 captures an image with the camera 6. The image is captured in a focused state.

[0073] In step 83, the focus position detection unit 52 detects the focus position when the image was captured. The focus position detection unit 52 detects a predetermined variable corresponding to the position of the lens, for example. In step 84, the parameter setting unit 53 sets parameters of a calculation model for calculating the three-dimensional position of the characteristic part based on the focus position and setting information.

[0074] Next, in step 85, the feature detection unit 54 detects feature portions in the image by performing pattern matching. In this embodiment, the edge portion 38c in the image is detected by performing pattern matching using a reference image of the edge portion 38c of the plate-shaped portion 38b. The feature position calculation unit 55 detects the position of the feature portion in the image.

[0075] Next, in this embodiment, when the position of a characteristic part of a workpiece cannot be detected, the position of the camera 6 relative to the workpiece 38 is changed and an image is captured. For example, the characteristic part may appear white due to reflection of the light from the illumination, making the characteristic part unclear. In such a case, the characteristic part may be captured clearly by moving the position of the camera.

[0076] In step 86, the image processing unit 51 determines whether or not the position of the characteristic part has been detected. If the characteristic position calculation unit 55 cannot detect the position of the characteristic part, the control proceeds to step 87.

[0077] In step 87, the action command generation unit 59 generates a command to change the position of the camera 6. For example, the action command generation unit 59 generates a command to translate the camera 6 in a predetermined direction by a predetermined amount. With reference to FIG. 1, for example, the action command generation unit 59 generates a command to move the camera 6 in the direction of the X axis of the camera coordinate system 72. The action command generation unit 59 sends a action command for the robot 1 to the action control unit 43. The action control unit 43 changes the position and posture of the robot 1. Then, the control returns to step 81. The image processing unit 51 repeats the control from step 81 to step 86.

[0078] In step 86, if the characteristic position calculation unit 55 calculates the position of the characteristic part, the control proceeds to step 88. Note that if the characteristic part cannot be detected even after changing the position and posture of the robot multiple times, the control may be stopped.

[0079] In step 88, the feature position calculation unit 55 calculates the three-dimensional position of the feature based on the position of the feature in the image. The coordinate values ​​in the reference coordinate system 71 are calculated based on the coordinate values ​​in the image coordinate system 73 in the image. The feature position calculation unit 55 calculates the position of the workpiece based on the three-dimensional position of the feature. The position of the workpiece can be calculated, for example, in the reference coordinate system 71.

[0080] In step 89, the motion command generation unit 59 calculates the position and posture of the robot 1 based on the position of the workpiece. Then, in step 90, the motion command generation unit 59 sends a motion command to drive the robot 1 to the motion control unit 43. The motion control unit 43 drives the robot 1 and the hand 5 based on the motion command.

[0081] In this way, the imaging device according to the present embodiment sets the parameters of the calculation model for calculating a three-dimensional position corresponding to a specific position in an image captured by the visual sensor according to the focus position, and then calculates the three-dimensional position of the specific position based on the parameters according to the focus position.

[0082] By performing this control, the three-dimensional position of the characteristic part can be detected with high accuracy when the focus position changes. In particular, there is no need to fix the focus position in advance; an image can be captured at any focus position and the three-dimensional position can be calculated with little error. In other words, there is no need to determine in advance the position where the camera is in focus relative to the workpiece; the camera can be placed at any position, focused, and captured. In this embodiment, even if the number of focus positions (relative positions of the camera to the workpiece) increases, there is no need to add new parameters to the calculation model. Compared to conventional technology, the imaging device of this embodiment expands the range in which the robot can be driven and increases the number of patterns in which the robot can be driven.

[0083] In particular, if the focus adjustment mechanism of the visual sensor has a function for automatically adjusting the focus, the focus position can be set to any position within a predetermined range. In this case, the imaging device can set parameters corresponding to the focus position, making it possible to detect the exact position of the workpiece. Note that the visual sensor does not necessarily have to have the function for automatically adjusting the focus. In this case, the operator can manually adjust the focus. For example, the operator may adjust the focus by operating the input unit 27 while looking at the image displayed on the display unit 28 of the teaching operation panel 26.

[0084] In addition, the robot system in this embodiment includes a robot as a moving device that moves at least one of the workpiece and the visual sensor. When the relative position of the camera with respect to the workpiece changes on the robot, the focal position changes. Even in this case, the imaging device can set parameters corresponding to the focal position, allowing it to accurately detect the position of the workpiece.

[0085] 2, display unit 28 of teaching operation panel 26 in this embodiment displays the values ​​of the parameters set by parameter setting unit 53. The operator can check the parameter values ​​by looking at the parameters displayed on display unit 28 of teaching operation panel 26. In particular, when the three-dimensional position calculated by feature position calculation unit 55 is incorrect, the operator can check the parameter values ​​set according to each focus position.

[0086] Incidentally, the distance calculation unit 56 of the image processing unit 51 in this embodiment can calculate the distance from the camera 6 to the workpiece 38 based on the focal position detected by the focal position detection unit 52. The focal position depends on the distance between the camera 6 and the workpiece 38. For this reason, once the focal position is determined, the distance between the camera 6 and the workpiece 38 can be estimated.

[0087] The distance calculation unit 56 estimates the distance from the origin of the camera coordinate system 72 to the surface of the plate-shaped portion 38b of the workpiece 38 when the surface of the plate-shaped portion 38b is focused. For example, the operator can create in advance a function for calculating the z-axis coordinate value of the camera coordinate system 72 using the focus position pp as a variable. The z-axis coordinate value of the camera coordinate system 72 corresponds to the distance from the camera 6 to the workpiece 38. The distance calculation unit 56 can calculate the z-axis coordinate value of the camera coordinate system 72 using the focus position pp and the function. Alternatively, the operator can determine the distance from the camera 6 to the workpiece 38 for each of a plurality of discrete focus positions. The distance calculation unit 56 can calculate the distance from the camera 6 to the workpiece 38 by calculation such as interpolation based on the actually detected focus position pp.

[0088] In this way, the distance calculation unit 56 of this embodiment can calculate the distance from the camera 6 to the object. Generally, the distance from the camera to the object needs to be determined in advance. However, by providing the image processing unit 51 with the distance calculation unit 56, the distance from the camera 6 to the workpiece 38 can be calculated. For example, when manually adjusting the focus while viewing the image, the distance from the camera to the workpiece can be calculated. Therefore, the image processing unit 51 can calculate the three-dimensional position of the characteristic part of the workpiece without setting the distance from the camera to the object.

[0089] FIG. 8 is a schematic diagram illustrating a first step of another control of the first robot system in this embodiment. In this control, the image processing unit 51 detects the three-dimensional position of the workpiece 38 based on an image captured by placing the camera 6 at a first imaging position. Based on the position of the workpiece 38, the image processing unit 51 calculates a second imaging position that is closer to the workpiece 38 than the first imaging position. The image processing unit 51 moves the position and posture of the robot 1 to the second imaging position, as indicated by arrow 102. The second imaging position is a position where the distance from the target to the visual sensor is shorter than that at the first imaging position. The second imaging position is also a position where the workpiece 38 is positioned approximately in the center of the image.

[0090] Next, the image processing unit 51 calculates the three-dimensional position of the workpiece 38 based on the image captured at the second imaging position, and then controls the robot 1 to grip the workpiece 38 based on the position of the workpiece 38.

[0091] When the workpiece 38 is placed on the stand 95, there may be a large deviation in the position of the workpiece 38. Therefore, the position and posture of the robot 1 can be determined in advance so that the camera 6 is placed at a first imaging position away from the workpiece 38. At the first imaging position, control is performed to automatically adjust the focus using an autofocus function. At the first imaging position, the workpiece 38 appears small in the image captured by the camera 6. However, the feature detection unit 54 and feature position calculation unit 55 can detect the position of the workpiece 38.

[0092] Next, the operation command generation unit 59 calculates a second imaging position for the camera 6 to capture an image of the workpiece 38 at a position closer than the first imaging position. The second imaging position is determined so that the workpiece is positioned approximately in the center of the image. The second imaging position is also set at a position directly above the workpiece 38 where the camera 6 is closer to the workpiece 38.

[0093] The relationship between the current position of the workpiece 38, the movement amount of the camera 6, and the position of the workpiece 38 in the camera coordinate system 72 is expressed, for example, by the relationship of the following equation (12).

[0094]

number

[0095] Here, in order to position the workpiece 38 almost in the center of the image, the workpiece should be positioned on the optical axis of the camera. In other words, x = y = 0 in the camera coordinate system 72. By substituting x = y = 0 into equation (12) and transforming the equation, the following equation (13) can be obtained. Then, from equation (13), the movement amount of the camera in the reference coordinate system 71 (t x ,t y ,t z ) can be calculated.

[0096]

number

[0097] Next, regarding the distance from the workpiece 38 to the camera 6, the coordinate value of the z-axis of the surface of the workpiece 38 in the camera coordinate system 72 at the first imaging position is defined as z'. The ratio of the workpiece to the size of the image at the first imaging position is defined as a. As the ratio of the workpiece to the size of the image, for example, the ratio of the length of the workpiece to the length of the image in one direction in the image can be used. The ratio of the workpiece to the size of the image can be detected by the feature position calculation unit 55.

[0098] Next, let z'' be the z-axis coordinate value of the surface of the workpiece 38 in the camera coordinate system 72 at the second imaging position. Let k be the desired ratio of the workpiece to the image size. This ratio can be determined in advance by the operator. Here, when the camera 6 is moved from the first imaging position to the second imaging position, the focus position of the camera 6 changes. However, in this calculation, it is assumed that the focal length corresponding to the focus position remains constant even when changing from the first imaging position to the second imaging position. The coordinate value z'' at the second imaging position can be expressed as z'' = (k / a)z'. The camera 6 can be moved closer to the workpiece 38 so that the distance from the camera 6 to the workpiece 38 becomes the coordinate value z''.

[0099] The motion command generation unit 59 changes the position and posture of the robot 1 based on the amount of movement in the x-axis direction, y-axis direction, and z-axis direction of the camera coordinate system 72, so as to place the camera 6 at the second imaging position, as shown by arrow 102.

[0100] FIG. 9 shows a schematic diagram illustrating the second step of another control of the first robot system. FIG. 9 shows a schematic diagram of the robot system when the camera is placed at the second imaging position. The second imaging position of the camera 6 is closer to the workpiece 38 than the first imaging position. At the second imaging position, the camera 6 performs control to automatically adjust the focus using its autofocus function. In the image captured at the first imaging position, the proportion of the workpiece 38 in the image is small, so the position of the workpiece 38 may not be accurately detected. In the image captured at the second imaging position, the proportion of the workpiece 38 in the image is larger. As a result, the position of the workpiece 38 can be accurately calculated.

[0101] In this way, the operation command generation unit 59 can calculate the second imaging position based on the three-dimensional position of the workpiece 38 in the image captured at the first imaging position. Then, the feature position calculation unit 55 calculates the three-dimensional position of the feature part based on the image captured at the second imaging position. Even if the camera focus positions at the first imaging position and the second imaging position are different, the three-dimensional position of the feature part can be detected at each imaging position. In particular, by having the robot system perform work based on the image captured at the second imaging position, the work tool can be moved to an accurate position and work can be performed with high precision.

[0102] In the first robot system 3, the position of the workpiece 38 placed on the stand 95 is detected in order to grasp the workpiece 38, but this is not limited to this. The image processing unit 51 can detect the workpiece 38 based on the captured image and inspect the workpiece 38. For example, the image processing unit 51 can measure the dimensions of the workpiece from the captured image. The image processing unit 51 can then inspect the dimensions of the workpiece based on predetermined dimension judgment values. In this case, it is also possible to attach only a visual sensor to the robot 1 without attaching a work tool to the robot 1.

[0103] The inspection of a workpiece is not limited to the inspection of the workpiece's dimensions, and any other inspection can be performed. For example, an inspection can be performed to determine whether a specific part is located on the surface of the workpiece. Alternatively, an inspection can be performed to determine whether scratches exist on the surface of the workpiece. In either case, the position of the characteristic part can be accurately detected depending on the focus position, allowing for highly accurate inspection.

[0104] In the first robot system 3, the workpiece is stationary and the camera is moved by a moving device, but this is not limited to this. The camera may be fixed in position and the workpiece may be moved by a moving device. Alternatively, the moving device may be configured to move both the camera and the workpiece.

[0105] 10 shows a schematic diagram of the second robot system in this embodiment. In the second robot system 4, the camera 6 is fixed to a base 96. The workpiece 38 is supported by the robot 1. The second robot system 4 transports the workpiece 38 placed on a base 97 to the base 98 as shown by arrow 103. As the position and posture of the robot 1 change, the workpiece 38 is transported from position P38s to position P38e. The imaging device of the second robot system 4 detects a positional deviation within the hand 5 when the workpiece 38 is grasped by the hand 5.

[0106] The control device 2 controls the position and posture of the robot 1 so that the workpiece 38 is placed at a predetermined imaging position for detecting the three-dimensional position of the workpiece 38. The image processing unit 51 detects the three-dimensional position of a characteristic part of the workpiece 38 based on the image captured by the camera 6. For example, the edge of the bottom surface of the workpiece 38 can be detected as the characteristic part. The image processing unit 51 detects the position of the workpiece 38. The reference position of the workpiece 38 in the predetermined position and posture of the robot 1 is stored in the memory unit 42. The image processing unit 51 can calculate the deviation of the grip of the workpiece 38 by the hand 5 based on the reference position of the workpiece 38.

[0107] The operation command generation unit 59 calculates the position and posture of the robot 1 on the platform 98 based on the positional deviation of the workpiece 38 in the hand 5 so as to place the workpiece 38 at a desired position P38e. Then, the operation control unit 43 drives the robot 1 to place the workpiece 38 at position P38e.

[0108] In the second robot system 4, when the workpiece 38 is placed at a predetermined imaging position, the camera 6 performs focusing. The parameter setting unit 53 calculates the parameters of the calculation model based on the focus position. The feature position calculation unit 55 calculates the three-dimensional positions of the feature parts based on the calculated parameters. Then, the position of the workpiece 38 is detected based on the positions of the feature parts.

[0109] In the second robot system 4, too, the imaging position of the workpiece 38 may deviate from the desired position when the workpiece 38 is imaged. Alternatively, there may be individual differences in the dimensions of the workpiece 38. In such cases, even if the camera 6 is focused, the accurate position of the workpiece 38 can be calculated. As a result, the robot system 4 can transport the workpiece 38 to the desired position.

[0110] In the second robot system as well, the image processing unit 51 can capture an image of the workpiece at a first imaging position away from the camera to detect the approximate position of the workpiece, and then calculate a second imaging position that is closer to the camera than the first imaging position. Then, based on the image captured at the second imaging position, the gripping deviation of the workpiece 38 may be calculated. Furthermore, in the second robot system as well, inspections such as dimensional inspection of the workpiece may be performed.

[0111] The other configurations, actions, and effects of the second robot system are similar to those of the first robot system, and therefore will not be described repeatedly here.

[0112] Fig. 11 shows a schematic diagram of a transfer system according to this embodiment. Fig. 12 shows a block diagram of the transfer system according to this embodiment. With reference to Figs. 11 and 12, the transfer system 9 includes an imaging device that captures an image of a workpiece 38 to detect the workpiece 38. The transfer system 9 includes a conveyor 7 as a moving device that moves the workpiece 38. The transfer system 9 has a configuration in which the conveyor 7 is arranged in place of the robot 1 of the second robot system 4.

[0113] The workpiece 38 moves in the direction indicated by the arrow 104 as the conveyor 7 is driven. That is, the position of the workpiece 38 changes as the conveyor 7 is driven. The camera 6 serving as a visual sensor is supported by a support member 99.

[0114] The conveying system 9 includes a control device 8 that controls the conveyor 7 and the camera 6. The control device 8 is configured with an arithmetic processing device including a CPU and the like. The control device 8 includes a conveyor drive unit 46. The conveyor 7 includes a conveyor drive device 30 having a drive motor for driving the belt. Each drive motor is provided with a position detector 31 that detects the rotation position of the drive motor. The control device 8 includes an image processing unit 51 that processes images captured by the camera 6.

[0115] The control device 8 includes a control panel 32. Similar to the teaching control panel 26, the control panel 32 has an input unit 27 and a display unit 28. The display unit 28 can display parameters set by the parameter setting unit 53. The rest of the configuration of the control device 8 is the same as that of the control device 2 of the robot system shown in FIG.

[0116] In the conveyance system 9, the camera 6 is fixed at a predetermined position where it can capture an image of the workpiece 38. In the conveyance system 9, the position of the workpiece 38 can be detected or the workpiece 38 can be inspected based on the image captured by the camera 6. The image capturing position at which the workpiece 38 being conveyed by the conveyor 7 stops may vary. Alternatively, there may be individual differences in the dimensions of the workpiece 38.

[0117] The conveyance system 9 also performs focusing when capturing an image of the workpiece 38. Then, based on the focus position, parameters of a calculation model that calculates a three-dimensional position corresponding to a specific position in the image are set. Based on the parameters, the position of a characteristic part of the workpiece 38 is calculated. Then, based on the position of the characteristic part of the workpiece 38, the position of the workpiece 38 can be calculated and the workpiece 38 can be inspected.

[0118] In this way, the position of the camera 6 may be fixed. The moving device may be any device that moves the object or the camera. The rest of the configuration, operation, and effects of the transport system are the same as those of the first and second robot systems described above, so a description thereof will not be repeated here.

[0119] In each of the above-described controls, the order of steps can be changed as appropriate within the scope that does not change the functions and actions. The above-described embodiments can be combined as appropriate.

[0120] In the above-mentioned drawings, the same or equivalent parts are denoted by the same reference numerals. Note that the above-mentioned embodiments are merely examples and do not limit the invention. Furthermore, the embodiments include modifications of the embodiments as set forth in the claims. [Explanation of symbols]

[0121] 1. Robot 2. Control device 3,4 Robot System 6. Camera 7 Conveyor 8 Control Device 9. Conveyor System 24 Focus adjustment mechanism 28 Display section 37 Lens 38 Work 38c edge 40 Control device main body 42 Storage section 43 Motion control section 51 Image processing section 52 Focus position detection unit 53 Parameter setting section 54 Feature detection unit 55 Feature position calculation unit 56 Distance calculation unit 59 Operation command generation section 61 Operating Program 63 Setting information 66,67 images 68a,68b Hole images 69 Characteristic parts

Claims

1. a visual sensor that captures an image of an object; a focus position detection unit that detects a focus position when the visual sensor is in focus; a parameter setting unit that sets parameters for calculating a three-dimensional position corresponding to a specific position in the image captured by the visual sensor; a storage unit that stores setting information for setting parameters corresponding to a focus position; a feature detection unit that detects a predetermined feature portion in the image of the object; a feature position calculation unit that calculates a three-dimensional position of a feature portion using the parameters set by the parameter setting unit, the visual sensor is configured to capture a two-dimensional image; the feature position calculation unit calculates a three-dimensional position of the feature from a specific position of the feature in the two-dimensional image using a calculation model; In the calculation model, a distance from the visual sensor to a workpiece is determined in advance, The parameter setting unit sets at least one parameter in the calculation model based on a focus position and the setting information.

2. The imaging device according to claim 1 , wherein the visual sensor has a function of automatically adjusting focus.

3. the setting information includes parameters that are predetermined for a plurality of discrete focus positions, The imaging device according to claim 1 , wherein the parameter setting unit sets parameters for calculating a three-dimensional position based on parameters determined for each of a plurality of focus positions stored in the storage unit.

4. The imaging device according to claim 1 , further comprising a moving device for moving at least one of the object and the visual sensor.

5. an operation command generation unit that generates an operation command for the mobile device so as to change an imaging position of at least one of the object and the visual sensor; the operation command generation unit calculates a second imaging position based on a three-dimensional position of the object in an image captured at a first imaging position; the second imaging position is a position where a distance from the object to the visual sensor is smaller than that of the first imaging position, The imaging device according to claim 4 , wherein the characteristic position calculation section calculates the three-dimensional position of the characteristic portion based on the image captured at the second imaging position.

6. The imaging device according to claim 1 , further comprising a display unit that displays values ​​of the parameters set by the parameter setting unit.

7. a distance calculation unit that calculates a distance from the visual sensor to the object; The imaging device according to claim 1 , wherein the distance calculation unit calculates the distance from the visual sensor to the object based on the focus position detected by the focus position detection unit.

Images