A photographing apparatus for calculating a three-dimensional position based on an image photographed by a vision sensor

Abstract

A photographing apparatus includes a posture detection section that detects a posture of a camera with respect to a gravitational direction, and a parameter setting section that sets a parameter used to calculate a three-dimensional position corresponding to a specific position in an image captured by the camera. A storage section stores setting information used to set a parameter corresponding to the posture of the camera. The parameter setting section sets the parameter based on the gravitational direction, the posture of the camera, and the setting information.

Description

Technical Field

[0001] This invention relates to an imaging device for calculating three-dimensional position based on images captured by a vision sensor. Background Technology

[0002] Previously, apparatuses for detecting the three-dimensional position of an object by processing images obtained from images taken by a camera are known. For example, apparatuses for detecting the position of an object based on images taken by two cameras included in a stereo camera, and performing operations based on the detected position of the object are known (e.g., Japanese Patent Application Publication No. 2009-269110).

[0003] Furthermore, the state of the camera at the time of image capture can sometimes affect the image. For example, when a photographer is holding the camera, hand tremors can sometimes cause image distortion. Corrections for these effects are known in the prior art (e.g., Japanese Patent Application Publication No. Hei 11-44900 and Japanese Patent Application Publication No. 2011-259314).

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2009-269110

[0007] Patent Document 2: Japanese Patent Application Publication No. 11-44900

[0008] Patent Document 3: Japanese Patent Application Publication No. 2011-259314 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] In the case of calculating a 3D position based on its location in a 2D image captured by a camera, the 3D position of an object is calculated using a pre-defined computational model. This computational model includes various parameters for calculating the 3D position from its location in the 2D image.

[0011] A camera can photograph objects from various angles. For example, it can photograph an object with the lens's optical axis tilted relative to the horizontal. Furthermore, the camera's position and orientation can be changed using actuators such as robots. For instance, by mounting a camera on a multi-jointed robot, it is possible to photograph objects from various angles and detect their position. Moreover, the device equipped with the robot can perform various tasks based on the detected object's position. Thus, by using the camera without fixing it, the number of tasks that can be performed with the camera can be increased, thereby enhancing its value.

[0012] Furthermore, when the camera's posture is changed by actuators or similar means, the direction of the camera's line of sight also changes. At this time, due to the weight of the camera or lens, the relative position of the lens and the camera's light-receiving element may slightly shift. As a result, the three-dimensional position of the object may sometimes be calculated with a slight deviation.

[0013] As a method to improve the accuracy of detecting the three-dimensional position of an object, the pose of the camera relative to the workpiece can be fixed. By setting the parameters of the computational model near the pre-determined camera pose, the error in the three-dimensional position can be reduced. That is, the position and pose of the camera used to photograph the object can be determined, and parameters corresponding to that position and pose can be used.

[0014] On the other hand, without setting the parameters of the computational model based on a predetermined camera position and orientation, it is difficult to obtain the 3D position of an object with high precision. For example, sometimes the camera orientation is changed to an arbitrary orientation to detect the 3D position of the object. It is necessary to set the parameters of the computational model corresponding to each orientation. However, since countless parameters need to be set, there is a practical problem of difficulty in parameter setting. Therefore, a device that can detect the position of an object with high precision based on the camera orientation through simple control is desired.

[0015] Methods for solving problems

[0016] The imaging apparatus of this disclosure includes a visual sensor for photographing an object. The imaging apparatus includes a gravity direction detection unit for detecting the direction of gravity and a posture detection unit for detecting the posture of the visual sensor relative to the direction of gravity. The imaging apparatus includes: a parameter setting unit that sets parameters for calculating a three-dimensional position corresponding to a specific position in an image captured by the visual sensor; and a storage unit that stores setting information for setting the parameters corresponding to the posture of the visual sensor. The imaging apparatus includes: a feature detection unit that detects predetermined feature portions in an image of the object; and a feature position calculation unit that calculates the three-dimensional position of the feature portions using the parameters set by the parameter setting unit. The parameter setting unit sets the parameters based on the direction of gravity, the posture of the visual sensor, and the setting information.

[0017] Invention Effects

[0018] According to the present disclosure, an imaging device can be provided that can detect the three-dimensional position of an object with high precision based on the pose of a visual sensor. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the robot system implemented in this way.

[0020] Figure 2This is a block diagram of the robot system implemented in this way.

[0021] Figure 3 This is a plan view of the first workpiece in the implementation method.

[0022] Figure 4 This is an example of an image taken when the first workpiece was photographed.

[0023] Figure 5 This is a schematic diagram of the robot system used to photograph the second workpiece.

[0024] Figure 6 This is a plan view of the second workpiece in the implementation method.

[0025] Figure 7 This is an example of an image taken when the second workpiece was photographed.

[0026] Figure 8 It is a three-dimensional diagram of a camera illustrating the camera's coordinate system and the direction of gravity.

[0027] Figure 9 This is a flowchart illustrating the control of a robot system. Detailed Implementation

[0028] Reference Figures 1 to 9 The imaging device in this embodiment will be described. The imaging device of this embodiment functions as a three-dimensional position acquisition device that calculates the three-dimensional position of a specific location in an image based on an image captured by a vision sensor.

[0029] Figure 1 This is a schematic diagram of the robot system equipped with a camera in this embodiment. Figure 2 This is a block diagram of the robot system according to this embodiment. (Refer to...) Figure 1 as well as Figure 2 In this embodiment, the robot system 3 detects the position of the first workpiece 38, which is the object, to grasp and transport the first workpiece 38. Additionally, the robot system 3 detects the position of the second workpiece 39, which is the object, to perform the operation of placing the first workpiece 38 into the recess 39a of the second workpiece 39.

[0030] The robot system 3 includes a manipulator 5 as a working tool for gripping a first workpiece 38 and a robot 1 for moving the manipulator 5. The robot system 3 also includes a control device 2 for controlling the robot system 3. In addition, the robot system 3 includes a platform 95 for placing the first workpiece 38 and a platform 96 for supporting a second workpiece 39. The second workpiece 39 is supported on the platform 96 by predetermined support members.

[0031] In this embodiment, the robotic arm 5 is a work tool for gripping or releasing the workpiece 38. As a work tool installed on the robot 1, it is not limited to this method; any work tool corresponding to the operation performed by the robot system 3 can be used. For example, a welding torch or similar tool for performing welding can be used as the work tool.

[0032] The robot 1 in this embodiment is a multi-joint robot comprising multiple joints 18. Robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported on a rotating base 13. The rotating base 13 is supported on a base 14. Robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 for securing a manipulator 5. The components of robot 1 are configured to rotate about a predetermined drive axis. As robot 1, it is not limited to this configuration; any robot capable of changing the posture of its working tool can be employed.

[0033] The robot 1 in this embodiment includes a robot drive unit 21, which has a drive motor for driving components such as the upper arm 11. The manipulator 5 includes a manipulator drive unit 22 for driving the manipulator 5. In this embodiment, the manipulator drive unit 22 drives the manipulator 5 by pneumatic pressure. The manipulator drive unit 22 includes an air pump for supplying compressed air to a cylinder and a solenoid valve.

[0034] The control device 2 includes a control device body 40 and a teaching operation panel 26 for the operator to operate the control device body 40. The control device body 40 includes an arithmetic processing unit (computer) with a CPU (Central Processing Unit) as a processor. The arithmetic processing unit has RAM (Random Access Memory) and ROM (Read Only Memory) connected to the CPU via a bus. The robot 1 is driven based on the action instructions from the control device 2. The robot 1 automatically transports the first workpiece 38 based on the action program 61.

[0035] The control device main body 40 includes a storage unit 42 for storing arbitrary information related to the robot system 3. The storage unit 42 can be constructed from a non-temporary storage medium capable of storing information. For example, the storage unit 42 can be constructed from a storage medium such as volatile memory, non-volatile memory, magnetic storage medium, or optical storage medium.

[0036] The control device 2 inputs a pre-generated motion program 61 for performing the actions of the robot 1. Alternatively, the operator can use the teach pendant 26 to drive the robot 1, thereby setting the teach point of the robot 1. The control device 2 can then generate the motion program 61 based on the teach point.

[0037] The motion program 61 is stored in the storage unit 42. The motion control unit 43 sends motion commands to the robot drive unit 44 to drive the robot 1 based on the motion program 61. The robot drive unit 44 includes circuitry for driving a drive motor and supplies power to the robot drive device 21 based on the motion commands. Additionally, the motion control unit 43 sends motion commands to the robot drive unit 45 to drive the robot drive device 22. The robot drive unit 45 includes circuitry for driving an air pump, etc., and supplies power to the air pump, etc., based on the motion commands.

[0038] The motion control unit 43 is equivalent to a processor that is driven according to the motion program 61. The processor is configured to read information stored in the storage unit 42. The processor reads the motion program 61 and implements the control determined by the motion program 61, thereby performing the function of the motion control unit 43.

[0039] Robot 1 includes state detectors for detecting the position and orientation of robot 1. In this embodiment, the state detectors include position detectors 23 mounted on the drive motors of each drive axis of the robot drive unit 21. The position detectors 23 can be, for example, encoders that detect the rotational position of the output shaft of the drive motor. The position and orientation of robot 1 are detected based on the output of each position detector 23.

[0040] A reference coordinate system 71 was established for robot system 3, which remains stationary when the position and orientation of robot 1 change. Figure 1 In the example shown, the origin of a reference coordinate system 71 is positioned on the base 14 of robot 1. The reference coordinate system 71 is also called the world coordinate system. In the reference coordinate system 71, the position of the origin is fixed, and the orientation of the coordinate axes is also fixed. The reference coordinate system 71 has mutually orthogonal X-axis, Y-axis, and Z-axis as coordinate axes. Additionally, the W-axis is set as the coordinate axis around the X-axis. The P-axis is set as the coordinate axis around the Y-axis. The R-axis is set as the coordinate axis around the Z-axis.

[0041] The teach pendant 26 is connected to the control unit main body 40 via a communication device. The teach pendant 26 includes an input section 27 for inputting information related to the robot 1 and the robotic arm 5. The input section 27 consists of input components such as a keyboard and dials. The teach pendant 26 also includes a display section 28 for displaying information related to the robot 1 and the robotic arm 5. The display section 28 can be a liquid crystal display panel or an organic EL (Electroluminescence) display panel capable of displaying information. Furthermore, if the teach pendant has a touch panel display, the display panel functions as both an input section and a display section.

[0042] A tool coordinate system 74 is established for the robot system 3. This tool coordinate system 74 has an origin set at any position on the working tool. The position and posture of the tool coordinate system 74 change along with the working tool. In this embodiment, the origin of the tool coordinate system 74 is set at the tool tip point of the robot arm 5. The position of the robot 1 corresponds to the position of the tool tip point in the reference coordinate system 71 (the position of the origin of the tool coordinate system 74). In addition, the posture of the robot 1 corresponds to the posture of the tool coordinate system 74 relative to the reference coordinate system 71.

[0043] The robot system 3 of this embodiment includes an imaging device for detecting the positions of workpieces 38 and 39. The imaging device includes a camera 6, which serves as a vision sensor for capturing images of workpiece 38. The camera 6 in this embodiment is a two-dimensional camera for capturing two-dimensional images. The camera 6 is supported on the robot 1. The camera 6 is fixed to the manipulator 5 via a support member. The position and orientation of the camera 6 change together with the manipulator 5.

[0044] Camera 6 is capable of capturing images within the field of view 6a. Camera 6 has a focus adjustment mechanism 24 for adjusting the focus. The focus adjustment mechanism 24 in this embodiment has an autofocus function. That is, camera 6 has an autofocus function. Camera 6 is configured to automatically focus on workpieces 38 and 39 to capture images of workpieces 38 and 39 when the position and posture of robot 1 change. As a focus adjustment mechanism, a mechanism that performs focusing through arbitrary control, such as contrast detection or phase difference, can be used.

[0045] Alternatively, a camera equipped with a liquid lens can be used as a vision sensor. In this case, a mechanism for changing the shape of the liquid lens can be used as the focus adjustment mechanism. For example, a mechanism for changing the voltage applied to the liquid lens or a mechanism for moving a liquid lens holding member to change the water pressure applied to the liquid lens can be used.

[0046] In robot system 3, a camera coordinate system 72 is established as a sensor coordinate system for camera 6. The position and orientation of camera coordinate system 72 change along with camera 6. The origin of camera coordinate system 72 is set at a predetermined position of camera 6, such as the center of the lens or the optical center. Camera coordinate system 72 has mutually orthogonal X-axis, Y-axis, and Z-axis. In this embodiment, camera coordinate system 72 is set such that the Z-axis extends in a direction parallel to the optical axis of the lens of camera 6. In this embodiment, since camera 6 is fixed to robot arm 5, the position and orientation of camera coordinate system 72 relative to tool coordinate system 74 are fixed.

[0047] The imaging device in this embodiment includes a moving device for the moving camera 6. In the robot system 3, the robot 1 functions as the moving device. If the position and posture of the robot 1 change, the position and posture of the camera 6 also change.

[0048] The imaging device includes an image processing unit that processes images captured by a vision sensor. In the robot system 3 of this embodiment, the control unit main body 40 functions as an image processing unit. The control unit main body 40 includes an image processing unit 50 that processes images captured by a camera 6. The image processing unit 50 includes an imaging control unit 58 that sends instructions to the camera 6 to capture images.

[0049] The image processing unit 50 includes a gravity direction detection unit 51 for detecting the direction of gravity and a posture detection unit 52 for detecting the posture of the camera 6 relative to the direction of gravity. The image processing unit 50 includes a parameter setting unit 53, which sets parameters for calculating a three-dimensional position corresponding to a specific position in an image captured by the camera 6.

[0050] The image processing unit 50 includes a feature detection unit 54 that detects predetermined feature portions in images of workpieces 38 and 39. The image processing unit 50 also includes a feature position calculation unit 55 that calculates the three-dimensional position of the feature portions using parameters set by the parameter setting unit 53. The image processing unit 50 includes an acceleration detection unit 56 that detects the acceleration of a vision sensor when the robot is driven. Finally, the image processing unit 50 includes a motion command generation unit 59 that generates motion commands for the robot 1 and the manipulator 5 based on the results of image processing.

[0051] The image processing unit 50 is equivalent to a processor driven according to the action program 61. In particular, each of the units—gravity direction detection unit 51, posture detection unit 52, parameter setting unit 53, feature detection unit 54, feature position calculation unit 55, acceleration detection unit 56, shooting control unit 58, and motion command generation unit 59—is equivalent to a processor driven according to the action program 61. The processor reads the action program 61 and implements the control determined by the action program 61, thereby performing its function as each unit.

[0052] Reference Figure 1In the robot system 3 of this embodiment, workpiece 38 is arranged on the surface of the platform 95 using a predetermined method. For example, an operator or other robot system arranges workpiece 38 at a predetermined position on the surface of the platform 95. Similarly, workpiece 39 is arranged on the surface of the platform 96 using the same method. Then, robot 1 changes its position and posture, and uses manipulator 5 to grasp the workpiece 38 arranged on the upper surface of the platform 95. As shown by arrow 101, robot system 3 arranges workpiece 38 in the recess 39a of workpiece 39 by changing the position and posture of robot 1. Workpieces 38 and 39, with workpiece 38 arranged in the recess 39a, are transported using a predetermined method. For example, an operator or other robot system transports workpieces 38 and 39 to a predetermined position.

[0053] When workpiece 38 is positioned on the surface of the stand 95, there is a possibility of positional offset of workpiece 38 on the stand 95. Figure 1 In the example shown, the reference position of workpiece 38 is determined during the teaching operation of the teaching robot 1, which determines its position and posture. The reference position is the desired location for workpiece 38. However, in practice, when workpiece 38 is placed on the upper surface of the platform 95, it is sometimes placed at a position offset from the reference position. The same applies to workpiece 39; when workpiece 39 is placed on the surface of the platform 96, it is sometimes offset from the reference position determined during the teaching operation. Furthermore, each workpiece 38 and 39 has dimensional errors. In the robot system 3 of this embodiment, the three-dimensional positions of workpieces 38 and 39 are detected based on images captured by the camera 6.

[0054] In the robot system 3 of this embodiment, when gripping the first workpiece 38, the camera 6 captures an image of the workpiece 38. The image processing unit 50 calculates the three-dimensional position of the workpiece 38 based on the image. The image processing unit 50 controls the position and posture of the robot 1 to correspond to the position of the workpiece 38. Then, the image processing unit 50 changes the position and posture of the robot 1 and grips the workpiece 38 with the robotic arm 5.

[0055] When placing the first workpiece 38 into the recess 39a of the second workpiece 39, the control of detecting the three-dimensional position of the workpiece 39 is implemented in the same way as the control of detecting the three-dimensional position of the workpiece 38. The workpiece 39 is photographed using the camera 6. The image processing unit 50 calculates the three-dimensional position of the workpiece 39 based on the image of the workpiece 39. The image processing unit 50 calculates the position and posture of the robot 1 for placing the workpiece 38 into the recess 39a of the workpiece 39. The image processing unit 50 changes the position and posture of the robot 1, placing the workpiece 38 into the recess 39a of the workpiece 39. Afterwards, the robot arm 5 releases the workpiece 38, and the robot 1 returns to the position and posture at the start of the operation.

[0056] Figure 3This is a plan view showing the first workpiece in this embodiment. (Refer to...) Figure 1 and Figure 3 The workpiece 38 has a plate-shaped portion 38a and a plate-shaped portion 38b formed on the upper side of the plate-shaped portion 38a. Each plate-shaped portion 38a, 38b has a cuboid shape. The plate-shaped portion 38b has an edge portion 38c on the outer periphery of its upper surface. The edge portion 38c corresponds to a corner formed on the plate-shaped portion 38b. In this embodiment, the edge portion 38c is a characteristic portion of the workpiece 38.

[0057] Reference Figure 1 In this embodiment, when gripping the workpiece 38, a camera 6 is positioned above the workpiece 38 in the vertical direction. The distance from the surface of the workpiece 38 with the feature portion to the camera 6 is predetermined. In this example, the position and posture of the robot 1 are controlled such that the position of the upper surface of the plate-shaped portion 38b is a predetermined Z-axis value of the camera coordinate system 72. Furthermore, the posture of the camera 6 is adjusted so that the optical axis of the camera 6 lens is approximately perpendicular to the surface of the plate-shaped portion 38b of the workpiece 38 with the feature portion.

[0058] Figure 4 This example illustrates an image captured by a camera of the first workpiece. The camera 6 focuses to capture image 66. Image 66 contains the first workpiece 38. Furthermore, an image coordinate system 73 is established in the image 66 captured by the camera. The feature detection unit 54 of the image processing unit 50 detects the edge portion 38c as a feature part of the workpiece 38 by performing pattern matching. A reference image used for detecting the position of the edge portion 38c during pattern matching is pre-generated and stored in the storage unit 42. The feature detection unit 54 uses the reference image to detect the edge portion 38c as a feature part in the image captured by the camera 6.

[0059] The feature position calculation unit 55 calculates the position of a feature portion of the workpiece 38 in the image captured by the camera 6. For example, the position of the corner of the edge 38c in the image 68 is detected using the coordinate values ​​of the image coordinate system 73. Based on the position of the feature portion in the image, the feature position calculation unit 55 uses a calculation model to detect the three-dimensional position of the feature portion. The position of the feature portion of the workpiece 38 can be calculated using the reference coordinate system 71. Furthermore, the feature position calculation unit 55 calculates the position of the workpiece 38 based on the three-dimensional position of the feature portion of the workpiece 38. As the position of the workpiece 38, the position of any set point set for the workpiece 38 can be calculated. The position of the workpiece 38 can also be calculated using the reference coordinate system 71.

[0060] The motion command generation unit 59 calculates the position and posture of the robot 1 based on the three-dimensional position of the workpiece 38 calculated by the feature position calculation unit 55. Then, the position and posture of the robot 1 used to grip the workpiece 38 are sent to the motion control unit 43. Based on the motion commands received from the motion command generation unit 59, the motion control unit 43 drives the robot 1 and the manipulator 5 to grip the workpiece 38.

[0061] Figure 5 This is a schematic diagram showing the robot system positioning the first workpiece in the recess of the second workpiece. When positioning the first workpiece 38 in the recess 39a of the second workpiece 39, the same control as the control for gripping the first workpiece 38 is implemented. After the robot arm 5 grips the first workpiece 38, the motion control unit 43 changes the position and posture of the robot 1. The robot 1 configures the camera 6 to capture the position and posture of the second workpiece 39.

[0062] Figure 6 A plan view showing the second workpiece in this embodiment. (Refer to...) Figure 5 and Figure 6 The second workpiece 39 has a cuboid shape. The upper surface of the workpiece 39 is planar. A recess 39a corresponding to the shape of the plate-shaped portion 38a of the first workpiece 38 is formed on the upper surface of the workpiece 39. The stepped portion of the recess 39a of the second workpiece 39 becomes a characteristic portion when an image is captured by the camera 6.

[0063] Figure 7 This shows an example of an image taken with a camera of the second workpiece. Image 67 contains an image 69 of the upper surface of the second workpiece 39. (See reference...) Figures 5 to 7 The distance from the upper surface of the workpiece 39 with the characteristic portion to the camera 6 when photographing the second workpiece 39 is predetermined. The Z-axis position of the upper surface of the workpiece 39 in the camera coordinate system 72 is predetermined. In addition, the posture of the camera 6 is adjusted so that the optical axis of the lens of the camera 6 is approximately perpendicular to the upper surface of the workpiece 39 with the characteristic portion. This position and posture, which serve as the reference for the robot 1, are predetermined.

[0064] Reference Figure 2 , Figure 5 as well as Figure 7The feature detection unit 54 of the image processing unit 50 detects the recess 39a, which is a feature portion of the workpiece 39, by performing pattern matching. A reference image for detecting the recess 39a is pre-generated and stored in the storage unit 42. The feature position calculation unit 55 calculates the three-dimensional position of the feature portion of the workpiece 39 using a calculation model based on the position of the feature portion in the image 69 captured by the camera 6. For example, the feature position calculation unit 55 can calculate the three-dimensional position of the angle of the planar shape of the recess 39a. Furthermore, the feature position calculation unit 55 can calculate the three-dimensional position of the workpiece 39 based on the three-dimensional position of the feature portion.

[0065] The motion command generation unit 59 calculates the position and posture of the robot 1 based on the position of the second workpiece 39 calculated in the feature position calculation unit 55. The motion command generation unit 59 calculates the position and posture of the robot 1 for positioning the first workpiece 38 in the recess 39a of the second workpiece 39. Then, the motion command generation unit 59 sends the position and posture of the robot 1 for positioning the workpiece 38 in the recess 39a of the workpiece 39 to the motion control unit 43. Based on the motion command received from the motion command generation unit 59, the motion control unit 43 drives the robot 1 and the manipulator 5 to position the first workpiece 38 in the recess 39a of the second workpiece 39.

[0066] In the control of the conveying workpiece in this embodiment, the feature detection unit 54 detects feature portions, and the feature position calculation unit 55 calculates the three-dimensional position of workpieces 38 and 39 with high precision based on the position of the feature portions. Therefore, the robot system 3 can grasp workpiece 38 with high precision and place workpiece 38 in the recess 39a of workpiece 39. Even if the position of workpiece 38 on the platform 95 deviates from the reference position or there is a dimensional error in workpiece 38, the robot system 3 can still grasp workpiece 38 with high precision. In addition, even if the position of workpiece 39 on the platform 96 deviates from the reference position or there is a dimensional error in workpiece 39, workpiece 38 can still be placed in the recess 39a of workpiece 39 with high precision.

[0067] The feature position calculation unit 55 of this embodiment calculates the three-dimensional position based on a specific position in a two-dimensional image using a predetermined calculation model. Here, when the camera's posture changes, the relative position of the camera's light-receiving element with respect to the lens may change due to the effects of the camera's own weight and the lens's own weight. Furthermore, in the case of a camera containing a liquid lens, the shape of the lens may change depending on the camera's posture. As a result, the three-dimensional position changes relative to a specific position in the two-dimensional image captured by the camera 6, and sometimes the three-dimensional position of the object cannot be accurately calculated.

[0068] The image processing unit 50 of this embodiment has the function of correcting the three-dimensional position considering the posture of the camera 6. Setting information 63 for setting parameters corresponding to the posture of the camera 6 is predetermined. The setting information 63 is stored in the storage unit 42. The parameter setting unit 53 calculates parameters in a calculation model based on the direction of gravity, the posture of the camera 6, and the setting information 63, for calculating the position of a three-dimensional feature portion based on the position of the feature portion in the image captured by the camera 6. The feature position calculation unit 55 uses the parameters set corresponding to the posture of the camera 6 to calculate the three-dimensional position of the feature portion on the workpiece surface. Furthermore, in this embodiment, when the posture of the camera 6 changes, the parameter setting unit 53 changes the value of the parameters according to the posture of the camera 6.

[0069] Next, a computational model for calculating a three-dimensional position in space based on a specific position in an image captured by a camera will be explained. The position in an image captured by a camera corresponding to any position in space is generally represented by the following equation (1) based on the pinhole camera model.

[0070] [Formula 1]

[0071]

[0072] [u, v]: Coordinates in the camera coordinate system

[0073] f x f y The product of focal length and effective pixel size

[0074] c x c y Image Center

[0075] [X, Y, Z]: Coordinate values ​​in the reference coordinate system

[0076] In this embodiment, the coordinates (X, Y, Z) of the three-dimensional position are represented by the reference coordinate system 71. The coordinates (u, v) of the position on the image are represented by the image coordinate system 73. The matrix of extrinsic parameters is a transformation matrix used to transform the three-dimensional position in space into coordinates in the camera coordinate system 72. Furthermore, the matrix of intrinsic parameters is a matrix used to transform the coordinates in the camera coordinate system 72 into coordinates in the image coordinate system 73. Here, the Z-axis value of the three-dimensional position in the reference coordinate system 71 or the Z-axis value in the camera coordinate system 72 is predetermined in relation to the distance from the camera to the workpiece.

[0077] Equation (1) above is an ideal example where there is no lens distortion, etc. In reality, we consider the changes in parameters caused by lens distortion, etc. Part of the matrix operation of the three-dimensional position in space and the external parameters in equation (1) can be represented by equation (2) below.

[0078] [Formula 2]

[0079]

[0080] t: [t1, t2, t3]

[0081] R: 3×3 rotation matrix

[0082] Equation (2) transforms the coordinate values ​​(X, Y, Z) represented by the reference coordinate system 71 into coordinate values ​​(x, y, z) represented by the camera coordinate system 72. Next, to account for lens distortion, variables x' and y' are defined as shown in equations (3) and (4). Furthermore, variables x' and y', taking distortion into account, are calculated as shown in equations (5) and (6). Here, the relationship between variables x', y', and r is shown in equation (7).

[0083] [Formula 3]

[0084] x′=x / z…(3)

[0085] y′=y / z…(4)

[0086]

[0087]

[0088] r 2 =x′ 2 +y′ 2 …(7)

[0089] In equations (5) and (6), coefficients k1 to k6 are coefficients related to lens distortion in the radial direction, and coefficients p1 and p2 are coefficients related to lens distortion in the circumferential direction. Using the variables x” and y” that take into account lens distortion, the coordinate values ​​(u, v) on the image in image coordinate system 73 can be calculated as shown in equations (8) and (9) below. Equations (8) and (9) are the parts corresponding to the matrix operations of the internal parameters in equation (1) above.

[0090] [Formula 4]

[0091] u = f x ·x″+c x …(8)

[0092] v = f y ·y″+c y …(9)

[0093] The above description explained a method for calculating the position in an image based on a three-dimensional position in space. However, in this embodiment, the three-dimensional position (X, Y, Z) in space is calculated based on the above-described relationship and the coordinate values ​​(u, v) of the position in the image, as well as the distance from the camera 6 to the workpieces 38 and 39 in the camera coordinate system 72 (the coordinate values ​​of the Z-axis of the camera coordinate system 72). The distance from the camera 6 to the workpiece 38 can be predetermined and stored in the storage unit 42. The feature position calculation unit 55 calculates the three-dimensional position (X, Y, Z) in space based on the calculation model and the coordinate values ​​(u, v) of a specific position in the image.

[0094] Here, referring to equations (2) to (9), in the calculation model for calculating the three-dimensional position based on the position in the image, the variable f, which is the product of the focal length and the effective size of the image, is the product f. x f y The variable c representing the image center is the image center c. x c y The parameters of the computational model depend on the orientation of the camera 6. Specifically, they depend on the camera's orientation relative to the direction of gravity. Alternatively, they depend on forces (accelerations) acting on the camera, such as gravity.

[0095] Figure 8 This is a three-dimensional view of a camera used to illustrate the orientation of the camera coordinate system relative to the direction of gravity. In this embodiment, the camera coordinate system 72 is set such that its Z-axis overlaps with the optical axis 6b of the lens. In this embodiment, the orientation of the camera coordinate system 72 is used to represent the orientation of the camera relative to the direction of gravity. The direction indicated by arrow 102 is the direction of gravity, and the camera coordinate system 72 is tilted relative to the direction of gravity.

[0096] The gravity direction detection unit 51 detects the direction of gravity. In this embodiment, the coordinate values ​​representing the gravity direction via the reference coordinate system 71 are pre-stored in the storage unit 42. The gravity direction detection unit 51 retrieves the coordinate values ​​of the reference coordinate system 71 representing the gravity direction from the storage unit 42. Alternatively, a gravity sensor can be configured on the wrist or similar part of a robot whose posture changes along with the camera. The gravity direction detection unit 51 can then detect the direction of gravity based on the output of the gravity sensor. Furthermore, sometimes the camera includes a gravity sensor. In this case, the gravity direction detection unit 51 can detect the direction of gravity from the camera's gravity sensor.

[0097] The posture detection unit 52 detects the posture of the camera 6 relative to the direction of gravity based on the position and posture of the robot 1 and the position of the camera 6 within the robot 1 on which the camera 6 is mounted. In this embodiment, the camera 6 is fixed to the manipulator 5. The position of the camera 6 within the robot 1 and its posture relative to the manipulator 5 are fixed. Therefore, the posture detection unit 52 calculates the position and posture of the camera coordinate system 72 based on the output of the position detector 23. The position and posture of the camera coordinate system 72 can be represented using the coordinate values ​​of the reference coordinate system 71. Alternatively, the posture detection unit 52 calculates the position and posture of the tool coordinate system 74 using the reference coordinate system 71 based on the output of the position detector 23. Furthermore, the posture detection unit 52 can calculate the position and posture of the camera coordinate system 72 based on the position and posture of the tool coordinate system 74.

[0098] Next, the posture detection unit 52 calculates the posture of the camera coordinate system 72 relative to the direction of gravity. In this embodiment, the direction of gravity is decomposed into components of the X-axis, Y-axis, and Z-axis of the camera coordinate system 72. That is, the direction of gravity is represented by the coordinate values ​​of the X-axis, Y-axis, and Z-axis. The length of the arrow 102 corresponding to the magnitude of gravity can be arbitrary. In this embodiment, the magnitude of gravity is set to 9.8. In this embodiment, the coordinate values ​​of the vector of the direction of gravity indicated by the arrow 102 (g... x g y g z The pose of the camera coordinate system 72 relative to the direction of gravity corresponds to the direction of gravity. When the variable g represents the magnitude of the vector of the direction of gravity indicated by arrow 102, the following relationship (10) holds.

[0099] [Formula 5]

[0100]

[0101] For example, when the optical axis 6b of the lens of camera 6 is parallel to the vertical direction, the coordinate values ​​(0, 0, 9.8) of the vector indicated by arrow 102 correspond to the orientation of the camera coordinate system 72 relative to the direction of gravity. Next, in this embodiment, setting information 63 is predetermined for setting parameters corresponding to the orientation of camera 6 relative to the direction of gravity. The parameter setting unit 53 uses the orientation of camera 6 relative to the direction of gravity and the setting information 63 to set the parameters of the calculation model. Table 1 shows a portion of the table containing the values ​​of the parameters corresponding to the orientation of camera 6, as part of the setting information 63.

[0102] [Table 1]

[0103] Table 1

[0104] <![CDATA[g x ]]> <![CDATA[g y ]]> <![CDATA[g z ]]> fx 9.8 0.0 0.0 <![CDATA[f x1 ]]> 9.0 3.9 0.0 <![CDATA[f x2 ]]> 8.0 5.7 0.0 <![CDATA[f x3 ]]> 7.0 6.9 0.0 <![CDATA[f x4 ]]> 6.0 7.7 0.0 <![CDATA[fx5]]> 5.0 8.4 0.0 <![CDATA[f x6 ]]> 4.0 8.9 0.0 <![CDATA[f x7 ]]> 3.0 9.3 0.0 <![CDATA[f x8 ]]> 2.0 9.6 0.0 <![CDATA[f x9 ]]> 1.0 9.7 0.0 <![CDATA[f x10 ]]> 9.0 3.0 2.5 <![CDATA[f x11 ]]> 9.0 2.0 3.3 <![CDATA[f x12 ]]> 9.0 1.0 3.7 <![CDATA[f x13 ]]>

[0105] Here, as an example of a parameter, the product f of focal length and effective pixel size is shown. x The value of . The setting information 63 here refers to multiple discrete variables, i.e., coordinate values ​​(g), representing the pose of camera 6. x g y g z The product f, which is a parameter, is predetermined. x1 ~f x13 The value of .

[0106] The parameter setting unit 53 sets the parameters in the calculation model based on the values ​​of parameters determined for each pose of the camera 6 relative to the direction of gravity. For example, the parameter setting unit 53 obtains the two coordinate values ​​(g) that are closest to the coordinate values ​​related to the pose of the camera 6 when it takes an image. x g y g z The product of f) x The parameter setting unit 53 can adjust the two products f using coordinate values ​​related to the camera's posture. x The parameters can be set by interpolating the values. Alternatively, the parameters can be set in any way using a table containing discrete parameter values. For example, the median value of the two parameters corresponding to two coordinate values ​​can be used, or the value of the parameter corresponding to any nearest coordinate value can be used. Regarding the product f as another parameter... y The variable c at the image center, and the variables k and p related to distortion, can also be set using the same method.

[0107] The parameter setting unit can set parameters corresponding to any camera pose by using setting information containing discrete parameter values ​​relative to the camera pose. The parameter setting unit can set parameters through simple calculations. Alternatively, even when it is difficult to set the function described below, parameters corresponding to the camera pose can still be set.

[0108] Next, as setting information, a function for calculating parameters based on the camera's pose can be determined. The parameters can be calculated using a formula that includes variables related to the camera's pose. For example, as shown in equation (11), coordinate values ​​related to the camera's pose can be predetermined as variables, and the parameters can be calculated using variable g. x g y g z The function f(g) that calculates variable f x g y g z Alternatively, as shown in equation (12), a function k(g) can be predetermined to calculate the distortion coefficient k for variables related to the camera's pose. x g y gz ).

[0109] [Formula 6]

[0110] f = f(g) x g y g z )…(11)

[0111] k=k(g x g y g z (12)

[0112] As such a function, it can, for example, take a variable g that is related to the camera's pose. x g y g z The parameter setting unit 53 can use functions to set each parameter. The feature position calculation unit 55 can calculate the three-dimensional position of the feature portion based on the parameters set by the parameter setting unit 53.

[0113] Alternatively, as setting information, machine learning can be employed to calculate parameters based on the camera's pose. For example, the parameter setting unit obtains multiple combinations of the camera's pose relative to the direction of gravity and the values ​​of the parameters corresponding to the camera's pose as training data (labels). The parameter setting unit generates a learning model through supervised learning. Then, the parameter setting unit can use the learning model to set the parameter values ​​based on variables related to the camera's pose.

[0114] In the above embodiment, as a variable representing the camera's attitude relative to the direction of gravity, the vector of the direction of gravity is decomposed into components along the X-axis, Y-axis, and Z-axis of the camera coordinate system, but this method is not limited to this. Any variable representing the camera's attitude relative to the direction of gravity can be used. For example, the coordinate values ​​of the camera coordinate system's attitude (the coordinate values ​​of the W-axis, P-axis, and R-axis of the reference coordinate system 71) expressed by the coordinate values ​​of the reference coordinate system 71 can be calculated as a variable representing the camera's attitude relative to the direction of gravity.

[0115] Figure 9 A flowchart illustrating the control process of this embodiment. (Refer to...) Figure 1 , Figure 2 as well as Figure 9 The operator pre-determines the parameter settings used to set the calculation model. Then, the operator stores the setting information 63 in the storage unit 42. First, the control of the robot system 3 gripping the workpiece 38 will be explained as an example.

[0116] In step 81, the motion control unit 43 moves the camera 6 to a predetermined shooting position for photographing the workpiece 38. In this embodiment, the camera 6 is positioned directly above the reference position of the workpiece 38. Next, in step 82, the focus adjustment mechanism 24 of the camera 6 focuses the camera 6. The focus adjustment mechanism 24 in this embodiment has an autofocus function, so it automatically focuses. The shooting control unit 58 uses the camera 6 to capture an image.

[0117] In step 83, the gravity direction detection unit 51 detects the gravity direction. For example, the gravity direction detection unit 51 obtains coordinate values ​​from the storage unit 42 representing the gravity direction as expressed by coordinate values ​​in the reference coordinate system 71. The pose detection unit 52 detects the pose of the camera 6 based on the position and pose of the robot 1. For example, the pose detection unit 52 can detect the pose of the camera coordinate system 72 using the coordinate values ​​in the reference coordinate system 71. Next, the pose detection unit 52 detects the pose of the camera relative to the gravity direction when an image is captured.

[0118] In step 84, the parameter setting unit 53 sets the parameters of the calculation model for calculating the three-dimensional position of the feature part based on variables and setting information related to the posture of the camera 6.

[0119] Next, in step 85, the feature detection unit 54 detects feature portions in the image by performing pattern matching. In this example, pattern matching is performed using a reference image of the edge 38c of the plate-shaped portion 38b, thereby detecting the edge 38c in the image. The feature position calculation unit 55 detects the position of the feature portions in the image.

[0120] Next, in the control example of this embodiment, when the position of the feature portion of the workpiece cannot be detected, control is implemented to change the position of the camera 6 relative to the workpiece 38 to capture an image. For example, sometimes the illumination light reflects off the feature portion, making it appear white and indistinct. In such cases, by moving the position of the camera, it is sometimes possible to capture a clear image of the feature portion.

[0121] In step 86, the image processing unit 50 determines whether the position of the feature portion has been detected. If the feature position calculation unit 55 cannot detect the position of the feature portion, control is transferred to step 87.

[0122] In step 87, the motion command generation unit 59 generates a command to change the position of the camera 6. For example, the motion command generation unit 59 generates a command to move the camera 6 in a predetermined direction by a predetermined amount of parallel movement. The motion command generation unit 59 can move the camera 6 in a parallel direction perpendicular to the Z-axis of the camera coordinate system 72. (Refer to...) Figure 1For example, the motion command generation unit 59 generates a command to move the camera 6 in the direction of the X-axis of the camera coordinate system 72.

[0123] The motion command generation unit 59 sends motion commands for robot 1 to the motion control unit 43. The motion control unit 43 changes the position and posture of robot 1. Then, control returns to step 82. The image processing unit 50 repeats the control steps 82 to 86.

[0124] In step 86, if the feature position calculation unit 55 detects the position of the feature portion, control is transferred to step 88. Furthermore, if the feature portion cannot be detected even after multiple changes to the robot's position and posture, control can be stopped.

[0125] In step 88, the feature position calculation unit 55 calculates the three-dimensional position of the feature portion based on its position in the image. The feature position calculation unit 55 calculates the coordinate values ​​of the reference coordinate system 71 based on the coordinate values ​​of the image coordinate system 73 in the image. The feature position calculation unit 55 calculates the position of the workpiece based on the three-dimensional position of the feature portion. The position of the workpiece can be calculated, for example, using the reference coordinate system 71.

[0126] In step 89, the motion command generation unit 59 calculates the position and posture of the robot 1 based on the position of the workpiece 38. Then, in step 90, the motion command generation unit 59 sends motion commands to drive the robot 1 to the motion control unit 43. The motion control unit 43 drives the robot 1 and the manipulator 5 based on the motion commands.

[0127] Reference Figure 2 , Figure 5 as well as Figure 9 When the first workpiece 38 is positioned in the recess 39a of the second workpiece 39, it is also possible to perform the same operation as... Figure 9 The control shown is the same. In step 81, robot 1 positions camera 6 in a direction perpendicular to the surface of workpiece 39. In step 82, camera 6 captures an image of the surface of workpiece 39. In step 83, gravity direction detection unit 51 detects the gravity direction. Posture detection unit 52 detects the posture of the camera relative to the gravity direction. In step 84, parameter setting unit 53 calculates parameters in the calculation model based on the posture of camera 6 relative to the gravity direction.

[0128] Next, in step 85, the feature detection unit 54 detects feature portions of the workpiece in the image. For example, it detects the position of the corner portion of the recess 39a of the workpiece 39. The feature position calculation unit 55 detects the position of the feature portions in the image.

[0129] In step 86, the image processing unit 50 determines whether the position of the feature portion has been detected. If the position of the feature portion cannot be detected, in step 87, the motion command generation unit 59 changes the position of the camera 6. For example, the camera 6 is moved parallel to the camera in a direction perpendicular to the Z-axis of the camera coordinate system 72. Then, after changing the position of the camera 6, the control steps 82 to 86 are repeated.

[0130] If the position of the feature portion is detected in step 86, the feature position calculation unit 55 calculates the position of the second workpiece 39 in step 88. Then, based on the position of the second workpiece 39, the position and posture of the robot 1 when configuring the first workpiece 38 are calculated. That is, the position and posture of the robot 1 when the first workpiece 38 is inserted into the recess 39a of the second workpiece 39 are calculated. Then, the robot is driven in step 90. The motion control unit 43 drives the robot 1 and the manipulator 5 based on motion commands. After the first workpiece 38 is configured in the recess 39a, the manipulator 5 releases the workpiece 38.

[0131] The imaging device of this embodiment sets parameters for a calculation model that calculates the three-dimensional position corresponding to a specific location in an image captured by a vision sensor, based on the camera's posture. Then, the three-dimensional position of the specific location is calculated based on the parameters corresponding to the vision sensor's posture. Through this control, the three-dimensional position of a feature portion can be detected with high precision based on the camera's posture. In particular, it is not necessary to pre-fix the camera's posture relative to the workpiece; images can be captured with any camera posture corresponding to the workpiece's posture, and the calculated three-dimensional position has minimal error. Compared to the prior art, the imaging device of this embodiment can expand the range of robot manipulation or increase the number of robot manipulation modes.

[0132] Furthermore, when the focus adjustment mechanism of the vision sensor has an autofocus function, the weight increases. Additionally, due to the mechanism that moves the lens, the internal wobbling of the lens increases. As a result, the relative positional relationship between the lens and the camera's light-receiving element changes more significantly. Therefore, when the camera has an autofocus function, the control effect of this embodiment is significantly improved.

[0133] Alternatively, the vision sensor may not have autofocus capability. For example, it could be a camera with a lens that has manual focus. Or, it could be a camera with a lens that has a fixed focus position.

[0134] Furthermore, the robot system of this embodiment includes a robot as a moving device for moving the vision sensor. The robot can change the camera's orientation. Even if the camera's orientation is changed by the robot, the imaging device of this embodiment can set parameters according to the camera's orientation, enabling high-precision detection of the workpiece's three-dimensional position.

[0135] Reference Figure 2 In this embodiment, the display unit 28 of the teach pendant 26 displays the values ​​of the parameters set by the parameter setting unit 53. The operator can confirm the values ​​of the parameters displayed on the display unit 28 of the teach pendant 26. In particular, if the three-dimensional position calculated by the feature position calculation unit 55 is incorrect, the operator can confirm the values ​​of the parameters set according to the posture of the camera 6.

[0136] Furthermore, the parameter setting unit 53 in this embodiment is configured to set parameters excluding the influence of gravity. The parameters excluding the influence of gravity can be predetermined and stored in the storage unit 42. The parameter setting unit 53 can retrieve these parameters from the storage unit 42.

[0137] The display unit 28 in this embodiment is configured to display the three-dimensional position of a feature portion calculated using parameters corresponding to the camera's posture relative to the direction of gravity, and the three-dimensional position of the feature portion calculated using parameters excluding the influence of gravity. With this configuration, the operator can confirm the magnitude of the positional shift caused by gravity. The operator can confirm the magnitude of the effect of gravity on the image. Furthermore, the operator can confirm the amount of positional shift of the feature portion in the image or the amount of three-dimensional positional shift of the workpiece. The operator can confirm the accuracy of the robot system's operation.

[0138] In the above embodiment, parameters for transforming a position in a two-dimensional image into a three-dimensional position are set based on the camera's orientation relative to the direction of gravity, but this method is not limited to this. The parameter setting unit 53 can set parameters based on setting information that includes temperature in the variables.

[0139] Reference Figure 2 In this embodiment, the robot system 3 is equipped with a temperature detector 31 that detects the temperature of the camera 6 body. The temperature is not limited to the temperature of the camera 6 body; any temperature associated with the camera 6 can be used. For example, the lens temperature or the ambient temperature of the robot system 3 can also be used.

[0140] If the camera is operated for an extended period, the temperature of the internal electronic components may rise. Alternatively, the temperature of the motor driving the lens may rise. As a result, the temperature of the camera body, the lens, and the support components that support the lens may change, causing changes in lens distortion or the relative position of the sensor with respect to the lens. Therefore, when the temperature changes, the parameters in the calculation model may change. Therefore, in addition to the camera's orientation relative to gravity, the parameter setting unit 53 can also use temperature as a variable to set the parameters of the calculation model.

[0141] As a method for adding such temperature to the variable parameter, in addition to the camera's attitude relative to the direction of gravity variable shown in Table 1, it is also possible to add temperature to the variable. That is, it is possible to generate a variable g that represents the coordinate value of the direction of gravity. x g y g z In addition, a table of parameter values ​​for temperature t is also included as a variable. Alternatively, as shown in equation (13), it can be obtained by using the variable g as a parameter. x g y g z In addition, temperature t is also used as a function f(g) x g y g z The parameters are calculated using t).

[0142] [Formula 7]

[0143] f = f(g) x g y g z ,t)…(13)

[0144] This control allows for high-precision calculation of 3D position by considering not only the camera's orientation relative to gravity but also the effects of temperature. This is particularly relevant in cameras containing liquid lenses, which are strongly affected by temperature. For example, the curvature of a liquid lens changes due to variations in surface tension, thus being significantly influenced by the liquid's temperature. Therefore, by setting parameters in the calculation model based on temperature in addition to the camera's orientation, high-precision calculation of 3D position becomes possible.

[0145] In the above embodiments, the workpiece is photographed while the camera is stationary, but this method is not limited to. The workpiece can also be photographed while the camera is moving. For example, refer to... Figure 1 The platform 95 can be replaced by a conveyor that moves the workpiece 38 in a horizontal direction.

[0146] In this case, the position and posture of robot 1 can be controlled so that camera 6 follows the movement of workpiece 38. For example, when workpiece 38 is moved along the Y-axis of reference coordinate system 71 by a conveyor, camera 6 takes pictures while moving along the Y-axis of reference coordinate system 71. Furthermore, workpiece 38 can be gripped during its movement. In such cases, workpiece 38 may be photographed during periods when camera 6 accelerates or decelerates. That is, images may be captured during periods when acceleration is applied to camera 6.

[0147] Reference Figure 2 In this embodiment, the parameter setting unit 53 of the image processing unit 50 can add the acceleration applied to the camera 6 when the camera 6 captures an image to the gravitational acceleration. The acceleration detection unit 56 calculates the position of the origin of the camera coordinate system 72 based on the output of the position detector 23. The acceleration detection unit 56 calculates the acceleration applied to the camera coordinate system 72 based on the position and time of the origin of the camera coordinate system 72. That is, the acceleration detection unit 56 calculates the magnitude and direction of the acceleration applied to the camera 6.

[0148] The parameter setting unit 53 can decompose the acceleration applied to the camera coordinate system 72 into the X-axis component ax and the Y-axis component a of the camera coordinate system 72. y And the Z-axis component az. The acceleration due to camera movement and gravitational acceleration are combined, and the acceleration components g' applied to the camera coordinate system 72 are the coordinate axis components of the acceleration. x g' y g' z It can be expressed by equations (14) to (16).

[0149] [Formula 8]

[0150] -a x +g x =g′ x …(14)

[0151] -a y +g y =g′ y …(15)

[0152] -a z +g z =g′ z …(16)

[0153] In this way, it can be determined that the coordinate value (g') in camera coordinate system 72 was applied to camera 6. x , g' y , g' zThe acceleration of ) . Then, the parameter setting unit 53 can be based on the variable g' x g' y g' z The values ​​of variables f', c', k', and p', which are parameters, are set using the setting information shown in Table 1. Alternatively, the parameter setting unit 53 can set the values ​​of variables g' based on the functions shown in equations (11) and (12). x g' y g' z The variables f', c', k', and p', which serve as parameters of the computational model, are calculated. Then, the feature position calculation unit 55 is able to calculate the three-dimensional position of the feature portion based on the calculated parameters of the computational model.

[0154] By implementing control to add the acceleration applied to the camera 6 by the robot 1 to the gravitational acceleration, the three-dimensional position can be detected with high precision even when the camera is accelerating or decelerating.

[0155] Furthermore, in the above embodiment, the parameters of the calculation model are calculated based on the gravity direction, and the calculated parameters are used to transform the position of the two-dimensional image captured by the camera into a three-dimensional position, but this method is not limited to this. Alternatively, variables f', c', k', and p' can be calculated as parameters based on the gravity direction detected by the gravity direction detection unit 51, and these parameters can be used to correct the position of the image represented by the two-dimensional image coordinate system 73. Then, the corrected position of the image represented by the two-dimensional image coordinate system 73 can be transformed into a three-dimensional position using parameters that exclude the influence of gravity.

[0156] For example, based on variables f', c', k', and p', which are parameters that take into account the influence of gravity, the coordinate values ​​(u1, v1) of any position in a feature portion of a two-dimensional image can be transformed into the corrected coordinate values ​​(u1', v1'). That is, it can be transformed into a two-dimensional image obtained without the influence of gravity. Then, using the parameters of a calculation model that excludes the influence of gravity, the three-dimensional position can be calculated based on the coordinate values ​​(u1', v1') in the image. In this way, image processing can be performed to transform an image acquired by a camera into an image that excludes the influence of gravity. Afterward, based on the position in the image, the three-dimensional position can be calculated using parameters that exclude the influence of gravity. This method also enables high-precision detection of three-dimensional position.

[0157] In the above-described embodiment, robot 1 changes the posture of camera 6, but is not limited to this method. The position and posture of the camera can be fixed. Furthermore, the moving device for changing the posture of the camera is not limited to a robot, and any device that can change the posture of the camera can be used.

[0158] In each of the above-described controls, the order of steps can be appropriately changed without altering the function or effect. The above embodiments can be appropriately combined. In the above figures, the same or equivalent parts are labeled with the same reference numerals. Furthermore, the above-described embodiments are illustrative and do not limit the invention. Additionally, the embodiments include modifications to the embodiments shown in the patent protection scope.

[0159] Explanation of reference numerals in the attached figures

[0160] 1 robot

[0161] 2 Control device

[0162] 3 Robot System

[0163] 6 cameras

[0164] 6b optical axis

[0165] 23 position detectors

[0166] 24 Focus Adjustment Institutions

[0167] 26 Teaching operation panel

[0168] 28 Display Section

[0169] 31 Temperature Detector

[0170] Workpieces 38 and 39

[0171] 38c edge

[0172] 39a recess

[0173] 40 Control device main body

[0174] 42 Storage Unit

[0175] 50 Image Processing Department

[0176] 51 Gravity Direction Detection Unit

[0177] 52 Posture Detection Department

[0178] 53 Parameter Setting Section

[0179] 54 Feature Detection Department

[0180] 55 Feature Position Calculation Unit

[0181] 56 Accelerometer Detection Department

[0182] 63 Setting Information

[0183] Images 66 and 67

[0184] Images of workpieces 68 and 69

[0185] 71 Reference Coordinate System

[0186] 72 camera coordinate system

[0187] 73 Image Coordinate System

[0188] 74. Tool coordinate system.

Claims

1. A photographing apparatus characterized by comprising: have: A visual sensor, which captures images of the object; Gravity direction detection unit, which detects the direction of gravity; An attitude detection unit detects the attitude of the visual sensor relative to the direction of gravity; The parameter setting unit sets parameters for calculating the three-dimensional position corresponding to a specific position in the image captured by the vision sensor. The storage unit stores setting information for setting parameters corresponding to the posture of the vision sensor; A feature detection unit detects predetermined feature portions in an image of the object; and The feature position calculation unit uses parameters set by the parameter setting unit to calculate the three-dimensional position of the feature portion. The parameter setting unit sets parameters based on the direction of gravity, the posture of the visual sensor, and the setting information.

2. The shooting device according to claim 1, characterized in that, When the posture of the vision sensor changes, the parameter setting unit changes the value of the parameter according to the posture of the vision sensor.

3. The shooting device according to claim 1 or 2, characterized in that, The imaging device includes a robot capable of changing the posture of the visual sensor. The posture detection unit detects the posture of the visual sensor relative to the direction of gravity based on the robot's position and posture, as well as the position of the visual sensor in the robot on which the visual sensor is installed.

4. The shooting device according to claim 1 or 2, characterized in that, The imaging device is equipped with a temperature detector to detect temperature. The parameter setting unit sets parameters based on the direction of gravity, the posture of the vision sensor, temperature, and the setting information.

5. The shooting device according to claim 1, characterized in that, The imaging device includes: The robot, which changes the posture of the vision sensor; and The acceleration detection unit detects the acceleration of the robot as it is driven by the vision sensor. The parameter setting unit sets parameters based on the direction of gravity, the acceleration of the visual sensor, the posture of the visual sensor, and the setting information.

6. The shooting device according to claim 1 or 2, characterized in that, The vision sensor has an autofocus function.

7. The shooting device according to claim 1 or 2, characterized in that, The imaging device includes a display unit that displays the three-dimensional position of the feature portion calculated by the feature position calculation unit. The parameter setting unit is configured to set parameters that exclude the influence of gravity. The display unit is configured to display the three-dimensional position of a feature portion calculated using parameters set according to the direction of gravity and the posture of the visual sensor, as well as the three-dimensional position of the feature portion calculated using parameters that exclude the influence of gravity.

8. The shooting device according to claim 1 or 2, characterized in that, The shooting device includes a display unit that displays the values ​​of parameters set by the parameter setting unit.

Images