Imaging device and robot

Abstract

To provide an imaging apparatus capable of highly accurately measuring a relative distance to a subject.SOLUTION: The imaging unit includes a first optical system capable of changing a focus position using a variable focus lens, a second optical system having a fixed focus position and a wider angle of view than the first optical system, an imaging element that outputs a captured image obtained by capturing both a subject image by incident light via the first optical system and a subject image by incident light via the second optical system, and a lens drive unit that drives the variable focus lens. The control unit includes a varifocal lens control section that outputs a command value to the lens drive section, a measurement section that measures a focus position of the first optical system, a calculation section that calculates a relative distance from a reference point to the subject on the basis of optical characteristic information of the first optical system and a rate of change of the focus position, and an image generation section that generates and outputs an image in which the relative distance is displayed superimposed on the captured image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device that calculates a relative distance to a subject, and a robot equipped with the imaging device. [Background technology]

[0002] There is a technology that uses multiple optical systems (compound optical systems) to acquire information about the surrounding space of a subject, which is difficult to acquire with conventional general imaging systems. Note that the information about the surrounding space of a subject here is a general term for various information used to express the subject space, such as subject distance information, position information, configuration information, light source information, subject spectral characteristics, and scattering characteristics.

[0003] In particular, the distance measuring camera of Patent Document 1 is known as a technology for measuring three-dimensional information and distance information of a subject by dividing an optical system. For example, claim 1 of the document describes a distance measuring camera comprising: "a first optical system for collecting light from a subject and forming a first subject image; a second optical system for collecting the light from the subject and forming a second subject image; an imaging unit for capturing the first subject image formed by the first optical system and the second subject image formed by the second optical system; and a distance calculation unit for calculating a distance to the subject based on the first subject image and the second subject image captured by the imaging unit, wherein the distance calculation unit calculates the distance to the subject based on an image magnification ratio between a magnification of the first subject image and a magnification of the second subject image, and is a fixed focus optical system, the second optical system is an autofocus optical system in which at least one of the lenses constituting the second optical system is configured to be drivable, the image magnification ratio between the magnification of the first subject image and the magnification of the second subject image changes depending on the distance to the subject, the first optical system and the second optical system are configured so that the focal length of the first optical system and the focal length of the second optical system are different from each other, thereby making the change in the magnification of the first subject image depending on the distance to the subject different from the change in the magnification of the second subject image depending on the distance to the subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-128518 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the distance measuring camera of Patent Document 1, if the subject is large and the entire subject cannot be captured, it is difficult to calculate the magnification ratio of the subject image, which may result in erroneous distance measurement. Also, with this distance measuring camera, if the change in image magnification due to differences in focal length is small, the resolution of distance measurement may become coarse, making it difficult to use in situations where high distance measurement accuracy is required.

[0006] Therefore, the present invention aims to provide an imaging device that can measure the relative distance to a subject with high accuracy even in situations where the subject is too large to capture the entire subject or where high distance measurement accuracy is required, and a robot equipped with such an imaging device. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the imaging device of the present invention is an imaging device comprising an imaging unit, a control unit, and a memory unit, wherein the imaging unit comprises a first optical system whose focus position is changeable using a variable-focus lens, a second optical system whose focus position is fixed and whose angle of view is wider than that of the first optical system, an imaging element that outputs a captured image obtained by capturing both an image of a subject formed by incident light passing through the first optical system and an image of a subject formed by incident light passing through the second optical system, and a lens driving unit that drives the variable-focus lens, and the control unit comprises a variable-focus lens control unit that outputs a command value to the lens driving unit, a measurement unit that measures the focus position of the first optical system, a calculation unit that calculates the relative distance from a reference point to a subject based on optical characteristic information of the first optical system and the rate of change of the focus position, and an image generation unit that generates and outputs an image in which the relative distance is superimposed on the captured image. [Effects of the Invention]

[0008] According to the imaging device of the present invention, even in a situation where the object is too large to capture the entire image or a situation where high accuracy in distance measurement is required, it is possible to measure the relative distance to the object with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of an imaging system according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of an imaging unit according to an embodiment. [Figure 3] FIG. 1 is an external view of a mobile manipulator according to an embodiment. [Figure 4] 10 is an example of an image captured of a robot hand equipped with a reference subject according to an embodiment. [Figure 5] 10 is a flowchart of a process for determining optical characteristics of a first optical system performed by the imaging apparatus according to an embodiment. [Figure 6] 10 illustrates an example of determining optical characteristics of a first optical system in an imaging device according to an embodiment. [Figure 7] 1 is an example of an image captured by the imaging device of the embodiment. [Figure 8] 10 is a flowchart of a three-dimensional information acquisition process performed by the imaging device of the embodiment. [Figure 9] FIG. 10 is a diagram showing details of a shape information label according to an embodiment. [Figure 10] FIG. 10 is a diagram showing measurement points of the in-focus position for a captured image in an embodiment. [Figure 11] FIG. 3 is a diagram showing optical characteristics of a first optical system of the imaging device according to the embodiment. [Figure 12] FIG. 2 is a diagram showing information necessary for acquiring three-dimensional information for a captured image in one embodiment. [Figure 13] 10 is an example of an image generated by an image generating unit according to an embodiment. [Figure 14] 10 is another example of an image generated by the image generation unit of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An imaging device 1 according to an embodiment of the present invention will be described in detail below with reference to the drawings.

[0011] 1 is a configuration diagram of an imaging system 100 equipped with an imaging device 1 of this embodiment. The imaging system 100 shown here is a system equipped with the imaging device 1, an output device 2, and an input device 3, and is a system used by, for example, a robot operator or a robot control device when causing a robot 4 to grasp an object 5 to be grasped. Each device will be described below in order.

[0012] <Output device 2> The output device 2 is a device that displays the images etc. output by the imaging device 1, and is specifically a liquid crystal display, an organic EL display etc. The operator can operate the robot 4 while visually checking the images etc. displayed on the output device 2 and checking the relative relationship between the robot 4 and the object 5 to be grasped.

[0013] <Input Device 3> The input device 3 is a device used by the operator to input desired information, such as identifying the object to be grasped 5 from the image displayed on the output device 2, and is specifically a keyboard, a mouse, etc. Note that a touch panel type liquid crystal display or the like that combines the functions of the output device 2 and the input device 3 may also be used as the input / output device.

[0014] <Imaging device 1> The imaging device 1 is a device with a function to calculate the relative distance between a graspable object 5 (referred to as a "subject 51" when focused on as an imaging target) grasped by a robot 4, and includes an imaging unit 10, a control unit 20, and a storage unit 30. Each unit will be explained below in order.

[0015] <<Imaging unit 10>> The imaging unit 10 includes a first optical system 11, a second optical system 12, an imaging element 13, a lens driving section 14, and an image processing section 15.

[0016] The first optical system 11 has a variable-focus lens 11a, a fixed-focus lens 11b, and prisms 11c and 11d. The variable-focus lens 11a is a lens for changing the focal position of the first optical system 11, and is specifically a lens that moves on the optical axis using an external power source, or a liquid lens whose optical characteristics change when its shape changes using an external power source. The variable-focus lens 11a may be composed of multiple lens groups. The fixed-focus lens 11b is a lens that is fixed on the optical axis of the first optical system 11. The fixed-focus lens 11b may be composed of multiple lens groups.

[0017] The second optical system 12 has a fixed-focus lens 12a and prisms 12b and 12c. The fixed-focus lens 12a is a lens that is fixed on the optical axis of the second optical system 12. This fixed-focus lens 12a may be made up of a group of multiple lenses.

[0018] The imaging element 13 is an image sensor installed so as to be able to receive both incident light via the first optical system 11 and incident light via the second optical system 12, and outputs a captured image P. Specifically, the imaging element 13 is a PSD (Position Sensitive Detector) or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor or the like.

[0019] The lens driver 14 is a functional unit that drives the variable-focus lens 11a when changing the focal position of the first optical system 11. For example, if a variable-focus lens 11a that changes the focal position by moving along the optical axis is used, the lens driver 14 is a one-two phase excitation or microstep stepping motor that can control the amount of movement of the variable-focus lens 11a. Alternatively, if a liquid lens is used as the variable-focus lens 11a, the lens driver 14 is a power supply device that applies a desired voltage to the liquid lens.

[0020] The image processing unit 15 is a functional unit that performs image processing to detect a contrast value for the captured image P output by the image sensor 13. As will be described in detail later, the contrast value detected here is used to identify the focus position of the variable-focus lens 11a.

[0021] 2 is a diagram showing an example of the configuration of the imaging unit 10. In the imaging unit 10 shown here, two optical systems (a first optical system 11 and a second optical system 12) are provided for one imaging element 13, and therefore the light receiving area of ​​the imaging element 13 is divided into a light receiving area for the first optical system 11 (the right area in the figure) and a light receiving area for the second optical system 12 (the left area in the figure). Therefore, the captured image P output by the imaging element 13 records two types of images: an image of the subject 51 captured via the first optical system 11, and an image of the subject 51 captured via the second optical system 12. Below, the configuration on the optical axis of each optical system and its function will be described in order.

[0022] First, the operation of first optical system 11 will be described starting from the subject 51 side and proceeding toward image sensor 13. Light incident on first optical system 11 is reflected by the mirror surface of prism 11c and then passes through variable-focus lens 11a and fixed-focus lens 11b to form an image of the subject. The formed image of the subject is reflected by the mirror surface of prism 11d and then received by the region of image sensor 13 on the right side in the drawing.

[0023] Next, the operation of second optical system 12 will be described in order from the subject 51 side to the image sensor 13 side. Light incident on second optical system 12 is reflected by the mirror surface of prism 12b and passes through fixed-focus lens 12a to form a subject image. The formed subject image is reflected by the mirror surface of prism 12c and is received by the region of image sensor 13 on the left side in the drawing.

[0024] <<Control unit 20>> The control unit 20 includes a detection unit 21, a variable-focus lens control unit 22, a measurement unit 23, a calculation unit 24, a three-dimensional information estimation unit 25, and an image generation unit 26. Specifically, the control unit 20 is a computer equipped with an arithmetic device such as a CPU (Central Processing Unit) or an SoC (System on a chip), a storage device such as a semiconductor memory, and a communication device, and the arithmetic device executes a predetermined program to realize each functional unit.

[0025] The detection unit 21 includes an object detection unit 21a and a shape determination unit 21b. The object detection unit 21a detects a subject 51 from within a captured image P captured by the first optical system 11. The shape determination unit 21b determines the shape of the subject 51 detected by the object detection unit 21a based on a feature amount.

[0026] The variable-focus lens control unit 22 transmits a command value to the lens driving unit 14 based on a focus position search method. Here, the focus control method refers to a method and sequence for searching for a position where the first optical system 11 focuses on the subject 51 using the variable-focus lens 11a. For example, when using a focus position search method that searches for a position where the contrast of the subject 51 is highest from among all lens states of the variable-focus lens 11a, the variable-focus lens control unit 22 transmits a command value corresponding to each lens state (step) of the variable-focus lens 11a to the lens driving unit 14 so as to cover all lens states.

[0027] The measurement unit 23 includes a measurement point determination unit 23a and a focus position measurement unit 23b. Based on the shape information determined by the shape determination unit 21b, the measurement point determination unit 23a determines the number of focus position measurement points necessary and sufficient to obtain three-dimensional information, as well as the plane coordinates to be measured. The focus position measurement unit 23b uses the variable-focus lens 11a to measure the focus position of the subject 51. Specifically, the lens driver 14 drives the variable-focus lens 11a, while the image sensor 13 continuously acquires captured images P. The image processor 15 then determines a certain region of the captured image P as a target region and calculates the focus level based on various calculation methods, such as the absolute value of the brightness gradient within the target region, the brightness variance, or edge intensity. The lens position at which the captured image P with a high focus level within the target region is obtained as a result of the calculation is determined to be the focus position. By determining the shape of the subject 51 in advance and determining measurement points according to that shape, the number of measurement points of the plane coordinates for measuring the focusing position can be minimized, and the calculation speed can be increased while maintaining the accuracy of the three-dimensional information acquisition. Note that the brightness gradient is calculated based on the difference between the brightness value of each pixel constituting the image in the target area and the brightness values ​​of one or more of the adjacent pixels, for example, and there are various calculation methods available, any of which can be used as appropriate.

[0028] The calculation unit 24 includes an optical characteristic calculation unit 24a and a distance information calculation unit 24b. The optical characteristic calculation unit 24a combines information about the focus position of the reference subject 6, whose distance from the image sensor 13 is known, with information from the variable-focus lens control unit 22 to determine the amount of change in focal length due to changes in the variable-focus lens 11a. The distance information calculation unit 24b calculates the distance from the reference point to each measurement point using the focus position of the reference subject 6, whose distance from the image sensor 13 is known, the amount of change in focal length due to changes in the variable-focus lens 11a calculated by the optical characteristic calculation unit 24a, the focus position at each measurement point determined by the measurement point determination unit 23a, and the command value of the variable-focus lens control unit 22 at the focus position at each point. The distance calculation by the distance information calculation unit 24b is performed using the amount of change in focal length due to changes in the variable-focus lens 11a determined by the focus position measurement unit 23b.

[0029] The three-dimensional information estimation unit 25 estimates three-dimensional information from the reference point to the subject 51 based on the distance from the reference point to each measurement point calculated by the calculation unit 24, the two-dimensional image of the subject 51 captured by the first optical system 11, and the shape information determined by the shape determination unit 21b. The reference point will be described later.

[0030] The image generation unit 26 uses the three-dimensional information from the reference point to the object estimated by the three-dimensional information estimation unit 25 to generate an image with distance data calculated for each pixel of the two-dimensional image of the subject 51 added, or an image as image data of the subject 51 virtually captured from multiple viewpoints from any location, and outputs the image to the output device 2.

[0031] <<Storage unit 30>> The storage unit 30 stores, in advance, for example, a shape information label 31, reference subject distance information 32, and optical characteristic information 33.

[0032] The shape information label 31 is a label used when the shape determination unit 21b acquires shape information of an object, and is data in which the name and shape information of an object are labeled for an object that is a candidate for the subject 51 (i.e., an object that is a candidate for the object to be grasped 5). Note that when selecting shape information of the object to be grasped 5 from the shape information label 31, the input device 3 can be used.

[0033] The reference subject distance information 32 is information used when the optical characteristics calculation unit 24a calculates the amount of change in focal length due to a change in the variable-focus lens 11a, and is information on the physical distance from the image sensor 13 to the reference subject 6.

[0034] The optical characteristic information 33 stores a function that approximates the relationship between the amount of change in focal length due to changes in the variable-focus lens 11a into a mathematical formula, and is composed of a group of variable parameters. The optical characteristic calculation unit 24a updates the parameters based on the function stored in the optical characteristic information 33 to identify the optical characteristics of the variable-focus lens 11a. The optical characteristic information 33 defined by the identified parameters is used for three-dimensional measurement, which will be described later. Note that, in addition to storing the optical characteristic information in a storage unit, there are other methods, such as inputting the optical characteristic information to an input device or acquiring the focus position of a reference point to generate the optical characteristic information.

[0035] <Use Pattern of Imaging System 100> 3 is a diagram illustrating the appearance of a mobile manipulator for grasping an object as an example of a robot 4 to which the imaging system 100 of this embodiment is applied. This robot 4 includes an arm 41 and a hand 42 as a grasping mechanism. The arm 41 is, for example, a vertical articulated robot arm configured to change the position and orientation of the hand 42 attached to the tip of the arm 41 in three-dimensional space with six degrees of freedom.

[0036] The lower part of FIG. 3 shows an enlarged schematic diagram of the hand 42. As shown here, the hand 42 includes a gripper 42a and a reference object jig 42b. The imaging unit 10 is installed at a position where it can capture an image of the reference object jig 42b. The gripper 42a is a mechanism capable of gripping a container or other object 5 to be grasped. For example, the gripper 42a has two claws and rotates around a fulcrum at the claw attachment point to open and close the gripper 42a. The reference object jig 42b is a detachable jig on which a reference object 6, such as a cross mark, is printed for focus determination. The optical characteristic calculation unit 24a uses the reference object 6 to determine the optical characteristics of the variable-focus lens 11a. In this embodiment, the reference object jig 42b is located at two locations: the tip and the midpoint of the gripper 42a.

[0037] By using a hand 42 configured in this manner, the optical characteristic calculation unit 24a can determine the optical characteristics of the variable-focus lens 11a based on the optical characteristic information 33 and the command value of the variable-focus lens control unit 22 at the focus position of the reference subject 6.

[0038] <<Example of captured image P>> 4 is an example of a captured image P captured when the reference subject jig 42b is attached. In the captured image P illustrated here, a first image P1 captured via the first optical system 11 is positioned on the right, and a second image P2 captured via the second optical system 12 is positioned on the left. Because the second optical system 12 has a shorter focal length than the first optical system 11, the second image P2 captured via the second optical system 12 has a wider angle of view and captures a wider range. On the other hand, because the first optical system 11 has a longer focal length than the second optical system 12, the first image P1 captured via the first optical system 11 has a narrower angle of view and captures a narrower range. Therefore, the first image P1 corresponds to an enlarged image of a partial region of the second image P2, and the resolution per pixel of the first image P1 is higher than that of the second image P2.

[0039] In the captured image P in Figure 4, the gripper 42a and the reference subject 6 on the reference subject jig 42b are captured. When measuring the optical characteristics of the variable-focus lens 11a, the gripper 42a is fixed at a predetermined position. As a result, the depth distance from the imaging unit 10 to each reference subject 6 becomes a known value registered in advance in the reference subject distance information 32. The calculation unit 24 of this embodiment uses the first image P1 captured under such an environment to calculate the focus degree from the edge intensity of each reference subject 6 and measure the focus position.

[0040] <<Flowchart of process for determining optical characteristics of first optical system>> 5 is a flowchart of a process for determining optical characteristic information of the first optical system 11. In each step described below, it is assumed that a captured image P such as the one exemplified in FIG. 4 is appropriately acquired.

[0041] First, in step S11, the variable-focus lens control section 22 of the control unit 20 sets the state of the variable-focus lens 11a to an initial value (initial position if it is a lens that can move on the optical axis, or initial shape if it is a liquid lens).

[0042] Next, in step S12, the object detection section 21a of the control unit 20 detects the reference subject 6, which is the focus target point, from within the first image P1.

[0043] In step S13, the imaging unit 10, and the varifocal lens control unit 22 and focus position measurement unit 23b of the control unit 20 cooperate to measure the focus position of the reference subject 6. Specifically, when the varifocal lens control unit 22 drives the varifocal lens 11a, the image processing unit 15 acquires the lens position at which a first image P1 with the maximum contrast value of the reference subject 6 is obtained. In this step, a command value V1 for the varifocal lens control unit 22 when the focus is on the reference subject 61 at the gripper intermediate position and a command value V2 for the varifocal lens control unit 22 when the focus is on the reference subject 62 at the gripper tip position are acquired. The acquired command values ​​V1 and V2 are stored in the storage unit 30.

[0044] In step S14, the optical characteristic calculation unit 24a corrects the optical characteristics of the first optical system 11 using the reference object distance information 32 and the optical characteristic information 33 in the storage unit 30. Specifically, the optical characteristic calculation unit 24a calculates the amount of change in the focus position due to a change in the command value of the variable-focus lens control unit 22 using information on the depth distance from the first optical system 11 to the reference object 6 stored in the reference object distance information 32 and the command values ​​V1 and V2 of the variable-focus lens control unit 22 when focusing on the reference object 6 obtained in step S13. By calculating the amount of change in the focus position, it becomes possible to obtain with high accuracy changes in the optical characteristics of the lens due to changes in the external environment, such as temperature changes, and changes in the lens over time.

[0045] 6 is a diagram illustrating the optical characteristics of the variable-focus lens 11a in this embodiment. Here, it is assumed that there is a proportional relationship between the command value output by the variable-focus lens control unit 22 to the lens drive unit 14 and the in-focus position of the variable-focus lens 11a. In this case, if the command value of the variable-focus lens control unit 22 is V and the in-focus position of the variable-focus lens 11a is F, the optical characteristic information in the storage unit 30 stores the following (Equation 1) as an equation showing the relationship between the command value V and the in-focus position F.

[0046] F=α×V+β (Formula 1) Here, α indicates the amount of change in the focal position of the variable-focus lens 11a due to a change in the command value of the variable-focus lens control unit 22, and β indicates the focal position when the command value of the variable-focus lens control unit 22 is the initial value. Note that these are parameters that change due to the effects of temperature rise and aging, and therefore require correction of the parameters at appropriate times.

[0047] When the in-focus position of the reference subject 61 at the gripper intermediate position in the first image P1 captured by the imaging unit 10 is F1 and the in-focus position of the reference subject 62 at the gripper tip position is F2, α and β are expressed by (Equation 2) and (Equation 3).

[0048] α=(F2−F1) / (V2−V1) (Formula 2) β=F1-(F2-F1) / (V2-V1)×V1 (Formula 3) Therefore, by using (Equation 2) and (Equation 3) and having the optical characteristic calculation unit 24a appropriately update the variable parameters α and β, the accuracy of the optical characteristic information of the variable-focus lens 11a can be ensured even if changes occur over time.

[0049] <<Another example of captured image P>> FIG. 7 shows another example of a captured image P of the gripper 42a and the object to be grasped 5 captured by the imaging device 1 of this embodiment. In this figure, the first image P1 is an image captured by focusing on the central region R of the second image P2. Note that the first image P1 and the second image P2 are originally captured simultaneously, but for convenience of explanation, FIG. 7 shows the first image P1, which was captured after the second image P2. In this example, the object detection unit 21a detects the cylindrical object to be grasped 5 from the group of objects in the second image P2. Then, based on the position of the object to be grasped 5 in the second image P2, the robot 4 is driven using inverse kinematics to move the gripper 42a so that the object to be grasped 5 falls within the range of the first image P1. After the object to be grasped 5 is captured in the first image P1, the shape determination unit 21b determines the object shape of the object to be grasped 5. In this example, the shape determination unit 21b determines the object to be grasped 5 to be cylindrical.

[0050] <<Flowchart of process for determining optical characteristics of first optical system>> Fig. 8 is a flowchart of a process for acquiring three-dimensional information of the detected grasp target 5. Note that in each step described below, it is assumed that a captured image P such as that shown in Fig. 7 has been captured as appropriate.

[0051] First, in step S21, the object detection unit 21a of the control unit 20 detects the subject 51 in the first image P1. In the example of Fig. 7, the object detection unit 21a detects a spherical object, a rectangular parallelepiped object, and a cone-shaped object as the subject 51 in addition to the cylindrical object that is the grasp target 5.

[0052] Next, in step S22, the shape determination unit 21b uses the shape information label 31 in the storage unit 30 to determine the shape of the desired grasp object 5 included in the subject 51 detected in step S21.

[0053] FIG. 9 shows details of the shape information label 31 of this embodiment. As shown here, the shape information label 31 of this embodiment provides four shapes as candidates for the shape of the grasped object 5: label number #1 "sphere," label number #2 "rectangular prism," label number #3 "cone," and label number #4 "cylinder." When an operator wants the robot 4 to grasp a cylindrical grasped object 5, the operator operates the input device 3 to select #4, which is the label number corresponding to the desired grasped object 5, from the table of shape information labels 31 displayed on the output device 2. By inputting this selection result, the shape determination unit 21b can determine the shape of the grasped object 5 that the operator wants the robot 4 to grasp.

[0054] In step S23, the measurement point determining unit 23a determines the number and coordinates of the focal positions to be measured based on the shape information of the grasped object 5 determined in step S22.

[0055] 10 is a diagram illustrating measurement points of the focus positions set on the grasping target 5 in the first image P1 in this step. In step S22, since the grasping target 5 is determined to have a cylindrical shape, the measurement point determination unit 23a determines three points A1, A2, and A3, which are predefined as measurement points for acquiring three-dimensional information about the cylinder, as measurement points.

[0056] In step S24, the imaging unit 10, the variable-focus lens control unit 22 of the control unit 20, and the focus position measurement unit 23b of the measurement unit 23 cooperate to measure the focus positions of the measurement points A1, A2, and A3 determined in step S23. Specifically, when the variable-focus lens control unit 22 drives the variable-focus lens 11a, command values ​​VA1, VA2, and VA3 of the variable-focus lens control unit 22 are obtained when a first image P1 with the maximum contrast value at each measurement point is obtained.

[0057] In step S25, the distance information calculation unit 24b calculates the distance from the tip position of the gripper 42a, which is the reference point, to each measurement point using the command values ​​VA1, VA2, and VA3 acquired in step S24 and the optical property information 33 in the memory unit 30.

[0058] 11 shows an example of a distance information acquisition method using optical characteristic information 33. In this example, distance information calculation unit 24b calculates focal positions FA1, FA2, and FA3 corresponding to each command value VA1, VA2, and VA3 of variable-focus lens control unit 22 based on the relational expression (Equation 1) between the amount of change in the command value of variable-focus lens control unit 22 and the amount of change in the focal position, which is stored in optical characteristic information 33. By calculating the difference between focal position F2 of the tip position of gripper 42a, which is measured in advance using reference object jig 42b according to the procedure shown in the flowchart of FIG. 5, and focal positions FA1, FA2, and FA3 at each measurement point on gripping target object 5, which are calculated in step S24, distance information calculation unit 24b can acquire distance information D1, D2, and D3 from the tip position of gripper 42a to each measurement point.

[0059] In step S26, the three-dimensional information estimation unit 25 estimates three-dimensional information of the object to be grasped 5 based on the two-dimensional information measured from the first image P1, the distance information D1, D2, and D3 from the reference point to each measurement point measured in step S25, and the shape information of the object to be grasped 5 determined by the shape determination unit 21b.

[0060] FIG. 12 is a diagram showing how three-dimensional information is estimated using the first image P1 via the first optical system 11. In this example, the two-dimensional information acquired includes the length L1 of the upper surface of the object to be grasped 5, the length L2 of the lower surface, and the height H from the upper surface to the lower surface. Distance information D1, D2, and D3 from the reference point to each measurement point A1, A2, and A3 indicates the distance from the tip position of the gripper 42a measured in step S25 to each measurement point. Shape information about the object to be grasped 5 is cylindrical shape information acquired by the shape determination unit 21b. Based on this information, three-dimensional relative position information between the gripper 42a and the object to be grasped 5 is estimated from the first image P1. This completes the three-dimensional information estimation process.

[0061] <<Example of an image generated by the image generating unit 26>> 13 shows an example of an image generated by the image generation unit 26 based on the three-dimensional relative position information of the object to be grasped 5 estimated by the three-dimensional information estimation unit 25 and displayed on the output device 2. In this example, the shortest distance from the tip of the gripper to the object to be grasped 5 is displayed as an X mark superimposed on the first image P1 captured by the imaging device 1, and "10.0 cm", which is the depth distance from the tip of the gripper to the X mark, is displayed as a superimposed display in the upper right corner of the first image P1. By displaying the depth distance superimposed, the operator can get a sense of the distance between the gripper 42a and the object to be grasped 5.

[0062] In this embodiment, the image generation unit 26 generates an image by adding information to the first image P1 about the position of the object to be grasped 5 that is closest to the gripper tip position and the depth distance from the gripper tip position to the X mark. However, it is also possible to generate an image from a third-person perspective using the three-dimensional relative position information of the object to be grasped 5 estimated by the three-dimensional information estimation unit 25.

[0063] FIG. 14 shows an example of a third-person perspective image P3 generated by the image generation unit 26 and displayed on the output device 2. The right half of the displayed image in this figure is the first image P1. Meanwhile, the left half is a third-person perspective image P3 in which the gripper 42a and the object to be grasped 5 are observed from an upper viewpoint, generated by the image generation unit 26 based on the estimation results of the relative positions of the gripper 42a and the object to be grasped 5 in the depth direction and the horizontal direction, based on the three-dimensional information estimated by the three-dimensional information estimation unit 25. Note that while FIG. 14 illustrates a third-person perspective image P3 in which the object to be grasped 5 is observed from an upper viewpoint, the image generation unit 26 can generate a third-person perspective image P3 in which the object to be grasped 5 is observed from any viewpoint. This allows the operator to grasp the spatial coordinates of the gripper 42a and the object to be grasped 5 from various viewpoints.

[0064] Furthermore, in this embodiment, the operator selected the shape of the object from the shape information label 31 using the input device 3, but it is also possible to prepare a shape image for object recognition within the shape information label 31, measure the degree of matching with the captured image P in the shape determination unit 21b, and determine the label number with the highest degree of matching as the shape information, thereby estimating three-dimensional information without the operator having to perform any operation.

[0065] According to the above embodiment, by calculating the focal position of a reference point whose physical relative distance to the imaging element is known, correction is possible based on changes in the optical characteristics of the first optical system due to changes over time, such as temperature changes. The corrected optical characteristics are used to measure the focal position at a measurement point corresponding to the shape of the subject 51, and distance information from the reference point is acquired. Three-dimensional information from the imaging device to the subject 51 is calculated using the distance information from the acquired reference point to each measurement point on the subject 51, shape information on the subject 51, and two-dimensional information of the subject 51 captured by the first optical system. Based on the calculated three-dimensional information, an image is generated in which calculated distance data is added to each pixel of the two-dimensional image of the subject 51, or an image is generated as image data of the subject 51 virtually captured from multiple viewpoints from any location, and the image is output to the output device 2. In this way, correcting the optical characteristics of the first optical system 11 based on the focal position of a reference point whose physical relative distance to the imaging element is known reduces deterioration in the accuracy of three-dimensional measurement due to changes over time, enabling highly accurate three-dimensional information to be acquired. Furthermore, by generating an image based on the calculated three-dimensional information, in which calculated distance data is added to each pixel of the two-dimensional image of the subject 51, or by generating an image as image data of the subject 51 virtually captured from multiple viewpoints from any location, the operator can easily visually recognize the positional relationship from the imaging device to the subject 51.

[0066] In particular, by attaching the imaging unit 10 to the tip of the arm of an industrial robot, it is possible to improve the recognition of the surrounding environment during remote control, etc. This leads to improved operability of the remote-controlled robot, etc. [Explanation of symbols]

[0067] 100...imaging system, 1...imaging device, 10...imaging unit, 11...first optical system, 11a...variable focus lens, 11b...Fixed focus lens, 11c, 11d...Prism, 12...Second optical system, 12a...Fixed focus lens, 12b, 12c...Prism, 13...imaging element, 14...Lens drive unit, 15...Video processing unit, 20...control unit, 21...detection unit, 21a...object detection unit, 21b...shape determination section, 22...variable focus lens control unit, 23...Measurement section, 23a...Measurement point determination section, 23b...Focus position measurement unit, 24...Calculation unit, 24a...optical property calculation section, 24b...Distance information calculation unit, 25...Three-dimensional information estimation section, 26...image generation unit, 30...storage units, 31...shape information label, 32...Reference subject distance information, 33...Optical property information, 2...output device, 3...input device, 4...Robot, 41...Arm, 42...Hand, 42a... Gripper, 42b...Reference object jig, 5...Gripped object 51...Subject, 6…Reference subject

Claims

1. An imaging device including an imaging unit and a control unit, The imaging unit a first optical system that is capable of changing a focus position using a variable focus lens; a second optical system having a fixed focal position and a wider angle of view than the first optical system; an imaging element that outputs a captured image obtained by capturing both a subject image formed by incident light passing through the first optical system and a subject image formed by incident light passing through the second optical system; a lens driving unit that drives the variable-focus lens, The control unit a variable-focus lens control unit that outputs a command value to the lens driving unit; a measurement unit that measures a focus position of the first optical system; a calculation unit that calculates a relative distance from a reference point to an object based on the optical characteristic information of the first optical system and a rate of change of a focus position; an image generating unit that generates and outputs an image in which the relative distance is superimposed on the captured image; An imaging device characterized by:

2. 2. The imaging device according to claim 1, the optical characteristic information is information that mathematically expresses a relationship between the command value and a focus position of the first optical system, The imaging device, wherein the calculation unit calculates the relative distance based on the optical characteristic information and a rate of change of the command value.

3. 2. The imaging device according to claim 1, An imaging device characterized in that the calculation unit updates optical characteristic information based on a command value when the first optical system is focused on a reference subject whose physical distance from the imaging element is known.

4. 2. The imaging device according to claim 1, The imaging device, wherein the calculation unit includes an optical characteristic calculation unit that updates the optical characteristic information based on the command value when the first optical system is focused at the reference point.

5. 2. The imaging device according to claim 1, An imaging device comprising: a storage unit for storing the optical characteristic information.

6. 6. The imaging device according to claim 5, the storage unit has a shape information label that labels shape information of the subject; an imaging device comprising: a shape determination unit that determines the shape of the subject based on the shape information label and feature amounts of the subject;

7. 7. The imaging device according to claim 6, an imaging device comprising: a measurement point determination unit that determines the number of focal positions to be measured and coordinates according to the shape of the subject determined by the shape determination unit;

8. 8. The imaging device according to claim 7, The imaging device is characterized in that the calculation unit includes a distance information calculation unit that calculates a depth distance from the reference point to the coordinate based on the command value for focusing at the coordinate determined by the measurement point determination unit.

9. 9. The imaging device according to claim 8, depth distance information from the reference point to the coordinates calculated by the distance information calculation unit; two-dimensional information of the subject captured by the first optical system; Based on the shape of the subject determined by the shape determination unit, An imaging device comprising: a three-dimensional information estimation unit that estimates three-dimensional relative position information from the reference point to the subject.

10. 10. The imaging device according to claim 9, an image generation unit that generates an image from an arbitrary viewpoint other than an image captured by the first optical system, including the reference point and the subject, based on three-dimensional relative position information from the reference point to the subject estimated by the three-dimensional information estimation unit.

11. 2. The imaging device according to claim 1, An imaging device connected to an input device for an operator to operate or select the image generated by the image generation unit.

12. A robot linked to the imaging device according to any one of claims 1 to 11, The robot is a manipulator equipped with an arm and a hand, the reference point is the tip of the hand, The robot is characterized in that the imaging unit is installed on the arm so as to be able to capture an image of the hand.

Images