Image processing device, vision system, and image processing program

Abstract

An image processing device 3 comprises an aperture control unit 311 that obtains, on the basis of image information from an image capture device 1 that has a lens and captures an image of an object 5, spatial position information of the object and controls the aperture of the lens. When the distance from the image capture device to the object is a first distance, the image capture device captures an image of the object to obtain a first image with the focus position for the lens adjusted and fixed, and with the aperture of the lens set to a first aperture by the aperture control unit. When the distance from the image capture device to the object is a second distance, the image capture device captures an image of the object to obtain a second image with the aperture of the lens set to a second aperture by the aperture control unit.

Description

Image processing device, vision system, and image processing program

[0001] The embodiments described in this application relate to an image processing device, a vision system, and an image processing program.

[0002] In recent years, vision systems (machine vision, robot vision) have been applied to industrial machinery, industrial robots, and collaborative robots, and the industrial machinery, etc. are made to perform predetermined operations based on image information acquired by the vision system.

[0003] For example, in an industrial robot to which robot vision is applied, the spatial position information of the workpiece is determined based on image information captured by an image capturing device (camera) of the target object (workpiece), and the industrial robot is controlled to perform a specified process on the workpiece.

[0004] For the sake of simplicity, this specification will be described using an industrial robot vision system as an example of a vision system to which the image processing device is applied. However, the image processing device, vision system, and image processing program according to this embodiment are not limited to industrial robots and robot vision systems, and can be widely applied to a variety of machines, robots, etc.

[0005] In addition, in this specification, the term vision system means a system that operates equipment based on the capture and processing of images, and includes all industrial and non-industrial applications that combine hardware and software.

[0006] Conventionally, various proposals have been made as control techniques for robot vision of industrial robots or cameras used in vision systems.

[0007] JP 2019-113680 A JP 2019-161553 A JP 2006-253998 A

[0008] As mentioned above, for example, in an industrial robot to which robot vision is applied, spatial position information is obtained from image information of a workpiece captured by a camera, and the industrial robot is controlled to perform a predetermined process on the workpiece.

[0009] As the camera for such robot vision, a fixed-focus camera is generally used, but recently, cameras with a focusing function (for example, autofocus cameras) have also come into use.

[0010] Incidentally, robot vision using a fixed-focus camera does not require adjustment of the focus position, so there is no need to change the parameters used in the conversion formula that converts the image information of the workpiece into spatial position information.

[0011] In contrast, robot vision using a camera with a focus function can capture clear images by using a focus position based on the camera's position relative to the workpiece (the distance between the workpiece and the camera). However, for example, when the focus position is adjusted, it becomes necessary to change the parameters used in the conversion formula that converts the image information of the workpiece into spatial position information each time.

[0012] In other words, when the focus position (focal length) of the lens changes, the parameters used in the conversion formula must be changed, so it is required to obtain in advance the parameters corresponding to the focus position when capturing an image each time.

[0013] As a result, there is a risk that the processing for capturing an image of a workpiece (object) will become complicated, or that the processing for acquiring the spatial position of the object will take a long time. Furthermore, since the focusing of the lens in the camera (image capturing device) is performed, for example, by a drive unit that changes the positional relationship of the lenses, errors are likely to occur in the lens position, which causes a problem of reduced accuracy in converting image information into spatial position information of the object.

[0014] The problem to be solved by the present invention is to provide an image processing device, a vision system, and an image processing program that can perform image processing to obtain spatial position information of an object based on image information in a short time with high accuracy.

[0015] According to one embodiment of the present invention, there is provided an image processing device that acquires spatial position information of an object based on image information from an image capturing device having a lens that captures an image of the object. The image processing device includes an aperture control unit that controls an aperture of the lens. When the distance of the image capturing device to the object is a first distance, the focus position of the lens is adjusted and fixed, and the aperture control unit sets the lens aperture to a first aperture, and the image capturing device captures an image of the object to acquire a first image. When the distance of the image capturing device to the object is a second distance, the aperture control unit sets the lens aperture to a second aperture, and the image capturing device captures an image of the object to acquire a second image.

[0016] The objects and advantages of the invention will be realized and obtained by means of the elements and combinations particularly pointed out in the claims. Both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

[0017] FIG. 1 is a diagram schematically illustrating an industrial robot system as an example to which an image processing device according to this embodiment is applied. FIG. 2 is a diagram illustrating parameters of a conversion equation used to acquire spatial position information of an object based on image information from the image capturing device shown in FIG. 1. FIG. 3 is a diagram illustrating the relationship between the lens aperture and the depth of field in the industrial robot system shown in FIG. 1. FIG. 4 is a diagram illustrating an example of an illumination device used in the industrial robot system shown in FIG. 1. FIG. 5 is a block diagram illustrating a vision system to which an example of an image processing device according to this embodiment is applied. FIG. 6 is a flowchart illustrating an example of processing of an image processing program executed by a processing unit (arithmetic processing device) of the image processing device shown in FIG. 5.

[0018] First, before describing in detail examples of the image processing device, vision system, and image processing program according to this embodiment, an industrial robot system as an example to which the image processing device according to this embodiment is applied, and an overview of this embodiment will be described with reference to Figures 1 to 4. Figure 1 is a diagram schematically showing an industrial robot system as an example to which the image processing device according to this embodiment is applied.

[0019] 1, reference numeral 100 denotes an industrial robot, 110 denotes a hand unit, 200 denotes a robot control device, and 3 denotes an image processing device. Reference numerals 1, 2, 4, and 5 denote an image capturing device (camera), an illumination device (ring illumination), a work table, and an object (workpiece). A robot vision system (vision system) is configured to include, for example, the camera 1, the illumination device 2, and the image processing device 3.

[0020] 1 shows a case where a hand unit 110 of an industrial robot 100 is moved along a preset teaching route and the hand unit 110 performs an operation of grasping a workpiece 5. However, in FIG. 1, the position of the workpiece 5 placed on the workbench 4 is shifted from its original position 5a (position based on the teaching route) indicated by a dashed line.

[0021] 1 shows a case where the hand unit 110 is moved along a preset teaching route to grasp a workpiece 5 on a workbench 4, but the actual spatial position of the workpiece 5 on the workbench 4 is deviated from the intended original position 5a. Here, the camera 1 and the lighting device 2 are attached near the hand unit 110 of the industrial robot 100 and are able to move together with the hand unit 110 to capture an image of the workpiece 5 placed on the workbench 4.

[0022] At this time, the image processing device 3 acquires the actual spatial position information of the workpiece 5 based on the image information of the workpiece 5 on the workbench 4 captured by the camera 1, and recognizes the deviation from the assumed position 5a. Then, the image processing device 3 recognizes and corrects the deviation between the assumed position 5a of the workpiece 5 and the actual spatial position via the robot control device 200, and controls the hand unit 110 so that the workpiece 5 is correctly grasped.

[0023] In this way, by applying robot vision to an industrial robot system, even if the position of the workpiece 5 on the work table 4 is misaligned, the misalignment can be corrected to allow the industrial robot 100 to perform the correct operation. In Figure 1, the image processing device 3 is built into the robot control device 200, but it can also be provided outside the robot control device 200 as a separate unit.

[0024] 1, the workpiece 5 is placed on a workbench 4, and the camera 1 is attached to and moves near a hand unit 110 (actuator) of the industrial robot 100. The image processing device 3 according to this embodiment can be widely applied to various machines and robots in which at least one of the camera 1 and the workpiece 5 is held by an actuator whose movement can be controlled.

[0025] Fig. 2 is a diagram for explaining parameters of a transformation equation used to acquire spatial position information of an object based on image information from the image capturing device shown in Fig. 1. In Fig. 2, equation (I) is a relational equation between a position in space and a position on an image obtained by projecting a three-dimensional coordinate point in a so-called pinhole camera model onto an image plane using perspective projection transformation. Furthermore, equation (II) is an expanded relational equation that takes into account distortions in the radial and circumferential directions of the lens of the actual camera 1.

[0026] In formula (I), reference symbol AA represents the position on the image (the position of the workpiece 5 in the image captured by camera 1), BB represents the focal length and image center of camera 1, CC represents the camera coordinates in space, and DD represents the position in space (the position of the workpiece 5 in actual space). Note that (X, Y, Z) represent the three-dimensional coordinates in the world coordinate system, (u, v) represent the coordinates of the point projected onto the image plane, (cx, cy) represent the principal point (usually the image center), and fx, fy represent the focal length expressed in pixel units.

[0027] In equation (II), t represents [t1, t2, t3], R represents a 3x3 r matrix, and k1 to k6, p1, and p2 represent parameters. Therefore, the actual spatial position of the workpiece 5 can be obtained by converting the position of the workpiece 5 in the image captured by camera 1 based on equation (II). At this time, by preparing in advance the spatial camera coordinates CC, the focal length and image center BB, and the distortion parameters (k1 to k6 and p1, p2: six types of k and two types of p), it becomes possible to obtain the spatial position DD from the position AA on the image.

[0028] Here, when the focus position of the lens is changed, the focal length f changes each time, and it becomes necessary to obtain in advance the parameters (six types of k and two types of p) that correspond to that focal length f. In other words, in a vision system that uses a camera with a focus function, when the focus position (focal length) of the lens changes, it is necessary to change the parameters used in the conversion formula.

[0029] In this way, when a camera with a focus function is used, it is necessary to acquire in advance parameters corresponding to the focus position when capturing an image, which may lead to problems such as complicated processing when capturing an image of a workpiece or prolonged processing time. Furthermore, since the lens is focused by a drive unit that changes the lens position, errors are likely to occur in the lens position, which may result in a decrease in the accuracy of converting image information into spatial position information of the workpiece.

[0030] Figure 3 is a diagram for explaining the relationship between the lens aperture and depth of field in the industrial robot system shown in Figure 1, and is intended to explain the relationship between the aperture (aperture value, aperture amount) and depth of field in the lens of camera 1. In Figure 3, reference symbol L indicates the distance to the subject (workpiece 5), δ indicates the permissible circle of confusion (a constant determined by the resolution), f indicates the focal length, and F indicates the aperture amount (aperture value).

[0031] The depth of field D is the range in which the image is in focus, and is the sum of the front depth of field Df and the rear depth of field Dr (D = Df + Dr), where Df and Dr can be expressed as follows: Df = (δFL 2 ) / (f 2 +δFL) Dr=(δFL 2 ) / (f 2 -δFL)

[0032] Here, when f, δ, and L are constant, if the aperture value F is increased, the depth of field D becomes larger (deeper), but the amount of light incident on the image sensor (CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) image sensor) decreases. Specifically, the amount of light incident on the image sensor is expressed as C / F, where F is the aperture value and C is a constant. 2 Therefore, for example, if the aperture amount F is doubled, the amount of light incident on the image sensor becomes 1 / 4.

[0033] One method for adjusting the brightness of an image is to adjust the exposure time of the image, and doubling the exposure time will result in an image that is twice as bright. Therefore, it is conceivable to increase the aperture value F of the lens to deepen the depth of field D of the lens, i.e., to widen the range over which the lens is in focus (focus range), and to compensate for the decrease in the amount of incident light due to the increase in aperture value F by increasing the exposure time.

[0034] Specifically, for example, if the lens aperture F is doubled to deepen the lens depth of field D, it is possible to obtain an image of equivalent brightness by increasing the exposure time by four times. In other words, the exposure time t is t = t0 × (F / F0) 2 The effect of changing the aperture value F can be compensated for by adjusting the exposure time, where t0 is the exposure time before changing F, F0 is the aperture value before changing F, and F is the aperture value after changing it.

[0035] 5 and 6, in an example of the image processing device according to this embodiment, when the distance of the camera 1 to the workpiece 5 is a first distance, the focus control unit 312 adjusts and fixes the focus position of the lens. Then, the aperture control unit 311 sets the lens aperture to the first aperture, and the workpiece 5 is imaged to obtain a first image.

[0036] Furthermore, when the distance of the camera 1 from the workpiece 5 is the second distance, the focus position of the lens is fixed as it is, and the aperture of the lens is set to the second aperture by the aperture control unit 311 so that the workpiece 5 is included in the depth of field (D = Df + Dr) of the lens set to the second aperture. In other words, when the workpiece 5 and the camera 1 are at the second distance, the workpiece 5 is imaged and a second image is acquired so that the workpiece 5 is focused (in focus) by the depth of field of the lens adjusted as the second aperture.

[0037] In this way, according to the example of the image processing device of this embodiment, for example, the focus position of the lens is fixed at both the first distance and the second distance, which are different distances between the camera 1 and the workpiece 5, and the lens aperture is adjusted at the second distance. This makes it possible to acquire the first image and the second image focused on the workpiece 5 at both the first distance and the second distance, which are different distances between the camera 1 and the workpiece 5.

[0038] In this way, when a lens with a focusing function is used, changing the focus position requires changing parameters according to the focus position. However, because the focus position of the lens is fixed at the same position at both the first distance and the second distance, changing parameters is not necessary. In other words, the same parameters can be used at both the first distance for capturing the first image and the second distance for capturing the second image, and there is no need to prepare new parameters. As a result, according to the example of the image processing device, vision system, and image processing program of this embodiment, it is possible to perform image processing to acquire spatial position information of the workpiece 5 based on image information in a short time with high accuracy.

[0039] 4A and 4B are diagrams showing examples of lighting devices used in the industrial robot system shown in FIG. 1, where FIG. 4A shows a case where a circular light 2a is applied as the lighting device 2, and FIG. 4B shows a case where a ring light 2b is applied as the lighting device 2.

[0040] As shown in FIG. 4(a), the lighting device 2 may be, for example, a circular lighting device 2a provided near the camera 1, or, as shown in FIG. 4(b), a ring lighting device 2b provided on the outer periphery of the lens of the camera 1 may be used.

[0041] Here, the circular light 2 a and the ring light 2 b may be configured to constantly emit a constant amount of illumination light, or, as will be described in detail later, may be configured to control the intensity of the illumination light. That is, when the distance between the workpiece 5 and the camera 1 is the second distance, the illumination device 2 (2 a, 2 b) may increase the intensity of the illumination light to compensate for the reduced exposure amount caused by increasing the aperture (aperture value) to focus on the workpiece 5. Specifically, for example, if the lens aperture F is doubled to deepen the depth of field of the lens, the illumination control unit 313 can adjust the intensity of the illumination light from the illumination device 2 to four times, thereby enabling the capture of an image of equivalent brightness.

[0042] It should be noted that the lighting device 2 is not limited to the circular lighting device 2a and the ring lighting device 2b, and various lighting devices can be applied, and it goes without saying that the number and arrangement of the lighting device 2 are not limited to providing only one lighting device near the camera 1.

[0043] Here, the workpiece 5 is illuminated by the light from the lighting device 2 (circular light 2a and ring light 2b). The brightness of the light received by the camera 1 from the workpiece 5 is expressed as C / r, where r is the distance from the camera 1 to the workpiece 5 and C is a constant. 2 Therefore, as mentioned above, if the lens aperture (F) is doubled, the amount of light incident on the camera 1 becomes 1 / 4.

[0044] Furthermore, the relationship with the aperture mentioned above (t = t0 × (F / F0) 2 ), the exposure time t is t = t0 × (F / F0) 2 ×(r / r0)2 Here, t0 is the exposure time before the lens aperture is changed, F0 is the lens aperture before the lens aperture is changed, F is the lens aperture after the lens aperture is changed, r is the distance before the lens aperture is changed, and r0 is the distance after the lens aperture is changed. In other words, there are two methods for compensating for exposure: adjusting the exposure time of the image and adjusting the intensity of the light emitted by the lighting device.

[0045] Hereinafter, examples of an image processing device, a vision system, and an image processing program according to the present embodiment will be described in detail with reference to the accompanying drawings.

[0046] Fig. 5 is a block diagram showing a vision system to which an example of the image processing device according to this embodiment is applied. Here, the vision system to which the example of the image processing device according to this embodiment is applied corresponds to, for example, the robot vision of the industrial robot system described with reference to Fig. 1. As mentioned above, the image processing device 3 can be built into the robot control device 200 that controls the industrial robot 100, but can also be provided as a separate unit outside the robot control device 200.

[0047] 5, at least one of the image capturing device (camera) 1 and the target object (workpiece) 5 is held by an actuator whose movement can be controlled, and the position of the camera 1 relative to the workpiece 5 is controlled by the actuator. That is, the image processing device 3 according to this embodiment can be widely applied to various machines and robots in which at least one of the camera 1 and the workpiece 5 is held by an actuator whose movement can be controlled.

[0048] 5, the image processing device 3 is for acquiring spatial position information of the workpiece 5 based on image information from a camera 1 having a lens (not shown) that captures an image of the workpiece 5, and includes a processing unit (arithmetic processing device) 31 and a storage unit 32. Reference numeral 2 denotes an illumination device for irradiating light onto the workpiece 5. In this embodiment, the illumination device 2 is controlled by the image processing device 3 as described below, but may also be a device that constantly irradiates a constant amount of light onto the workpiece 5, for example.

[0049] The processing unit 31 includes an aperture control unit 311, a focus control unit 312, an illumination control unit 313, a feature extraction unit 314, a position calculation unit 315, and an exposure control unit 316. The aperture control unit 311 controls the aperture of the lens of the camera 1 that captures the image of the workpiece 5, and the focus control unit 312 controls the focus of the lens of the camera 1. The illumination control unit 313 controls the light intensity of the illumination device 2, i.e., the intensity of the light emitted from the illumination device 2 that illuminates the workpiece 5.

[0050] The storage unit 32 stores an image processing program 320 to be executed by the processing unit 31. The image processing program 320 includes a feature storage unit (feature information) 321, a parameter storage unit (parameter information) 322, and an aperture storage unit (aperture information) 323. The feature storage unit 321 stores, for example, feature information of the workpiece 5 to be detected from image information of the workpiece 5 captured by the camera 1. The parameter storage unit 322 stores, for example, focus position information (focus information) and parameter information used in a conversion formula for acquiring the spatial position of the workpiece 5 from the image information. The aperture storage unit 323 stores, for example, the relationship (aperture information) between the distance between the workpiece 5 and the camera 1 (distance of the camera 1 from the workpiece 5) and the depth of field due to the aperture amount (aperture value).

[0051] The feature extraction unit 314 extracts feature information stored in the feature storage unit 321 from the image information of the workpiece 5 captured by the camera 1. The position calculation unit 315 calculates the spatial position of the workpiece 5 based on the image information of the workpiece 5 captured by the camera 1 and parameters acquired from the parameter storage unit 322.

[0052] Here, for example, when the workpiece 5 and the camera 1 are at a first distance, the position calculation unit 315 reads and acquires from the parameter storage unit 322 parameters corresponding to the lens focus position adjusted and fixed by the focus control unit 312. Then, the position calculation unit 315 calculates the spatial position of the workpiece 5 from the first image captured at the first distance by the aperture control unit 311 with the lens aperture set to the first aperture.

[0053] Here, the parameters acquired when the workpiece 5 and camera 1 are at the first distance are also used when the workpiece 5 and camera 1 are at the second distance. That is, when the workpiece 5 and camera 1 are at the second distance, the position calculation unit 315 uses the same parameters as when the workpiece 5 and camera 1 are at the first distance, for example, and calculates the spatial position of the workpiece 5 from the second image captured by the aperture control unit 311 with the lens aperture set to the second aperture. At this time, the aperture control unit 311 controls the aperture amount of the camera 1 by referring to the aperture information stored in the aperture memory unit 323.

[0054] Specifically, when the hand unit 110 is moved from a distance to a closer position relative to the workpiece 5 on the work table 4, the aperture control unit 311 increases the aperture amount and deepens the depth of field so that the workpiece 5 at the second distance is also in focus at the focus position where it was in focus at the first distance. In other words, when the first distance is greater than the second distance, the aperture control unit 311 controls the lens aperture amount to be increased so that the workpiece and the camera are in focus on the workpiece 5 even at the second distance at the focus position that was fixed when the workpiece and the camera were at the first distance, and enter the depth of field.

[0055] The exposure control unit 316 adjusts the exposure time (shutter speed) of the camera 1, and when the workpiece 5 and the camera 1 are at the second distance, compensates for the reduction in exposure corresponding to the increased aperture amount in order to deepen the depth of field and focus on the workpiece 5. Specifically, when the workpiece 5 and the camera 1 are at the second distance and an image is captured with the aperture amount narrowed to twice that when the workpiece 5 and the camera 1 are at the first distance, the exposure control unit 316 controls the exposure time to be four times that when the first image is captured.

[0056] The lighting control unit 313 adjusts the intensity (amount of light) of light emitted from the lighting device 2, and when the workpiece 5 and the camera 1 are at the second distance, compensates for the reduction in exposure corresponding to the increased aperture amount in order to deepen the depth of field and focus on the workpiece 5. Specifically, when the workpiece 5 and the camera 1 are at the second distance and an image is captured with the aperture amount narrowed to twice that when the workpiece 5 and the camera 1 are at the first distance, the lighting control unit 313 controls the intensity of the light emitted from the lighting device 2 to be four times that when the first image is captured.

[0057] Here, if there is a problem with increasing the exposure time, it is preferable that the exposure control unit 316 does not increase the exposure time of the camera 1, but that the illumination control unit 313 increases the intensity of the light irradiated from the illumination device 2. On the other hand, if it is difficult to increase the intensity of the light irradiated from the illumination device 2, it is preferable that the illumination control unit 313 does not increase the intensity of the light irradiated from the illumination device 2, but that the exposure control unit 316 increases the exposure time of the camera 1.

[0058] Furthermore, compensation for exposure reduction when the workpiece 5 and the camera 1 are at the second distance is not limited to being performed by either the exposure control unit 316 or the illumination control unit 313, but may be performed by both the exposure control unit 316 and the illumination control unit 313. In this case, compensation for exposure time by the exposure control unit 316 and compensation for exposure amount by the illumination control unit 313 can be divided at a predetermined ratio. Specifically, if it is necessary to narrow the aperture to twice the amount at the second distance compared to the first distance, for example, it is possible to control the exposure time of the second image to be twice that of the first image, and also control the exposure amount (intensity of the irradiated light) of the second image to be twice that of the first image.

[0059] The first distance and second distance described above are not limited to the case where the first distance is greater than the second distance, i.e., the case where the hand unit 110 to which the camera 1 or the like is attached is moved from far to near the workpiece 5. The image processing device (vision system) of this embodiment can also be applied to the case where the first distance is equal to the second distance, i.e., the case where the hand unit 110 to which the camera 1 or the like is attached is moved at an equal distance from the workpiece 5. Furthermore, the image processing device of this embodiment can also be applied to the case where the first distance is less than the second distance, i.e., the case where the hand unit 110 to which the camera 1 or the like is attached is moved from near to far from the workpiece 5.

[0060] Specifically, when the first distance = the second distance, the workpiece 5 can be focused on as is even when the workpiece 5 and the camera 1 are at the second distance, so imaging can be performed with the lens aperture amount remaining the same. Also, when the first distance < the second distance, the workpiece 5 can be focused on even when the depth of field is shallow when the workpiece 5 and the camera 1 are at the second distance, so the aperture amount of the camera 1 can be reduced by referring to the aperture information stored in the aperture memory unit 323. In this case, it is also possible to compensate (offset) the increase in exposure corresponding to the reduced aperture amount by at least one of the exposure control unit 316 and the illumination control unit 313.

[0061] Thus, according to one example of the image processing device of this embodiment, the lens focus position is fixed at both the first distance and the second distance between the camera 1 and the workpiece 5. This makes it possible to calculate the spatial position information of the workpiece 5 from the image information using the same parameters for the first image captured at the first distance and the second image captured at the second distance. As a result, image processing to acquire the spatial position information of the workpiece based on the image information can be performed with high accuracy in a short time. Note that in the image processing device (vision system) of this embodiment, the camera 1 is not limited to an autofocus camera as long as it has a lens focusing function. It goes without saying that it may also be, for example, a camera with manual focusing.

[0062] Fig. 6 is a flowchart for explaining an example of the processing of an image processing program executed by the processing unit (arithmetic processing unit) of the image processing device shown in Fig. 5. Here, Fig. 6 is for explaining the processing in the industrial robot system described with reference to Fig. 1 when, for example, the hand unit 110 has been taught in advance to move from a distance to a distance close to the workpiece 5 (when the first distance is greater than the second distance). The workpiece 5 is placed on the workbench 4, and the camera 1 and the lighting device 2 are attached near the hand unit 110 of the industrial robot 100.

[0063] 6, when an example of processing of the image processing program starts (START), in step ST1, for example, the camera 1 (illumination device 2) attached near the hand unit 110 of the industrial robot 100 is moved to a position at a first distance from the workpiece 5. Here, the hand unit 110 is moved to a position at the first distance from the workpiece 5 based on a movement route that has been taught in advance by the robot control device 200, for example.

[0064] Next, the process proceeds to step ST2, where the focus control unit 312 adjusts and fixes the focus of the lens in the camera 1 at the first distance position so that it is focused on the workpiece 5. Further, the process proceeds to step ST3, where the aperture control unit 311 adjusts the aperture of the lens in the camera 1 at the first distance position, and the process proceeds to step ST4.

[0065] In step ST4, a position transformation parameter corresponding to the first distance is acquired from the image and set in parameter storage unit 322. Then, the process proceeds to step ST5, where a first image is captured at the position of the first distance by camera 1. Then, the process proceeds to step ST6, where a first process is executed based on the first image.

[0066] In this first process, for example, spatial position information of the workpiece 5 is acquired based on a first image with a fairly wide angle of view that includes the periphery of the workpiece 5, using the parameters set in step ST4 and a conversion formula that converts image information of the workpiece 5 into spatial position information. Then, the feature extraction unit 314 applies the features of the workpiece 5 read from the feature storage unit 321 to the acquired spatial position information of the workpiece 5, and calculates or modifies a movement route for moving the hand unit 110 to a position at a second distance closer to the workpiece 5, for example.

[0067] Next, the process proceeds to step ST7, where the hand unit 110 is moved, for example, based on the movement route calculated or corrected in step ST6. That is, the movement of the hand unit 110 in step ST7 can be performed by correcting a pre-taught movement route based on the spatial position information of the workpiece 5 acquired in the first process in step ST6, and moving the hand unit 110 from a position at the first distance to a position at the second distance. Note that the movement of the hand unit 110 in step ST7 can also be performed according to a pre-taught movement route, regardless of the spatial position information of the workpiece 5 acquired in the first process in step ST6. These can be selected as appropriate depending on, for example, the accuracy required for the system to which the industrial robot 200 is applied, the magnitude of the expected positional deviation of the workpiece 5, and the like.

[0068] Further, the process proceeds to step ST8, where the aperture control unit 311 adjusts the aperture of the lens at the second distance position, and the process proceeds to step ST9. Here, the focus position set in the lens in step ST8 is the same as the fixed position adjusted so as to focus on the workpiece 5 at the first distance position.

[0069] That is, in step ST8, the aperture control unit 311 changes only the aperture amount of the lens, and adjusts so that the workpiece 5 is within the depth of field (in focus) at the second distance. Compensation for exposure reduction when the workpiece 5 and the camera 1 are at the second distance may be performed by either the exposure control unit 316 or the illumination control unit 313, but can also be performed using both the exposure control unit 316 and the illumination control unit 313, as described above.

[0070] In step ST9, a second image is captured at a position of the second distance, for example, with an angle of view in which the workpiece 5 occupies most of the screen. Here, the parameters used when acquiring the spatial position information of the workpiece 5 from the image information of the second image are the same as those used when acquiring the spatial position information of the workpiece 5 from the image information of the first image, because the focus position of the lens is fixed.

[0071] That is, at the position of the second distance, there is no need to acquire new parameters, and the processing time for acquiring the spatial position of the workpiece can be shortened. Furthermore, the lens focus uses the focus position fixed at the first distance as it is at the second distance, so it is possible to prevent the occurrence of lens position errors due to the driver that changes the lens position, and improve the accuracy of converting image information into spatial position information of the workpiece.

[0072] Then, proceed to step ST10, and for example, based on the image information of the second image in which the workpiece 5 is captured enlarged, calculate the spatial position information of the workpiece 5 using the same parameters as those used to process the first image, and then end the process (END).

[0073] In the above-described flowchart, for example, in the processing of step ST7, the workpiece 5 is moved to the second position according to a movement route that has been taught in advance, and then in step 10, a first processing based on the first image and a second processing based on the second image are performed. Specifically, when the first distance is equal to the second distance, the first image and the second image are, for example, images of the workpiece 5 captured from different angles, and by processing these two images to calculate the spatial position of the workpiece 1, the calculation accuracy can be improved.

[0074] Furthermore, the spatial position of the workpiece 1 can be calculated by processing the first and second images taken at different positions and angles. The images to be captured are not limited to the first and second images, but similar processing can be performed to capture a third image, a fourth image, and so on to obtain the spatial position of the workpiece 5. Needless to say, the position of the workpiece 5 (spatial position information of the object) can be obtained from one or more captured images (image information of the object) by applying, for example, various known image processing techniques.

[0075] The image processing program or application software may be provided by being recorded on a computer-readable non-transitory recording medium or a non-volatile semiconductor storage device, or may be provided via a wired or wireless connection. Examples of computer-readable non-transitory recording media include optical disks such as CD-ROMs (Compact Disc Read Only Memory) and DVD-ROMs, or hard disk drives. Examples of non-volatile semiconductor storage devices include PROMs (Programmable Read Only Memory) and flash memories (registered trademark). Furthermore, distribution from a server device may include provision via a wired or wireless WAN (Wide Area Network), LAN (Local Area Network), or the Internet.

[0076] As described above in detail, the image processing device, vision system, and image processing program according to this embodiment make it possible to perform image processing to obtain spatial position information of an object based on image information with high accuracy in a short time.

[0077] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the invention or the concept and spirit of the present invention derived from the content of the claims and their equivalents. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0078] REFERENCE SIGNS LIST 1 Image capturing device (camera) 2 Illumination device 2a Circular illumination 2b Ring illumination 3 Image processing device 4 Work table 5 Object (workpiece) 31 Processing unit (arithmetic processing device) 32 Memory unit 100 Industrial robot 110 Hand unit 200 Robot control device 311 Aperture control unit 312 Focus control unit 313 Illumination control unit 314 Feature extraction unit 315 Position calculation unit 316 Exposure control unit 320 Image processing program 321 Feature memory unit (feature information) 322 Parameter memory unit (parameter information) 323 Aperture memory unit (aperture information)

Claims

1. 1. An image processing device that acquires spatial position information of an object based on image information from an image capturing device having a lens that captures an image of the object, an aperture control unit for controlling the aperture of the lens; When a distance of the image capturing device relative to the object is a first distance, a focus position of the lens is adjusted and fixed, and the aperture of the lens is set to a first aperture by the aperture control unit, and the object is captured by the image capturing device to obtain a first image; When the distance of the image capturing device to the object is a second distance, the aperture control unit sets the lens aperture to a second aperture, and the image capturing device captures an image of the object to obtain a second image.

2. Further, a focus control unit that controls the focus of the lens is provided, The image processing device according to claim 1 , wherein the focus control unit adjusts and fixes a focus position of the lens when a distance of the image capturing device from the object is the first distance.

3. The image processing device according to claim 1 , further comprising a storage unit that stores an image processing program for acquiring spatial position information of the object by referring to the first image and the second image.

4. Further, an exposure control unit is provided for controlling an exposure amount of the image capturing device, 3. The image processing device according to claim 1, wherein the exposure control unit adjusts the exposure amount of the image capturing device based on the aperture of the lens set to the second aperture when the focus position of the lens is at the second distance from the object.

5. The imaging device further includes an illumination light control unit that controls the intensity of illumination light from an illumination device provided near the imaging device, 3. The image processing device according to claim 1, wherein the illumination light control unit adjusts an intensity of the illumination light from the lighting device based on an aperture of the lens set to the second aperture when a focus position of the lens is at the second distance from the object.

6. The image processing device according to claim 1 , wherein at least one of the image capturing device and the object is held by an actuator whose movement is controllable.

7. The length of the first distance is greater than the length of the second distance, The image processing device according to claim 1 , wherein the first aperture value is smaller than the second aperture value.

8. The second distance is defined with reference to the first image; The image processing device according to claim 1 , wherein the spatial position information of the object is acquired by referring to the second image.

9. The image processing device according to claim 1 , wherein the spatial position information of the object is acquired by referring to the first image and the second image.

10. The image processing device according to claim 1 , wherein the second aperture is defined by referring to the first distance, the second distance, and aperture information created in advance.

11. The image processing device according to claim 1 or 2, A vision system comprising the image capturing device.

12. An image processing program for an image processing device that acquires spatial position information of an object based on image information from an image capturing device having a lens that captures an image of the object, A processing unit includes: When a distance of the image capturing device to the object is a first distance, a focus position of the lens is adjusted and fixed, and an aperture of the lens is set to a first aperture, and an image of the object is captured by the image capturing device to obtain a first image; An image processing program that executes a process in which, when the distance of the image capturing device to the object is a second distance, the lens aperture is set to a second aperture, and the object is captured by the image capturing device to obtain a second image.