Image processing device, vision system, and image processing program

Abstract

An image processing device (3) acquires spatial position information of an object (5) on the basis of image information from an image capturing device (1) having a lens for capturing an image of the object (5), and has an aperture control unit (311) for controlling an aperture of the lens. When the distance of the image capturing device with respect to the object is a first distance, the focus position of the lens is adjusted and fixed, the aperture of the lens is set as a first aperture by the aperture control unit, and the object is captured by the image capturing device to acquire a first image. When the distance of the image capturing device with respect to the object is a second distance, the aperture control unit sets the aperture of the lens as a second aperture, and the image capturing device captures an image of the object to acquire a second image.

Description

Technical Field

[0001] The embodiments mentioned in this application relate to an image processing device, a visual system, and an image processing program. Background Art

[0002] In recent years, vision systems (machine vision, robot vision) have been applied to industrial machinery, industrial robots, and collaborative robots, and the industrial machinery and the like are caused to perform predetermined operations based on image information acquired by the vision systems.

[0003] For example, in an industrial robot using robot vision, spatial position information of an object (workpiece) is inferred based on image information obtained by an image capturing device (camera), and the industrial robot is controlled to perform specified processing on the workpiece.

[0004] In addition, in this specification, for the sake of simplicity, the robot vision of an industrial robot is used as an example of a visual system to which the image processing device is applied. However, the image processing device, the visual system, and the image processing program of this embodiment are not limited to industrial robots and robot vision, but can be widely applied to various machines and robots.

[0005] In this specification, the term visual system refers to a system that operates a device by taking in and processing an image, and includes all industrial and non-industrial applications that are a combination of hardware and software.

[0006] Conventionally, various proposals have been made as control techniques for cameras used in robot vision or vision systems of industrial robots.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Publication No. 2019-113680

[0010] Patent Document 2: Japanese Patent Application Publication No. 2019-161553

[0011] Patent Document 3: Japanese Patent Application Publication No. 2006-253998 Summary of the invention

[0012] Problems to be solved by the invention

[0013] As described above, for example, in an industrial robot using robot vision, it is known to acquire spatial position information from image information of a workpiece captured by a camera and control the industrial robot to perform predetermined processing on the workpiece.

[0014] As cameras used in such robot vision, fixed-focus cameras are generally used, but recently, a configuration using cameras having a focusing function (for example, auto-focus cameras) has also been used.

[0015] However, robot vision using a fixed-focus camera does not require adjustment of the focus position, and therefore, does not require changing parameters used in a conversion formula for converting image information of a workpiece into spatial position information.

[0016] In contrast, robot vision using a camera with a focus function can capture a clear image by adjusting the focus position of the camera relative to the workpiece (the distance between the workpiece and the camera). However, when adjusting the focus position, for example, it is necessary to change the parameters used in the conversion formula for converting the image information obtained by capturing the workpiece into the spatial position information each time.

[0017] That is, when the focus position (focal length) of the lens changes, the parameters used in the conversion equation need to be changed, and therefore, it is required to acquire in advance the parameters corresponding to the focus position at the time of shooting each time.

[0018] As a result, the processing when photographing the workpiece (object) may become complicated, or the processing for obtaining the spatial position of the object may take a long time. In addition, the focusing of the lens in the camera (image capturing device) is performed by, for example, a driving unit that changes the positional relationship of the lens. Therefore, there is also a problem that the position of the lens is prone to errors, and the accuracy of converting the image information into the spatial position information of the object is reduced.

[0019] The problem to be solved by the present invention is to provide an image processing device, a visual system, and an image processing program that can perform image processing to obtain spatial position information of an object from image information in a short time with high accuracy.

[0020] Means for solving problems

[0021] According to one embodiment of the present invention, there is provided an image processing device for obtaining spatial position information of an object based on image information from an image capturing device having a lens for capturing the object. The image processing device has an aperture control unit for controlling the aperture of the lens. When the distance between the image capturing device and the object is a first distance, the focus position of the lens is adjusted and fixed, and the aperture of the lens is set to the first aperture by the aperture control unit, and the object is captured by the image capturing device to obtain a first image. When the distance between the image capturing device and the object is a second distance, the aperture of the lens is set to the second aperture by the aperture control unit, and the object is captured by the image capturing device to obtain a second image.

[0022] The objects and effects of the present invention will be realized and achieved by using the structural elements and combinations particularly pointed out in the claims. Both the above general description and the detailed description to be given below are exemplary and explanatory and do not limit the present invention described in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. 1 is a diagram schematically showing an industrial robot system as an example of an image processing device to which the present embodiment is applied.

[0024] Figure 2 is used to Figure 1 FIG. 2 is a diagram for explaining parameters of a conversion formula used to acquire spatial position information of an object from image information of an image pickup device shown.

[0025] Figure 3 It is used for Figure 1 FIG. 1 is a diagram for explaining the relationship between the aperture and depth of field of a lens in the industrial robot system shown in FIG.

[0026] Figure 4 Yes means Figure 1 FIG. 1 is a diagram showing an example of a lighting device used in an industrial robot system.

[0027] Figure 5 This is a block diagram of a visual system showing an example of an image processing device to which this embodiment is applied.

[0028] Figure 6 is used for Figure 5 The flowchart is used to explain an example of the processing of the image processing program executed by the processing unit (arithmetic processing device) of the image processing device shown in FIG. DETAILED DESCRIPTION

[0029] First, before describing in detail the embodiments of the image processing device, the visual system, and the image processing program of the present embodiment, Figure 1 to Figure 4 , an industrial robot system as an example of an image processing device to which this embodiment is applied and an overview of this embodiment are described. Figure 1 FIG. 1 is a diagram schematically showing an industrial robot system as an example of an image processing device to which the present embodiment is applied.

[0030] exist Figure 1In the figure, reference numeral 100 denotes an industrial robot, 110 denotes a hand, 200 denotes a robot control device, and 3 denotes an image processing device. In addition, reference numeral 1 denotes an image capturing device (camera), 2 denotes an illumination device (ring illumination), 4 denotes a workbench, and 5 denotes an object (workpiece). In addition, the robot vision (vision system) is configured to include, for example, the camera 1, the illumination device 2, and the image processing device 3.

[0031] Figure 1 FIG. 1 shows a case where the hand 110 of the industrial robot 100 is moved according to a preset teaching path, and the hand 110 performs an action of grasping the workpiece 5. Figure 1 In FIG. 1 , the position of the workpiece 5 placed on the work table 4 is shifted from the original position (position based on the teaching path) 5 a indicated by the dotted line.

[0032] Right now, Figure 1 The figure shows a situation where the hand 110 is moved according to a preset teaching path, and the workpiece 5 on the workbench 4 is grasped by the hand 110, but the actual spatial position of the workpiece 5 on the workbench 4 is offset from the assumed original position 5a. Here, the camera 1 and the lighting device 2 are installed near the hand 110 of the industrial robot 100, and can move with the hand 110 to photograph the workpiece 5 placed on the workbench 4.

[0033] At this time, the image processing device 3 obtains the actual spatial position information of the workpiece 5 based on the image information of the workpiece 5 on the workbench 4 captured by the camera 1, and recognizes the deviation from the assumed position 5a. In addition, the image processing device 3 is controlled via the robot control device 200 so as to recognize and correct the deviation between the assumed position 5a of the workpiece 5 and the actual spatial position, and the workpiece 5 is correctly grasped by the hand 110.

[0034] In this way, by applying robot vision to the industrial robot system, for example, even if the position of the workpiece 5 on the workbench 4 is offset, the offset can be corrected so that the industrial robot 100 can perform correct operations. Figure 1 In the embodiment, the image processing device 3 is built in the robot control device 200 , but may be separately provided outside the robot control device 200 .

[0035] Here, in Figure 1 In the industrial robot system shown, a workpiece 5 is placed on a workbench 4, and a camera 1 is installed near a hand 110 (actuator) of the industrial robot 100 and moves. In addition, the image processing device 3 of this embodiment can be widely used in various machines and robots in which at least one of the camera 1 and the workpiece 5 is held by an actuator that can be moved and controlled.

[0036] Figure 2 is used to Figure 1 FIG. 1 is a diagram for explaining the parameters of a conversion formula used to obtain spatial position information of an object from image information of an image capturing device shown in FIG. Figure 2 In the equation (I), the three-dimensional coordinate points in the so-called pinhole camera model are projected onto the image plane using perspective projection transformation, thereby forming a relationship between the position in space and the position on the image. In addition, the equation (II) is an expanded relationship that takes into account the distortion in the radial direction and the circumferential direction of the lens of the actual camera 1.

[0037] In formula (I), reference symbol AA represents the position on the image (the position of the workpiece 5 in the captured image captured by the camera 1), BB represents the focal length and image center of the camera 1, CC represents the camera coordinates in space, and DD represents the position in space (the position of the workpiece 5 in the actual space). In addition, (X, Y, Z) represents the three-dimensional coordinates of the world coordinate system, (u, v) represents the coordinates of the point projected to the image plane, (cx, cy) represents the principal point (usually the image center), and fx, fy represent the focal length expressed in pixel units.

[0038] In formula (II), t represents [t 1 , t 2 , t 3 ], R represents a 3×3 matrix of r, k 1 ~k 6 and p 1 、p 2 Therefore, the actual position of the workpiece 5 in space can be obtained by converting the position of the workpiece 5 in the image captured by the camera 1 according to formula (II). At this time, by preparing the camera coordinates CC, focal length and image center BB in space, and distortion parameters (k 1 ~k 6 and p 1 、p 2 : 6 types of k and 2 types of p), thereby being able to obtain the spatial position DD from the position AA on the image.

[0039] Here, when the focus position of the lens is changed, the focal length f changes each time it is changed, and it is necessary to obtain in advance the parameters (6 types of k and 2 types of p) corresponding to the focal length f. That is, in a visual system using a camera with a focus function, when the focus position (focal length) of the lens changes, it is necessary to change the parameters used in the conversion formula.

[0040] Thus, when using a camera with a focus function, it is necessary to obtain parameters corresponding to the focus position when shooting in advance, which may cause the process of shooting the workpiece to become complicated or take a long time. In addition, since the lens is focused by a drive unit that changes the position of the lens, there is also a problem that the position of the lens is prone to errors, and the accuracy of converting the image information into the spatial position information of the workpiece is reduced.

[0041] Figure 3 It is used for Figure 1 The diagram for explaining the relationship between the aperture and depth of field of the lens in the industrial robot system shown in FIG. 1 is a diagram for explaining the relationship between the aperture (aperture value, aperture amount) and depth of field in the lens of the camera 1. Figure 3 In the figure, reference symbol L represents the distance to the object (workpiece 5), δ represents the permissible circle of confusion (a constant determined by the resolution), f represents the focal length, and F represents the aperture amount (aperture value).

[0042] The depth of field D is a focusing range, and is a value obtained by adding the front depth of field Df and the rear depth of field Dr (D=Df+Dr). Df and Dr can be expressed as follows.

[0043] Df=(δFL 2 ) / (f 2 +δFL)

[0044] Dr=(δFL 2 ) / (f 2 -δFL)

[0046] Here, under the condition that f, δ, and L are fixed, if the aperture value F is increased, the depth of field D becomes larger (deeper), but the amount of light incident on the imaging element (CMOS (Complementary Metal Oxide Semiconductor), CCD (Charge Coupled Device) image sensor) decreases. Specifically, the amount of light incident on the imaging element can be expressed as C / F when the aperture value is set to F and C is set to a constant. 2 Therefore, for example, when the aperture value F is set to 2 times, the amount of light incident on the image sensor is 1 / 4.

[0047] However, as a method of adjusting the brightness of an image, there is a method of adjusting the exposure time of the image. If the exposure time is set to 2 times, an image with twice the brightness is obtained. Therefore, it is considered to increase the aperture F of the lens, deepen the depth of field D of the lens, that is, expand the focus range (focus range) of the lens, and compensate for the decrease in the amount of incident light caused by the increase in the aperture F by increasing the exposure time.

[0048] Specifically, for example, when the lens aperture F is doubled to increase the depth of field D of the lens, a captured image with the same brightness can be obtained by increasing the exposure time by four times. That is, the exposure time t is expressed as t=t 0 ×(F / F 0 ) 2 , the effect of changing the aperture value F can be compensated by adjusting the exposure time. 0 Indicates the exposure time before changing F. 0 represents the aperture amount before F is changed, and F represents the aperture amount after the change.

[0049] As referenced later Figure 5 and Figure 6 As described in detail, in the example of the image processing device of this embodiment, when the distance between the camera 1 and the workpiece 5 is the first distance, the focus position of the lens is adjusted and fixed by the focus control unit 312. In addition, the aperture control unit 311 sets the aperture of the lens as the first aperture, and the workpiece 5 is photographed to obtain a first image.

[0050] Furthermore, when the distance between the camera 1 and the workpiece 5 is the second distance, the focus position of the lens is fixed directly, and the aperture of the lens is set to the second aperture by the aperture control unit 311, so that the depth of field (D=Df+Dr) of the lens set to the second aperture includes the workpiece 5. That is, when the workpiece 5 is at the second distance from the camera 1, the workpiece 5 is focused (focused) according to the depth of field of the lens adjusted by setting the second aperture, and the workpiece 5 is photographed to obtain the second image.

[0051] Thus, according to the embodiment of the image processing device of the present embodiment, for example, the focus position of the lens is fixed at both the first distance and the second distance where the distances between the camera 1 and the workpiece 5 are different, and the aperture of the lens is adjusted at the second distance. Thus, at both the first distance and the second distance where the distances between the camera 1 and the workpiece 5 are different, the first image and the second image focused on the workpiece 5 can be obtained.

[0052] In this way, when the lens with a focus function changes the focus position, it is necessary to change the parameters according to the focus position, but the focus position of the lens is fixed at the same position at both the first distance and the second distance, so there is no need to change the parameters. That is, the parameters used at both the first distance for shooting the first image and the second distance for shooting the second image can be the same, and there is no need to prepare new parameters. As a result, according to the embodiments of the image processing device, the visual system, and the image processing program of the present embodiment, it is possible to perform image processing to obtain the spatial position information of the workpiece 5 based on the image information in a short time with high accuracy.

[0053] Figure 4 It means in Figure 1FIG. 1 is a diagram showing an example of a lighting device used in an industrial robot system. Figure 4 (a) in the figure shows the case where a circular lighting 2a is used as the lighting device 2. Figure 4 (b) in the figure shows a case where a ring lighting 2b is used as the lighting device 2.

[0054] like Figure 4 As shown in (a) of FIG. 1 , as the lighting device 2, for example, a circular lighting 2a disposed near the camera 1 can be applied. Figure 4 As shown in (b) in FIG. 1 , a ring-shaped lighting 2 b provided on the outer periphery of the lens of the camera 1 may also be applied.

[0055] Here, the circular lighting 2a and the annular lighting 2b can always irradiate a certain amount of irradiation light, but the intensity of the irradiation light can also be controlled as described in detail later. That is, the lighting device 2 (2a, 2b) can also increase the intensity of the irradiation light when the distance between the workpiece 5 and the camera 1 is the second distance, so as to compensate for the exposure amount that is reduced by increasing the aperture (aperture value) in order to focus on the workpiece 5. Specifically, for example, when the aperture F of the lens is set to 2 times and the depth of field of the lens is deepened, the lighting control unit 313 adjusts the intensity of the irradiation light from the lighting device 2 to 4 times, thereby obtaining a captured image with the same brightness.

[0056] Furthermore, the lighting device 2 is not limited to the circular lighting 2 a and the ring lighting 2 b , and various lighting devices can be applied. Also, the number and arrangement of the lighting devices 2 are of course not limited to only one being provided near the camera 1 .

[0057] Here, the workpiece 5 is illuminated by the illumination light from the illumination device 2 (circular illumination 2a and annular illumination 2b). However, if the distance from the camera 1 to the workpiece 5 is set to r and C is set to a constant, the brightness of the light received by the camera 1 from the workpiece 5 becomes C / r. 2 Therefore, as described above, when the aperture value (F) of the lens is set to 2 times, the amount of light incident on the camera 1 is 1 / 4.

[0058] Furthermore, if we also consider the relationship with the aperture (t = t 0 ×(F / F 0 ) 2 ), the exposure time t can be expressed as t=t 0 ×(F / F 0 ) 2 ×(r / r 0 ) 2 Here, t 0 Indicates the exposure time before changing the lens aperture. 0represents the aperture value before the lens aperture value is changed, F represents the aperture value after the lens aperture value is changed, r represents the distance before the lens aperture value is changed, and r 0 Indicates the distance after changing the aperture of the lens. That is, in order to compensate for exposure, there are a method of adjusting the exposure time of the image and a method of adjusting the intensity of the irradiated light of the lighting device.

[0059] Hereinafter, examples of the image processing device, the visual system, and the image processing program according to the present embodiment will be described in detail with reference to the accompanying drawings.

[0060] Figure 5 1 is a block diagram showing a visual system of an embodiment of an image processing device to which the present embodiment is applied. Here, the visual system of an embodiment of an image processing device to which the present embodiment is applied is equivalent to, for example, Figure 1 The image processing device 3 may be built in the robot control device 200 that controls the industrial robot 100 , but may also be separately provided outside the robot control device 200 as described above.

[0061] And, in Figure 5 In the visual system shown, at least one of the image capturing device (camera) 1 and the object (workpiece) 5 is held by a movable actuator, and the position of the camera 1 relative to the workpiece 5 is controlled. That is, the image processing device 3 of this embodiment can be widely used in various machines and robots that hold at least one of the camera 1 and the workpiece 5 by a movable actuator.

[0062] like Figure 5 As shown, the image processing device 3 is used to obtain spatial position information of the workpiece 5 based on image information from a camera 1 having a lens (not shown) for photographing the workpiece 5, and includes a processing unit (calculation processing unit) 31 and a storage unit 32. In addition, reference numeral 2 is an illumination device for irradiating light to the workpiece 5. In this embodiment, as described later, the illumination device 2 is controlled by the image processing device 3, but, for example, a fixed amount of light may be always irradiated to the workpiece 5.

[0063] The processing unit 31 includes an aperture control unit 311, a focus control unit 312, an illumination control unit 313, a feature extraction unit 314, a position calculation unit 315, and an exposure control unit 316. The aperture control unit 311 controls the aperture of the lens of the camera 1 that photographs the workpiece 5, and the focus control unit 312 controls the focus of the lens of the camera 1. The illumination control unit 313 controls the intensity of the light of the illumination device 2, that is, the intensity of the irradiation light from the illumination device 2 that irradiates the workpiece 5.

[0064] The storage unit 32 stores an image processing program 320 executed by the processing unit 31. The image processing program 320 includes a feature storage unit (feature information) 321, a parameter storage unit (parameter information) 322, and an aperture storage unit (aperture information) 323. The feature storage unit 321 stores, for example, feature information of the workpiece 5 to be detected from the image information of the workpiece 5 captured by the camera 1. The parameter storage unit 322 stores, for example, information on the focus position (focus information) and information on parameters used in a conversion formula for obtaining the spatial position of the workpiece 5 from the image information. The aperture storage unit 323 stores, for example, a relationship between the distance between the workpiece 5 and the camera 1 (the distance of the camera 1 relative to the workpiece 5) and the aperture amount (aperture value) and the depth of field (aperture information).

[0065] The feature extraction unit 314 extracts feature information stored in the feature storage unit 321 from the image information of the workpiece 5 captured by the camera 1 . The position calculation unit 315 calculates the spatial position of the workpiece 5 based on the image information of the workpiece 5 captured by the camera 1 and the parameters acquired from the parameter storage unit 322 .

[0066] Here, for example, when the workpiece 5 is at a first distance from the camera 1, the position calculation unit 315 reads out and obtains the parameter corresponding to the focus position of the lens adjusted and fixed by the focus control unit 312 from the parameter storage unit 322. Furthermore, the position calculation unit 315 calculates the spatial position of the workpiece 5 based on the first image captured at the first distance by setting the aperture of the lens to the first aperture by the aperture control unit 311.

[0067] Here, the parameters obtained when the workpiece 5 is at the first distance from the camera 1 are also used when the workpiece 5 is at the second distance from the camera 1. That is, for example, when the workpiece 5 is at the second distance from the camera 1, the position calculation unit 315 uses the same parameters as when the workpiece 5 is at the first distance from the camera 1 to calculate the spatial position of the workpiece 5 based on the second image captured by setting the aperture of the lens to the second aperture by the aperture control unit 311. At this time, the aperture control unit 311 controls the aperture amount of the camera 1 with reference to the aperture information stored in the aperture storage unit 323.

[0068] Specifically, when the hand 110 is moved from far to near relative to the workpiece 5 on the workbench 4, the aperture control unit 311 increases the aperture amount and deepens the depth of field at the focus position focused at the first distance so as to also focus on the workpiece 5 at the second distance. That is, when the first distance is greater than the second distance, the aperture control unit 311 controls so that the aperture amount of the lens is increased to enter the depth of field at the focus position fixed when the workpiece and the camera are at the first distance, so that the workpiece and the camera are also focused on the workpiece 5 at the second distance.

[0069] The exposure control unit 316 adjusts the exposure time (shutter speed) of the camera 1 to compensate for the decrease in exposure corresponding to the increase in the aperture amount due to focusing on the workpiece 5 to increase the depth of field when the workpiece 5 is at the second distance from the camera 1. Specifically, when the workpiece 5 is at the second distance from the camera 1, when the aperture amount is reduced to twice the aperture amount when the workpiece 5 is at the first distance from the camera 1 and photographing is performed, the exposure control unit 316 controls the exposure time to be four times that when photographing the first image.

[0070] The lighting control unit 313 adjusts the intensity (light amount) of the light irradiated from the lighting device 2 to compensate for the reduction in exposure corresponding to the aperture amount increased to focus on the workpiece 5 to deepen the depth of field when the workpiece 5 is at the second distance from the camera 1. Specifically, when the workpiece 5 is at the second distance from the camera 1, when the aperture is reduced to twice the aperture amount when the workpiece 5 is at the first distance from the camera 1 and shooting is performed, the lighting control unit 313 controls so that the intensity of the irradiated light from the lighting device 2 becomes four times that when the first image is shot.

[0071] Here, when there is a problem in increasing the exposure time, it is preferable that the exposure time of the camera 1 is not increased by the exposure control unit 316, but the intensity of the irradiation light from the illuminating device 2 is increased by the illumination control unit 313. On the other hand, when it is difficult to increase the intensity of the irradiation light from the illuminating device 2, it is preferable that the exposure time of the camera 1 is increased by the exposure control unit 316, instead of increasing the intensity of the irradiation light from the illuminating device 2 by the illumination control unit 313.

[0072] Furthermore, the compensation for the reduced exposure when the workpiece 5 and the camera 1 are at the second distance is not limited to being performed by one of the exposure control unit 316 and the lighting control unit 313, and may be performed by both the exposure control unit 316 and the lighting control unit 313. In this case, the compensation for the exposure time by the exposure control unit 316 and the compensation for the exposure amount by the lighting control unit 313 may also be divided into a predetermined ratio. Specifically, if it is necessary to reduce the aperture to twice the aperture amount of the first distance at the second distance, for example, the exposure time of the second image may be controlled to be twice that of the first image, and the exposure amount (intensity of the irradiated light) of the second image may be controlled to be twice that of the first image.

[0073] In addition, the first distance and the second distance described above are not limited to the case where the first distance is greater than the second distance, that is, the case where the hand 110 equipped with the camera 1 etc. is moved from far to near the workpiece 5. The image processing device (visual system) of this embodiment can also be applied to the case where the first distance = the second distance, that is, the case where the hand 110 equipped with the camera 1 etc. is moved at an equal distance relative to the workpiece 5. Furthermore, the image processing device of this embodiment can also be applied to the case where the first distance is less than the second distance, that is, the case where the hand 110 equipped with the camera 1 etc. is moved from near to far of the workpiece 5.

[0074] Specifically, when the first distance = the second distance, the workpiece 5 can be directly focused even when the workpiece 5 and the camera 1 are at the second distance, so that the image can be taken without changing the aperture of the lens. In addition, when the first distance is less than the second distance, the workpiece 5 can be focused even when the depth of field is reduced when the workpiece 5 and the camera 1 are at the second distance, so the aperture of the camera 1 can be reduced by referring to the aperture information stored in the aperture storage unit 323. At this time, the increase in exposure corresponding to the reduction in the aperture can be compensated (offset) by at least one of the exposure control unit 316 and the lighting control unit 313.

[0075] Thus, according to an embodiment of the image processing device of the present embodiment, when the distance between the camera 1 and the workpiece 5 is both the first distance and the second distance, the focus position of the lens is fixed. Thus, for the first image taken at the first distance and the second image taken at the second distance, the spatial position information of the workpiece 5 can be calculated based on the image information using the same parameters. As a result, image processing for obtaining the spatial position information of the workpiece based on the image information can be performed in a short time with high precision. In addition, in the image processing device (visual system) of the present embodiment, the camera 1 only needs to have the focusing function of the lens, and is not limited to an autofocus camera. For example, it can also be a camera that focuses manually.

[0076] Figure 6 is used for Figure 5 1 is a flowchart for explaining an example of processing of an image processing program executed by a processing unit (arithmetic processing device) of an image processing device shown in FIG. Figure 6 Is used for reference Figure 1 The figure is a diagram for explaining the processing of the case where the hand 110 is pre-taught in a manner such that it moves from a far distance to a near distance of the workpiece 5 (a case where the first distance is greater than the second distance) in the industrial robot system described. In addition, the workpiece 5 is placed on the workbench 4, and the camera 1 and the lighting device 2 are installed near the hand 110 of the industrial robot 100.

[0077] like Figure 6As shown, when an example of the processing of the image processing program is started (START), in step ST1, for example, the camera 1 (illumination device 2) installed near the hand 110 of the industrial robot 100 is moved to a position at a first distance from the workpiece 5. Here, for example, the hand 110 is moved to a position at a first distance from the workpiece 5 according to a movement path taught in advance by the robot control device 200.

[0078] Next, the process proceeds to step ST2, and at the position of the first distance, the focus of the lens in the camera 1 is adjusted and fixed by the focus control unit 312 to focus on the workpiece 5. Then, the process proceeds to step ST3, and the aperture of the lens in the camera 1 is adjusted by the aperture control unit 311 at the position of the first distance, and the process proceeds to step ST4.

[0079] In step ST4, the position conversion parameter corresponding to the first distance is obtained from the image and set in the parameter storage unit 322, and then the process proceeds to step ST5, where the first image is captured at the first distance by the camera 1. Then, the process proceeds to step ST6, where the first process is performed based on the first image.

[0080] As the first processing, for example, the spatial position information of the workpiece 5 is acquired based on the first image having a wide field angle to a certain extent including the periphery of the workpiece 5, using the parameters set in step ST4 and the conversion formula for converting the image information of the workpiece 5 into the spatial position information. Then, the features of the workpiece 5 read from the feature storage unit 321 by the feature extraction unit 314 are applied to the acquired spatial position information of the workpiece 5, and, for example, a moving path for moving the hand 110 to a position at a second distance closer to the workpiece 5 is calculated or corrected.

[0081] Next, the process proceeds to step ST7, and the hand 110 is moved, for example, according to the movement path calculated or corrected in step ST6. That is, the movement of the hand 110 in step ST7 can correct the pre-taught movement path, for example, according to the spatial position information of the workpiece 5 obtained in the first process in step ST6, and move from the position of the first distance to the position of the second distance. In addition, the movement of the hand 110 in step ST7 can also be moved according to the pre-taught movement path regardless of the spatial position information of the workpiece 5 obtained by the first process in step ST6. These are appropriately selected according to, for example, the accuracy required by the system to which the industrial robot 200 is applied, the size of the expected positional deviation of the workpiece 5, and the like.

[0082] Then, the process proceeds to step ST8, where the aperture of the lens is adjusted at the second distance by the aperture control unit 311, and the process proceeds to step ST9. Here, in step ST8, the focus position set for the lens is directly adjusted and fixed at the first distance to focus on the workpiece 5.

[0083] That is, in step ST8, the aperture control unit 311 changes only the aperture amount of the lens, and adjusts the position of the second distance so that the workpiece 5 is within the depth of field (focused). In addition, the compensation for the reduced exposure when the workpiece 5 and the camera 1 are at the second distance may be performed by either the exposure control unit 316 or the illumination control unit 313, but as described above, it can be performed by both the exposure control unit 316 and the illumination control unit 313.

[0084] In step ST9, a second image is captured at a second distance, for example, at a field angle in which the workpiece 5 occupies most of the screen. Here, the parameters used when obtaining the spatial position information of the workpiece 5 from the image information of the second image are the same as those used when obtaining the spatial position information of the workpiece 5 from the image information of the first image because the focus position of the lens is fixed.

[0085] That is, at the position of the second distance, it is not necessary to obtain new parameters, and the processing time for obtaining the spatial position of the workpiece can be shortened. In addition, in the focus of the lens, even at the second distance, the focus position fixed to the first distance is directly used, thereby preventing the occurrence of lens position errors caused by the drive unit that changes the position of the lens, and improving the accuracy of converting the image information into the spatial position information of the workpiece.

[0086] Then, the process proceeds to step ST10 , where, for example, based on the image information of the second image obtained by capturing the workpiece 5 in a larger image, the spatial position information of the workpiece 5 is calculated using the same parameters as those used to process the first image, and the process ends (END).

[0087] In the above flowchart, for example, as the processing of step ST7, it is also possible to move to the second position according to the pre-taught moving path, and in step 10, the first processing based on the first image and the second processing based on the second image are performed. Specifically, when the first distance = the second distance, the first image and the second image become images obtained by photographing the workpiece 5 from different angles, for example, but by processing such two images to calculate the spatial position of the workpiece 1, the calculation accuracy can be improved.

[0088] Furthermore, the spatial position of the workpiece 1 can be calculated by processing the first image and the second image at different positions and angles. In addition, the captured images are not limited to the first image and the second image, and the same processing can be performed to capture the third image, the fourth image, ... to obtain the spatial position of the workpiece 5. In addition, in order to obtain the position of the workpiece 5 (spatial position information of the object) from one or more captured images (image information of the object), it is of course possible to apply various known image processing methods, for example.

[0089] In addition, the image processing program or application software can be provided by recording on a non-temporary recording medium or a non-volatile semiconductor storage device that can be read by a computer, or can be provided via wired or wireless. Here, as a non-temporary recording medium that can be read by a computer, for example, a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM or other optical disk, or a hard disk device, etc. are considered. In addition, as a non-volatile semiconductor storage device, a PROM (Programmable Read Only Memory), a flash memory (Flash Memory) (registered trademark), etc. are considered. And, as distribution from a server device, provision via a wired or wireless WAN (Wide Area Network), a LAN (Local Area Network) or the Internet, etc. is considered.

[0090] As described above in detail, according to the image processing device, visual system, and image processing program of the present embodiment, it is possible to perform image processing for acquiring spatial position information of an object from image information in a short time with high accuracy.

[0091] The embodiments of the present disclosure are described in detail, but the present disclosure is not limited to the above-mentioned embodiments. These embodiments can be variously added, replaced, changed, partially deleted, etc. without departing from the scope of the main purpose of the invention, or without departing from the concept and main purpose of the present invention derived from the contents recorded in the claims and their equivalents. For example, in the above-mentioned embodiments, the order of each action and the order of each processing are shown as an example, and are not limited to this. In addition, the same is true for the case where numerical values ​​or mathematical formulas are used in the description of the above-mentioned embodiments.

[0092] Description of Reference Numerals

[0093] 1 Image capture device (camera)

[0094] 2 lighting fixtures

[0095] 2a Circular lighting

[0096] 2b Ring lighting

[0097] 3 Image processing device

[0098] 4 Workbench

[0099] 5 Object (workpiece)

[0100] 31 Processing unit (calculation processing device)

[0101] 32 Storage

[0102] 100 Industrial robots

[0103] 110 Hands

[0104] 200 Robot control device

[0105] 311 Aperture control unit

[0106] 312 Focus control unit

[0107] 313 Lighting Control Department

[0108] 314 Feature Extraction Department

[0109] 315 Position calculation unit

[0110] 316 Exposure Control Department

[0111] 320 Image Processing Program

[0112] 321 Feature storage unit (feature information)

[0113] 322 parameter storage unit (parameter information)

[0114] 323 Aperture storage unit (aperture information).

Claims

1. An image processing device for obtaining spatial position information of an object based on image information from an image pickup device having a lens for picking up an object, characterized in that: The image processing device comprises: an aperture control unit which controls the aperture of the lens; When the distance between the image capturing device and the object is a first distance, the focus position of the lens is adjusted and fixed, and the aperture of the lens is set to a first aperture by the aperture control unit, and the object is captured by the image capturing device to obtain a first image. When the distance between the image capturing device and the object is a second distance, the aperture of the lens is set to the second aperture by the aperture control unit, and the image capturing device captures the object to obtain a second image.

2. The image processing device according to claim 1, characterized in that The image processing device further comprises: a focus control unit, which controls the focus of the lens. When the distance between the image capturing device and the object is the first distance, the focus control unit adjusts and fixes the focus position of the lens.

3. The image processing device according to claim 1 or 2, characterized in that: The image processing device further includes a storage unit storing an image processing program for acquiring spatial position information of the object by referring to the first image and the second image.

4. The image processing device according to any one of claims 1 to 3, characterized in that: The image processing device further comprises: an exposure control unit that controls the exposure amount of the image capturing device. When the focus position of the lens is at the second distance from the object, the exposure control unit adjusts the exposure amount of the image capture device according to the aperture of the lens set to the second aperture.

5. The image processing device according to any one of claims 1 to 4, characterized in that: The image processing device further includes: an irradiation light control unit that controls the intensity of irradiation light from a lighting device provided near the image capturing device. The irradiation light control unit adjusts the intensity of the irradiation light from the lighting device according to the aperture of the lens set to the second aperture when the focus position of the lens is at the second distance from the object.

6. The image processing device according to any one of claims 1 to 5, characterized in that: At least one of the image pickup device and the object is held by a movement-controllable actuator.

7. The image processing device according to any one of claims 1 to 6, characterized in that: The length of the first distance is longer than the length of the second distance, The value of the first aperture is smaller than the value of the second aperture.

8. The image processing device according to any one of claims 1 to 7, characterized in that: defining the second distance with reference to the first image, The spatial position information of the object is acquired by referring to the second image.

9. The image processing device according to any one of claims 1 to 8, characterized in that: The spatial position information of the object is acquired by referring to the first image and the second image.

10. The image processing device according to any one of claims 1 to 9, characterized in that: The second aperture is defined with reference to the first distance and the second distance, and pre-created aperture information.

11. A visual system, characterized in that: have: The image processing device according to any one of claims 1 to 10; and The image capturing device.

12. An image processing program for an image processing device, the image processing device acquiring spatial position information of an object based on image information from an image pickup device having a lens for picking up an object, characterized in that: The image processing program causes the operation processing device to perform the following processing: When the distance between the image capturing device and the object is a first distance, the focus position of the lens is adjusted and fixed, and the aperture of the lens is set to a first aperture, and the object is captured by the image capturing device to obtain a first image, When the distance between the image capturing device and the object is a second distance, the aperture of the lens is set to the second aperture, and the image capturing device captures the object to obtain a second image.

Images