IMAGE PROCESSING DEVICE, IMAGE PROCESSING SYSTEM AND IMAGE PROCESSING PROGRAM

Abstract

An image processing device 3 includes an aperture control unit 311, which, based on image information from an image acquisition device 1 (which has a lens and captures an image of an object 5), obtains spatial position information of the object and controls the aperture of the lens. When the distance between the image acquisition device and the object is a first distance, the image acquisition device captures an image of the object to obtain a first image in which the focus position for the lens is set and fixed, and in which the aperture of the lens is set to a first aperture by the aperture control unit. When the distance between the image acquisition device and the object is a second distance, the image acquisition device captures an image of the object to obtain a second image in which the aperture of the lens is set to a second aperture by the aperture control unit.

Description

FIELD

[0001] The embodiments described here relate to an image processing apparatus, an image processing system and an image processing program. BACKGROUND

[0002] In recent years, an image processing system (machine vision, robot vision) has been applied to an industrial machine, an industrial robot, or a cooperative robot, and a predetermined operation is performed on the industrial machine or the like based on an image obtained by the image processing system.

[0003] For example, in an industrial robot to which robot vision is applied, the industrial robot is controlled to perform a predetermined process on the workpiece based on image information obtained by capturing a target object (workpiece) with an image capturing device (camera).

[0004] Note that, for the convenience of explanation, in this specification, robot vision of an industrial robot is described as an image processing system to which an image processing device is applied. However, the image processing device, image processing system, and image processing program according to the present embodiment are not limited to the industrial robot and robot vision, and can be widely applied to various machines and robots, etc.

[0005] Furthermore, the term “vision system” in this description refers to a system that controls various machines based on image acquisition and processing operations and includes all industrial and non-industrial applications that combine hardware and software.

[0006] So far, various proposals have been made for robot vision of an industrial robot or for controlling a camera used in an image processing system. [CITATION LIST][PATENT LITERATURE] [PTL 1] Unexamined Japanese Patent Publication (Kokai) No. 2019-113680 [PTL 2] Unexamined Japanese Patent Publication (Kokai) No. 2019-161553 [PTL 3] Unexamined Japanese Patent Publication (Kokai) No. 2006-253998 SUMMARY[TECHNICAL PROBLEM]

[0007] As described above, for example, it is known that in the industrial robot to which robot vision is applied, spatial position information is obtained from image information of a workpiece taken by a camera, and the industrial robot is controlled to perform a predetermined process on the workpiece.

[0008] However, in recent years, a camera with a focus function (e.g. an autofocus camera) has also been used.

[0009] In robot vision using a fixed-focus camera, no adjustment of the focus position is required, so the parameters in the conversion equations for converting the image information of the workpiece into spatial position information do not need to be changed.

[0010] On the other hand, a vision robot using a camera with a focusing function can acquire a sharp image by focusing based on the camera's position relative to a workpiece (the distance between the workpiece and the camera). However, when adjusting the focus position, for example, it is necessary to change the parameters used in the conversion equations for converting the image information obtained by capturing an image of the workpiece into spatial position information each time.

[0011] In particular, since it is necessary to change the parameters used in the conversion formula when changing the focus position (focal length) of a lens, the parameters used in the conversion equations should be changed. Therefore, it may be necessary to specify the parameters used in the conversion equations for each of the focus positions in advance.

[0012] As a result, there is a risk that the processing for acquiring the image of the workpiece (object) will become complicated, or that the processing for determining the spatial position of the object will take a long time. Furthermore, the focusing of the lens of the camera (image acquisition device), for example, may be performed by a drive unit that changes the positional relationship of the lens. Therefore, errors in the position of the lens are likely to occur, and the accuracy of converting the image information into the spatial position information of the object may be reduced.

[0013] The object to be achieved by the present invention is to provide an image processing apparatus, an image processing system and an image processing program capable of performing image processing to obtain spatial position information of an object based on image information in a short time and with high accuracy. (SOLUTION OF THE PROBLEM)

[0014] According to one embodiment of the present invention, an image processing device is provided for obtaining spatial position information of an object based on image information output from an image capturing device including a lens configured to capture an image of the object. The image processing device includes an aperture control unit configured to control an aperture of the lens. When a distance from the image capturing device to the object is a first distance, a focus position of the lens is adjusted and fixed, the aperture control unit sets the aperture of the lens to a first aperture, and the image capturing device captures the object to obtain a first image when a distance from the image capturing device to the object is a first distance.When the distance between the image capture device and the object is a second distance, the aperture control unit sets the aperture of the lens to a second aperture, and the image capture device captures the object to obtain a second image.

[0015] The objects and effects of the present invention will be realized and attained through the use of the components and combinations recited in the claims. Both the above general description and the following detailed description are exemplary and descriptive, and are not limiting of the invention described in the claims. BRIEF DESCRIPTION OF THE DRAWINGS [ Fig. 1] Fig. 1 is a block diagram illustrating an industrial robot system as an example to which an image processing apparatus according to the present embodiment is applied. [ Fig. 2] Fig. 2 is a diagram for explaining the parameters of conversion equations used to obtain spatial position information of an object based on image information obtained from the Fig. 1 shown image capture device. [ Fig. 3] Fig. 3 is a diagram for explaining the relationship between a lens aperture and a depth of field in the Fig. 1 shown industrial robot system. [ Fig. 4] Fig. 4 is a diagram showing an example of a lighting device used in the Fig. 1 shown industrial robot system. [ Fig. 5] Fig. 5 is a block diagram illustrating an image processing system to which an example of the image processing apparatus according to the present embodiment is applied. [ Fig. 6] Fig. 6 is a flowchart for explaining an example of processing of an image processing program executed by a processing unit (arithmetic processing unit) of the Fig. 5 shown image processing device. DESCRIPTION OF THE EMBODIMENTS

[0016] First, an example of an industrial robot system in which an image processing apparatus according to the present embodiment is applied will be described with reference to Fig. 1 to Fig. 4 before describing in detail an image processing apparatus, an image processing system, and an image processing program according to the present embodiment. Fig. 1 is a block diagram showing an industrial robot system as an example to which an image processing apparatus according to the present embodiment is applied.

[0017] In Fig. In FIG. 1, reference numeral 100 denotes an industrial robot, 110 a hand unit, 200 a robot control unit, and 3 an image processing device. Furthermore, reference numeral 1 denotes an image acquisition device (camera), 2 an illumination device (ring illumination), 4 a workbench, and 5 an object (workpiece). Note that a robot image processing system is composed, for example, of the camera 1, the illumination device 2, and the image processing device 3.

[0018] Fig. 1 shows a case in which the hand unit 110 of the industrial robot 100 is moved according to a teaching route that has been previously set in order to operate the hand unit 110 to grip a workpiece 5. In this case, Fig. 1 a position of the workpiece 5 placed on the workbench 4 deviated from an original position (position based on the learning path) 5a indicated by a dashed line.

[0019] Specifically, Fig. 1 illustrates a case in which the hand unit 110 is moved according to a predetermined learning route and the workpiece 5 placed on the workbench 4 is grasped by the hand unit 110, but an actual spatial position of the workpiece 5 on the workbench 4 deviates from an assumed starting position 5a. In this case, the camera 1 and the lighting device 2 are mounted near the hand unit 110 of the industrial robot 100 and moved together with the hand unit 110 so that the workpiece 5 placed on the workbench 4 can be captured by the camera 1.

[0020] At this time, the image processing device 3 obtains information about the actual spatial position of the workpiece 5 based on image information of the workpiece 5 on the workbench 4 acquired by the camera 1, and detects a deviation from an assumed position 5a. Furthermore, the image processing device 3 detects and corrects the deviation between the assumed position 5a and the actual spatial position of the workpiece 5 via the robot controller 200 and controls the hand unit 110 to correctly grip the workpiece 5.

[0021] In this way, the industrial robot 100 can perform correct operation even if the position of the workpiece 5 on the workbench 4 deviates by applying robot vision to the industrial robot system. Fig. 1, the image processing device 3 is provided in the robot control device 200, but it may also be provided outside the robot control device 200 as a separate body.

[0022] In the Fig. In the industrial robot system illustrated in Figure 1, the workpiece 5 is placed on the workbench 4, and the camera 1 is mounted near the hand unit 110 (drive) of the industrial robot 100 to move together. It should be noted that the image processing device 3 according to this embodiment can be widely applied to various machines and robots or the like in which at least one of the two components, the camera 1 and the workpiece 5, is gripped by a motion-controllable drive.

[0023] Fig. 2 is a diagram for explaining the parameters of conversion equations used to obtain spatial position information of an object based on image information obtained from the Fig. 1 shown image capture device. In Fig. 2, Equation (I) is a relationship between a spatial position and a position on an image by projecting three-dimensional coordinates in a so-called pinhole camera model onto an image plane using a perspective projection transformation. Furthermore, Equation (II) is an extended relationship considering distortions of an actual lens of the camera 1 in a radial direction, a circumferential direction, and the like.

[0024] In equation (I), the reference symbol AA represents a position on an image (position of a workpiece 5 in an image captured by camera 1), BB represents a focal length and an image center of camera 1, CC represents camera coordinates in a space, and DD represents a spatial position (position of workpiece 5 in an actual space). Here, the reference symbols (X, Y, Z) represent 3D coordinates in a world coordinate system, (u, v) represent the coordinates of a point projected onto an image plane, (cx, cy) represent principal points (usually an image center), and fx, fy represent focal lengths expressed in pixels.

[0025] In equation (II), reference symbol t represents [t1, t2, t3], R represents a 3×3 r matrix, and k1 to k6, and p1, p2 represent parameters. Therefore, the position of the workpiece 5 in actual space can be obtained by converting the position of the workpiece 5 in the image captured by the camera 1 based on equation (II). At this time, it is possible to obtain the spatial position DD from the position AA on the image by preparing in advance the camera coordinates CC in space, the focal length and image center BB, and the distortion parameters (k1 to k6 and p1, p2: 6 types of k and 2 types of p).

[0026] Here, when changing the focus position of the lens, the focal length f changes each time the focus position is changed, and therefore, the parameters (6 types of k and 2 types of p) corresponding to the focal length f may need to be determined in advance. Especially in an image processing system that uses a camera with a focus function, it is necessary to change the parameters used in the conversion equations when the focus position (focal length) of a lens changes.

[0027] As described above, when using the camera with the focus function, it is necessary to obtain parameters corresponding to the focus position at the time of imaging the workpiece in advance, so the processing becomes complicated when imaging the workpiece or the processing may cause problems such as elongation. In addition, since the focusing of the lens is performed by the drive unit to change the position of the lens, errors in the position of the lens are easily caused, so a problem of reducing the accuracy of converting the spatial position information of the workpiece from the image information may also be caused.

[0028] Fig. 3 is a diagram for explaining the relationship between a lens aperture and a depth of field in the Fig. 1 and to explain the relationship between an aperture (aperture value, aperture amount) and the depth of field of a camera 1. In Fig. 3, the reference symbol L stands for the distance to a recording object (workpiece 5), δ for the permissible circle of confusion (constant determined by the resolution), f for the focal length and F for the aperture value (f-stop).

[0029] It should be noted that the depth of field D represents a range of sharpness and is determined by the sum of the front depth of field Df and the rear depth of field D r is determined: (D = Df + Dr), where Df and Dr are expressed as follows: Df=(δFL2) / (f2+δFL) Dr=(δFL2) / (f2+δFL)

[0030] Under the condition that f, δ, and L are constant, the depth of field D increases as the aperture value F is increased, and the amount of light incident on a picture element (CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) image sensor) decreases. Specifically, the amount of light incident on the image sensor can be expressed as C / F 2 where F is an aperture value and C is a constant. So, for example, if the aperture value F is doubled, the amount of light reaching the image sensor is 1 / 4.

[0031] Furthermore, a method for adjusting an exposure time of an image to adjust the brightness of the image is provided, and when the exposure time is doubled, an image with twice the brightness can be obtained. Therefore, it can be considered that the depth of field D of the lens is increased by increasing the aperture value F of the lens or by expanding the depth of field of the lens (field of focus) in accordance with compensation for a decrease in the incident light intensity by increasing the exposure time.

[0032] For example, if the aperture value F of the lens is doubled to increase the depth of field D of the lens, an image with the same brightness can be obtained by increasing the exposure time fourfold. Note that the exposure time t is given by t = t0 × (F / F)0 2where t0 represents an exposure time before the change in F, F0 represents an aperture value before the change in F, and F represents an aperture value after the change.

[0033] As explained later with reference to Fig. 5 and Fig. As described in detail in FIG. 6, in an example of an image processing apparatus according to the present embodiment, when a distance of a camera 1 to a workpiece 5 is a first distance, a focus control unit 312 adjusts and fixes a focus position of a lens. Further, an aperture control unit 311 controls and sets an aperture of the lens to a first aperture, and a first image is obtained by capturing the workpiece 5.

[0034] When the distance between the camera 1 and the workpiece 5 is a second distance, the focus position of the lens is maintained unchanged, and the aperture control unit 311 sets the aperture of the lens to a second aperture so that the workpiece 5 is included in a depth of field (D = Df + Dr). When the distance between the workpiece 5 and the camera 1 is the second distance, the workpiece 5 is focused (brought into focus) by the depth of field of the lens whose aperture is set to the second aperture, and the workpiece 5 is captured at the second distance.

[0035] As described above, in the example of the image processing device of the present embodiment, for example, when the distance between the camera 1 and the workpiece 5 is different, the focus position of the lens is fixed at both the first distance and the second distance, and the aperture of the lens is adjusted at the second distance. Therefore, the first image and the second image of the workpiece 5 can be captured at both the first distance and the second distance, which are different distances between the camera 1 and the workpiece 5.

[0036] When the focus position of the lens with a focus function is changed, it is necessary to change the parameters based on the focus position. However, in the present embodiment, the focus positions of the first and second distances are fixed, so no parameter changes are required. In particular, when detecting both the first distance for acquiring the first image and the second distance for acquiring the second image, it is not necessary to prepare new parameters, and the same parameters can be used. Therefore, according to the image processing apparatus, the image processing system, and the image processing program of the present embodiment, it is possible to perform image processing for obtaining spatial position information of an object 5 based on image information in a short time and with high accuracy.

[0037] Fig. 4 is a diagram showing an example of a lighting device used in the Fig. 1 shown industrial robot system, where Fig. 4(a) illustrates a circular lighting device 2a as an example of the lighting device 2, and Fig. 4(b) illustrates a ring illumination device 2b as another example of the illumination device 2.

[0038] As in Fig. 4(a), the illumination device 2 may be, for example, the circular illumination device 2a provided near the camera 1. As shown in Fig. However, as shown in Fig. 4(b), the illumination device 2 may be, for example, the ring illumination device 2b provided on the outer periphery of the lens of the camera 1.

[0039] Here, the circular illumination device 2a and the ring illumination device 2b may always emit a constant amount of irradiation light, but they may be capable of controlling the intensity of the irradiation light, which will be described in detail later. Specifically, when the distance between the workpiece 5 and the camera 1 is the second distance, the illumination device 2 (2a, 2b) may increase the intensity of the irradiation light by increasing an aperture value (f-stop) to compensate for a reduced exposure amount and focus on the workpiece 5. For example, when the aperture value F is doubled to increase the depth of field of the lens, the illumination control unit 313 adjusts the intensity of the light emitted by the illumination device 2 to four times, so that an image with the same brightness can be captured.

[0040] Note that the illumination device 2 is not limited to the circular illumination device 2a and the ring illumination device 2b, but various illumination devices can be mounted thereon. Furthermore, it goes without saying that the number and position of the illumination devices are not limited to providing only one illumination device near the camera 1.

[0041] Here, the workpiece 5 is illuminated with the irradiation light of the illumination device 2 (circular illumination device 2a and ring illumination device 2b), and when determining a distance from the camera 1 to the workpiece 5 at r and a constant value at C, the relationship C / r is obtained. 2 . Therefore, as described above, if the aperture (F) of the lens is doubled, the amount of light entering the camera 1 becomes 1 / 4.

[0042] Taking into account the relationship to the aperture (t = t0 × (F / F)0 2) an exposure time t can be defined as t = t0 × (F / F)0 2 × (r / r)0 2 Here, the reference t0 represents an exposure time before changing an aperture of the lens, F0 represents an aperture before changing the aperture of the lens, F represents an aperture after changing the aperture of the lens, r represents a distance before changing the aperture of the lens, and r0 represents a distance after changing the aperture of the lens. Specifically, in order to compensate for the exposure, there are a technique for adjusting the exposure time of the image and a technique for adjusting the intensity of irradiation light by the illuminator.

[0043] Hereinafter, examples of an image processing apparatus, an image processing system, and an image processing program according to the present embodiment will be described in detail with reference to the accompanying drawings.

[0044] Fig. 5 is a block diagram showing an image processing system to which an example of the image processing apparatus according to the present embodiment is applied. Here, the image processing system to which an example of the image processing apparatus according to the present embodiment is applied corresponds, for example, to a robot image processing of an industrial robot system described with reference to Fig. 1. It should be noted that the image processing device 3 may be integrated into the robot controller 200 for controlling the industrial robot 100, but may also be provided separately outside the robot controller 200, as described above.

[0045] Furthermore, in the Fig. In the image processing system shown in Figure 5, at least one of the image capturing device (camera) 1 and the object (workpiece) 5 is gripped by a motion-controllable drive, and the position of the camera 1 relative to the workpiece 5 can be controlled to move. In other words, the image processing device 3 according to the present embodiment can be widely applied to various machines and robots in which the camera 1 and / or the workpiece 5 is gripped by the motion-controllable drive.

[0046] As in Fig. 5, the image processing device 3, which is used to obtain spatial position information of the workpiece 5 based on image information from a camera 1 having a lens (not shown) for imaging the workpiece 5, includes a processing unit (arithmetic processing unit) 31 and a storage unit 32. Reference numeral 2 here denotes an illumination device for irradiating light onto the workpiece 5. In this embodiment, the illumination device 2 is controlled by an image processing device 3, as described later. For example, the workpiece 5 can always be irradiated with a constant amount of light.

[0047] The processing unit 31 includes an aperture control unit 311, a focus control unit 312, an illumination control unit 313, a feature extraction unit 314, a position calculation unit 315, and an exposure control unit 316. The aperture control unit 311 controls an aperture of the lens of the camera 1 that captures the workpiece 5, and the focus control unit 312 controls a focus of the lens of the camera 1. The illumination control unit 313 controls the intensity of the light of the illumination device 2, that is, the intensity of the radiant light emitted by the illumination device 2 that illuminates the workpiece 5.

[0048] The storage unit 32 stores an image processing program 320 to be executed by the processing unit 31. The image processing program 320 includes a feature storage unit (feature information) 321, a parameter storage unit (parameter information) 322, and an aperture storage unit (aperture information) 323. The feature storage unit 321 stores, for example, feature information of the workpiece 5 to be detected from the image information of the workpiece 5 captured by the camera 1. The parameter storage unit 322 stores, for example, parameter information used in conversion equations to obtain a spatial position of the workpiece 5 from focus position information (focus information) and image information. The aperture storage unit 323 stores, for example,the relationship (aperture information) between the distance between the workpiece 5 and the camera 1 (distance of the camera 1 to the workpiece 5) and the aperture value (aperture value).

[0049] The feature extraction unit 314 extracts the feature information stored in the feature storage unit 321 from the image information of the workpiece 5 captured by the camera 1. The position calculation unit 315 calculates the spatial position of the workpiece 5 based on the image information of the workpiece 5 captured by the camera 1 and the parameters obtained from the parameter storage unit 322.

[0050] For example, when the workpiece 5 and the camera 1 are at the first distance, the position calculation unit 315 reads parameters from the parameter storage unit 322 corresponding to the focus position of the lens adjusted and fixed by the focus control unit 312. Furthermore, the position calculation unit 315 calculates the spatial position of the workpiece 5 from the first image at the first distance captured by the aperture control unit 311 with the aperture of the lens as the first aperture.

[0051] Note that the parameters obtained when the workpiece 5 and the camera 1 are at the first distance are also used when the workpiece 5 and the camera 1 are at the second distance. For example, when the workpiece 5 and the camera 1 are at the second distance, the position calculation unit 315 uses the same parameters as when the workpiece 5 and the camera 1 are at the first distance, and the aperture control unit 311 sets the aperture of the lens to the second distance. At this time, the aperture control unit 311 controls the aperture amount of the camera 1 by referring to the aperture information stored in the aperture storage unit 323.

[0052] Specifically, when the hand unit 110 is moved closer to the workpiece 5 on the workbench 4 from a long distance, the aperture control unit 311 increases the aperture value to focus the workpiece 5 from the second distance at the focused focus position at the time of the first distance to increase the depth of field. That is, when the first distance > the second distance, the aperture control unit 311 controls to input the depth of field by increasing the aperture of the lens at the fixed focus position when the workpiece and the camera are at the first distance, so that the workpiece and the camera are focused on the workpiece 5 even at the second distance.

[0053] The exposure control unit 316 adjusts the exposure time (shutter speed) of the camera 1, and when the workpiece 5 and the camera 1 are at the second distance, the aperture is increased to deepen the depth of field and focus on the workpiece 5. Specifically, when the workpiece 5 and the camera 1 are at the second distance, and when the workpiece 5 and the camera 1 are at twice the aperture size of the first distance, the exposure control unit 316 controls the exposure time to be four times longer than the time at which the first image was captured.

[0054] The illumination control unit 313 adjusts the intensity (amount of light) of the light emitted by the illumination device 2 and, when the workpiece 5 and the camera 1 are at the second distance, compensates for a reduction in exposure corresponding to an increased aperture to focus on the workpiece 5 by increasing the depth of field. Specifically, the illumination control unit 313 controls the illumination device when the workpiece 5 and the camera 1 are at the second distance, while doubling the aperture compared to the first distance between the workpiece 5 and the camera 1, to control the intensity of the light emitted by the illumination device 2 to be four times that when the first image was captured.

[0055] If there is a problem with extending the exposure time, it may be advantageous to increase the intensity of the light emitted from the illumination device 2 by the illumination control unit 313 without extending the exposure time of the camera 1 by the exposure control unit 316. On the other hand, if it is difficult to increase the intensity of the irradiation light from the illumination device 2, it may be advantageous to increase the exposure time of the camera 1 by the exposure control unit 316 without increasing the intensity of the irradiation light from the illumination device 2 by the illumination control unit 313.

[0056] Furthermore, the exposure reduction compensation performed when the workpiece 5 and the camera 1 are at the second distance is not limited to being performed by the exposure control unit 316 or the illumination control unit 313, but can be performed by either the exposure control unit 316 or the illumination control unit 313. At this time, the exposure time compensation by the exposure control unit 316 and the exposure amount compensation by the illumination control unit 313 can also be divided into predetermined ratios.In particular, when it is necessary to limit the squeezing amount of the first distance at the second distance, for example, it is also possible to control the exposure amount of the second image (intensity of irradiated light) to twice that of the first image while controlling the exposure time of the second image to twice that of the first image.

[0057] It should be noted that the above-described first distance and the second distance are not limited to the case where the first distance > the second distance, that is, they are not limited to the case where the hand unit 110 to which the camera 1 or the like is attached is moved from a far portion to a near portion of the workpiece 5. Furthermore, the image processing apparatus (image processing system) of this example can also be applied when the first distance = second distance, that is, the case where the hand unit 110 to which the camera 1 or the like is attached is moved from a near portion to a far portion of the workpiece 5. Moreover, the image processing apparatus of this example can also be applied to the case where the first distance < the second distance, that is, the hand unit 110 to which the camera 1 or the like is attached is moved far away from the near portion of the workpiece 5.

[0058] Specifically, when the first distance = the second distance, the workpiece 5 can be focused as it is even if the workpiece 5 and the camera 1 are at the second distance. When the first distance < the second distance, the workpiece 5 can be focused even if the depth of field is shallow when the workpiece 5 and the camera 1 are at the second distance. In this case, with reference to the obtained aperture information, the aperture of the camera 1 can be decreased, and the exposure control unit 316 and / or the illumination control unit 313 can compensate (equalize) the increase in exposure corresponding to the decrease in the aperture amount.

[0059] As described above, according to an example of the image processing apparatus of the present embodiment, the focus position of the lens is fixed when the distance between the camera 1 and the workpiece 5 is both the first distance and the second distance. This makes it possible to calculate the spatial position information of the workpiece 5 from the image information using the same parameters for the first image captured at the first distance and the second image captured at the second distance. Consequently, image processing for obtaining the spatial position data of a workpiece based on the image data can be performed in a short time with high accuracy.It should be noted that in the image processing apparatus (image processing system) according to the present embodiment, the camera 1 is not limited to an autofocus camera as long as it has a lens focusing function, and it is of course possible for a camera to perform focusing manually.

[0060] Fig. 6 is a flowchart for explaining an example of processing of an image processing program executed by a processing unit (arithmetic processing unit) of the Fig. 5 shown image processing device. Fig. Fig. 6 explains a case where the hand unit 110 is previously taught to move from a distant position to a near position of the workbench 4 (first distance > second distance) in the industrial robot system described with reference to Fig. 1. Note that the workpiece 5 is placed on the workbench 4, and the camera 1 and the lighting device 2 are mounted near the hand unit 110 of the industrial robot 100.

[0061] As in Fig. 6, when the example of processing the image processing program is started (START), in step ST1, for example, the camera 1 (illumination device 2) mounted near the hand unit 110 of the industrial robot 100 is moved to a position at a first distance from the workpiece 5. At this time, a movement operation of the hand unit 110 to the position of the first distance from the workpiece 5 is performed based on a movement route previously taught by the robot controller 200.

[0062] In the next step ST2, the focus control unit 312 adjusts and fixes a focus of the lens of the camera 1 to focus the workpiece 5 at a position of the first distance. Further, in step ST3, the aperture control unit 311 adjusts an aperture of the lens of the camera 1 at a position of the first distance, and the process proceeds to step ST4.

[0063] In step ST4, a position conversion parameter corresponding to the first distance is obtained from the image and set in the parameter storage unit 322, and the process proceeds to step ST5, where the camera 1 captures the first image at the position of the first distance. Then, in step ST6, the first process is executed based on the first image.

[0064] In this first method, for example, based on the first image with a wide view angle including the circumference of the workpiece 5, the parameters set in step ST4 and the conversion equations are used to convert the image information of the workpiece 5 into spatial position information to obtain the spatial position information of the workpiece 5. Then, the feature of the workpiece 5 read by the feature extraction unit 314 from the feature storage unit 321 is applied to the obtained spatial position information of the workpiece 5, and the hand unit 100 is moved, for example, to a second distance position closer to the workpiece 5. The calculation or modification of the movement route for the movement 110 is performed.

[0065] In the next step ST7, the hand unit 110 is moved, for example, based on the movement route calculated or corrected in step ST6. That is, the movement of the hand unit 110 in step ST7 is performed by correcting a previously learned movement route based on the spatial position information of the workpiece 5 obtained in the first process in step ST6, for example. It may be moved to a position a second distance from the position. Note that the movement of the hand unit 110 in step ST7 may also be performed according to the predetermined movement route regardless of the spatial position information of the workpiece 5 obtained by the first process in step ST6. These are selected, for example, according to the required accuracy of the system in which the industrial robot 200 is used, the magnitude of the expected positional deviation of the workpiece 5, and the like.

[0066] Further, the process proceeds to step ST8, the aperture control unit 311 adjusts the aperture of the lens at the second distance position, and the process proceeds to step ST9. Here, in step ST8, a focus position set on the lens is adjusted to focus the workpiece 5 at the first distance position, and the fixed position is used as is.

[0067] That is, in step ST8, only the aperture of the lens is changed by the aperture control unit 311, and the workpiece 5 is adjusted to be located at the second distance within the depth of field (so that it is in focus). The exposure reduction compensation performed when the workpiece 5 and the camera 1 are at the second distance can be performed by either the exposure control unit 316 or the illumination control unit 313, but as described above, both the exposure control unit 316 and the illumination control unit 313 can be used.

[0068] In step ST9, for example, at the second distance position, a second image is captured with a viewing angle where the workpiece 5 occupies most of the screen. Since the focus position of the lens is fixed, the parameters used to obtain the spatial position information of the workpiece 5 from the image information of the second image can be the same as the spatial position information of the workpiece 5 from the image information of the first image.

[0069] This means that at the second distance, it is not necessary to obtain new parameters, and the processing time for obtaining the spatial position of the workpiece can be shortened. Furthermore, the focus position of the lens is fixed at the first distance even at the second distance, and therefore it is also possible to improve the accuracy of converting the spatial position information of the workpiece.

[0070] Furthermore, for example, in step ST10, the spatial position information of the workpiece 5 is calculated with the same parameters as in the processing of the first image on the basis of the image information of the second image in which the workpiece 5 is enlarged, and the process is ended (END).

[0071] In the flowchart described above, for example, in step ST7, the robot is moved to the second position according to the predetermined movement route, and in step ST10, the first process based on the first image and the second process based on the second image 2 may also be performed. Specifically, when the first distance = the second distance, and the first image and the second image are, for example, images that capture the workpiece 5 from different angles, it is possible to improve the calculation accuracy by calculating the spatial position of the workpiece 1 by processing such two images.

[0072] Furthermore, the spatial position of the workpiece 1 can also be calculated by processing the first and second images at different positions and angles. Furthermore, the image to be imaged is not limited to the first and second images, and the same processing can be performed to obtain the spatial position of the workpiece 5 by imaging the third image, the fourth image, and so on. Needless to say, to determine the position (spatial position information of the object) of the workpiece 5 from the captured image(s) (image information of the object), for example, various known image processing techniques can be applied and performed.

[0073] It should be noted that the image processing program or application software described above can be recorded and provided in a computer-readable, non-temporary recording medium or a non-volatile semiconductor storage device, or can be provided via wired or wireless communication. Examples of computer-readable, non-temporary recording media include an optical disk such as a CD-ROM (Compact Disc Read Only Memory) or a DVD-ROM, or a hard disk, etc. Other non-volatile semiconductor memories include a PROM (Programmable Read Only Memory), a Flash Memory (registered trademark), and the like. Furthermore, distribution from the server device can be via a wired or wireless WAN (Wide Area Network), LAN (Local Area Network), or the Internet.

[0074] As described above in detail, the image processing apparatus, the image processing system, and the image processing program according to the present embodiment can perform image processing for obtaining spatial position information of an object based on image information in a short time with high accuracy.

[0075] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. These embodiments include various additions and replacements without departing from the gist of the invention, or without departing from the idea and spirit of the invention, which are evident from the content described in the claims and their equivalents, and changes, partial deletion, and the like are possible. In the embodiments described above, for example, the order of each operation and the order of each method are exemplified and are not limited to them. The same applies when numerical values ​​or equations are used in the description of the embodiments described above. REFERENCE SYMBOL LIST 1 image capture device (camera) 2 lighting device 2a Circular lighting device 2b Ring lighting device 3 Image processing device 4 workbench 5 Target object (workpiece) 31 Processing unit (arithmetic processing unit) 32 storage units 100 industrial robots 110 Hand Unit 200 robot control unit 311 Aperture control unit 312 Focus control unit 313 Lighting control unit 314 Feature extraction unit 315 Unit for position calculation 316 Exposure control unit 320 image editing program 321 Unit for storing features (feature information) 322 Unit for parameter storage (parameter information) 323 Aperture storage unit (aperture information) QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2019-113680

[0006] JP 2019-161553

[0006] JP 2006-253998

[0006]

Claims

[1] An image processing apparatus for obtaining spatial position information of an object based on image information output from an image capturing apparatus including a lens configured to capture an image of the object, comprising: an aperture control unit configured to control an aperture of the lens, wherein a focus position of the lens is adjusted and fixed, the aperture control unit sets the aperture of the lens to a first aperture, and the image capture device captures the object to obtain a first image when a distance from the image capture device to the object is a first distance; and the aperture control unit sets the aperture of the lens to a second aperture, and the image capture device captures the object to obtain a second image when the distance from the image capture device to the object is a second distance. [2] The image processing apparatus according to claim 1, further comprising: a focus control unit configured to control the focus position of the lens, wherein the focus control unit adjusts and fixes the focus position of the lens when the distance between the image capturing device and the object is the first distance. [3] The image processing apparatus according to claim 1 or 2, further comprising: a storage unit configured to store an image processing program for obtaining the spatial position information of the object with respect to the first image and the second image. [4] The image processing apparatus according to any one of claims 1 to 3, further comprising: an exposure control unit configured to control an exposure amount of the image capturing device, wherein the exposure control unit controls the exposure amount of the image capturing device based on the second aperture of the lens when the focus position of the lens is at the second distance from the image capturing device to the object. [5] The image processing apparatus according to any one of claims 1 to 4, further comprising: an illumination control unit configured to control the intensity of irradiation light output from an illumination device located near the image capturing device, the illumination control unit controlling the intensity of the irradiation light output from the illumination device based on the second aperture of the lens when the focus position of the lens is at the second distance from the image capturing device to the object. [6] The image processing device according to any one of claims 1 to 5, wherein at least one of the image capturing devices and the object is gripped by a motion-controllable drive. [7] The image processing apparatus according to any one of claims 1 to 6, wherein a length of the first distance is greater than a length of the second distance, and a value of the first aperture is smaller than a value of the second aperture. [8] The image processing apparatus according to any one of claims 1 to 7, wherein the second distance is determined with respect to the first image, and the information about the spatial position of the object is obtained with respect to the second image. [9] The image processing apparatus according to any one of claims 1 to 8, wherein the information about the spatial position of the object is obtained with respect to the first image and the second image. [10] The image processing apparatus according to any one of claims 1 to 9, wherein the second aperture is determined with reference to the first distance and the second distance and previously generated aperture information. [11] An image processing system comprising: the image processing apparatus according to any one of claims 1 to 10; and the image capture device. [12] An image processing program of an image processing device for obtaining spatial position information of an object based on image information output from an image capturing device including a lens configured to capture an image of the object, the image processing program causing an arithmetic processing unit to execute: Adjusting and fixing a focus position of the lens, adjusting the aperture of the lens to a first aperture, and capturing the object to obtain a first image when a distance from the image capturing device to the object is a first distance, and Setting the aperture of the lens to a second aperture and capturing the object to obtain a second image when the distance between the image capture device and the object is a second distance.

Images