Image processing device

Abstract

In one embodiment, this image processing device comprises a steady image generation unit that generates a steady image as an image by steady light in which brightness is maintained, on the basis of a plurality of images acquired from a visual sensor that images an object. The image processing device comprises an unstable image generation unit that generates an unstable image as an image by unstable light corresponding to a brightness change portion, on the basis of the plurality of images acquired from the visual sensor.

Description

Image processing device

[0001] The present disclosure relates to an image processing device.

[0002] In conventional technology, it is known to detect an object by processing an image of the object captured by a camera. In particular, it is known to estimate the dimensions of the object to inspect the object or to detect the position of the object in a work space. For example, an image processing device can detect the contour of the object based on the pixel values ​​of each pixel in the image captured by the camera. Then, it is possible to perform dimensional inspection based on the contour of the object. Alternatively, it is possible to detect a feature of the object based on the detected contour and a reference image of the feature created in advance. Then, it is possible to detect the position of the workpiece based on the position of the feature. For example, a luminance value indicating the brightness of the pixel can be used as the pixel value of a pixel.

[0003] The pixel value of an image pixel depends not only on the performance of the camera capturing the image of the workpiece but also on the brightness of the surroundings of the object. For example, if the surroundings of the object are bright, the brightness value of the pixel will also be high. In conventional technology, it is known to install a lighting device to illuminate the object when the brightness of the work space is insufficient.

[0004] Japanese Patent Application Laid-Open No. 2017-125716

[0005] In order for an image processing device to accurately detect an object, it is preferable that the work space be sufficiently bright. It is also preferable that the light illuminating the object be constant regardless of time. If the brightness or brightness distribution on the surface of the workpiece changes, the image processing device may not be able to correctly detect the object. For example, the contour of the object can be detected by changes in pixel values. However, if the pixel values ​​are unstable or change, the contour of the object may not be correctly detected. In such cases, the image processing device may determine that an abnormality has occurred in the image processing and stop processing.

[0006] For example, light may enter a building through a window. The brightness of a work space within a building varies between morning, noon, and night. The brightness also changes depending on the weather. Furthermore, if the building where work is performed is lit with fluorescent or mercury lamps, flickering occurs according to the power supply frequency. This means that the light periodically brightens and darkens. In such situations, it may not be possible to accurately detect an object using image processing.

[0007] However, there was a problem in that it was difficult for workers to evaluate changes in pixel values ​​in images just by looking at the images they had taken.Also, even if light coming in through factory windows was blocked or lighting was placed that directly illuminated the target, it was difficult to determine whether the countermeasures were sufficient just by looking at the images of the target.

[0008] An image processing device according to one aspect of the present disclosure includes a steady-state image generating unit that generates a steady-state image as an image created by steady light whose brightness is maintained, based on a plurality of images acquired from a visual sensor that captures an image of an object. The image processing device also includes an unstable image generating unit that generates an unstable image as an image created by unstable light corresponding to a change in brightness, based on the plurality of images acquired from the visual sensor.

[0009] 1 is a perspective view of a robot device according to an embodiment; FIG. 2 is a block diagram of a robot device according to an embodiment; FIG. 3 is a schematic diagram illustrating lighting arranged in a building and light entering through windows; FIG. 4 is a flowchart of image processing control according to a first embodiment; FIG. 5 is a diagram illustrating control for generating a steady image and an unstable image from a captured image according to the first embodiment; FIG. 6 is a diagram illustrating control for generating a comparison image from a steady image and an unstable image from a captured image according to a second embodiment; FIG. 7 is a diagram illustrating control for generating a steady image, an unstable image, and a comparison image from a captured image according to a third embodiment; FIG. 8 is a diagram illustrating control for multiplying pixel values ​​of an unstable image by a predetermined constant according to a fourth embodiment; and FIG. 9 is a diagram illustrating control for generating a binary image from a comparison image according to a fifth embodiment.

[0010] First Embodiment An image processing device and a robot device including the image processing device according to a first embodiment will be described with reference to Figures 1 to 6. The robot device of this embodiment performs a task of transporting a workpiece. The robot device of this embodiment captures an image of the workpiece with a camera, corrects the position and posture of the robot, and then grasps the workpiece.

[0011] 1 is a perspective view of a robot device 5 according to the present embodiment. The robot device 5 includes a robot 1 and a hand 2 as a work tool. The robot device 5 also includes a control device 4 that controls the robot 1 and the hand 2. The robot 1 according to the present embodiment is an articulated robot including multiple joints.

[0012] The robot 1 of this embodiment includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a swivel base 13. The swivel base 13 is supported by a base 14. The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 to which the hand 2 is fixed. These components of the robot 1 are configured to rotate around predetermined drive axes. The robot of this embodiment has six drive axes, but is not limited to this form. Any robot that can change the position and orientation of a work tool can be used.

[0013] In this embodiment, the workpiece 85 is formed in a rectangular parallelepiped shape. An outer edge 85a, which is the outline of the top surface of the workpiece 85, has a rectangular shape. A recess 85b, which has a rectangular planar shape, is formed on the top surface of the workpiece 85.

[0014] The hand 2 serving as the work tool in this embodiment grips and releases the workpiece 85. The work tool is not limited to a hand, and any work tool can be used depending on the work to be performed by the robot device. For example, a robot device that performs spot welding can use a spot welding gun as the work tool.

[0015] A block diagram of the robot device according to this embodiment is shown in Figure 2. Referring to Figures 1 and 2, the robot 1 includes a robot drive device that changes the position and posture of the robot 1. The robot drive device includes a robot drive motor 22 that drives components such as an arm and a wrist. When the robot drive motor 22 is driven, each component of the robot 1 rotates around its drive shaft.

[0016] The robot device 5 includes a hand driving device that drives the hand 2. The hand driving device includes a hand driving motor 21 that drives the hand 2. In this embodiment, the claws of the hand 2 open and close when driven by the hand driving motor 21. The hand may be configured to be driven by air pressure or the like.

[0017] The control device 4 includes an arithmetic processing device (computer) having a CPU (Central Processing Unit) as a processor, and RAM (Random Access Memory), ROM (Read Only Memory), and the like connected to the CPU via a bus. The control device 4 also includes a storage unit 42 that stores information related to the control of the robot device 5. The storage unit 42 can be configured with a non-transitory storage medium capable of storing information. For example, the storage unit 42 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium.

[0018] An operation program 41 created in advance for operating the robot 1 is input to the control device 4. The operation program 41 is stored in a storage unit 42. The robot device 5 of this embodiment transports a workpiece 85 based on the operation program 41. The robot 1 can automatically transport the workpiece 85 to a predetermined position.

[0019] The control device 4 includes an operation control unit 43 that generates operation commands. The processor functions as the operation control unit 43 by performing predetermined control based on the operation program 41. The operation control unit 43 is configured to be able to read information stored in the memory unit 42. The operation control unit 43 sends operation commands to the robot driving unit 45 to drive the robot 1 based on the operation program 41. The robot driving unit 45 includes an electric circuit that drives the robot driving motor 22. The robot driving unit 45 supplies electricity to the robot driving motor 22 based on the operation commands.

[0020] The operation control unit 43 sends an operation command to the hand driving unit 44 to drive the hand 2 based on the operation program 41. The hand driving unit 44 includes an electric circuit that drives the hand driving motor 21. The hand driving unit 44 supplies electricity to the hand driving motor 21 based on the operation command.

[0021] The robot 1 includes a state detector for detecting the position and orientation of the robot 1. In this embodiment, the state detector includes a position detector 18 attached to a robot drive motor 22 corresponding to the drive shaft of a component such as an arm. The position detector 18 is configured, for example, by an encoder. The output of the position detector 18 makes it possible to obtain the orientation of the component for each drive shaft. The position and orientation of the robot 1 are detected based on the output of the position detector 18.

[0022] The robot device 5 of this embodiment includes a camera 6 as a visual sensor for detecting a workpiece 85. The camera 6 of this embodiment is a camera that captures two-dimensional monochrome images. The camera 6 of this embodiment is configured to be able to capture grayscale images. For example, the camera 6 is configured to be able to capture images with 256 gradations of brightness (darkness).

[0023] The camera 6 is placed in a position where it can capture an image of a workpiece 85 as an object to be photographed. In this embodiment, the camera 6 is fixed to a support member 83. The camera 6 in this embodiment is placed above a stand 81 on which the workpiece 85 is placed. The operation control section 43 of the control device 4 sends a command to the camera 6 to capture an image based on the operation program 41. The camera 6 is controlled by the control device 4.

[0024] Although the camera 6 in this embodiment is fixed in position, this is not limiting. For example, the camera 6 may be fixed to a component of the robot 1. The camera 6 may be configured to change its position and posture as the robot 1 changes its position and posture.

[0025] The robot device 5 is equipped with an LED light 7 including an LED (Light Emitting Diode) as an illumination device that illuminates the workpiece 85 when an image of the workpiece 85 is captured by the camera 6. The LED light 7 is supported by a support member 82. In this embodiment, the LED light 7 is fixed in position, but this is not limiting. The illumination device may be fixed to a component of the robot and move together with the component of the robot.

[0026] A reference coordinate system 89 that remains stationary when the position and posture of the robot 1 change is set in the robot device 5. In the example shown in FIG. 1 , the origin of the reference coordinate system 89 is located on the base 14 of the robot 1. The reference coordinate system 89 is also referred to as a world coordinate system. The reference coordinate system 89 has a fixed origin and the orientation of its coordinate axes is also fixed. Furthermore, a tool coordinate system having its origin at the tool tip point is set in the work tool of this embodiment. The position of the robot corresponds to the position of the origin of the tool coordinate system in the reference coordinate system. The posture of the robot corresponds to the orientation of the tool coordinate system relative to the reference coordinate system.

[0027] In this embodiment, the control device 4 that controls the robot 1 functions as an image processing device. The control device 4 includes a processing unit 50 that performs image processing. The processing unit 50 includes a motion correction unit 59 that corrects the position and posture of the robot based on an image of the workpiece 85 captured by the camera 6 when the robot device 5 performs work.

[0028] The robot device 5 of this embodiment grasps the workpiece 85 placed on the base 81 with the hand 2 and transports it to a predetermined position. The workpiece 85 is placed on the base 81 by, for example, an operator. At this time, the position of the workpiece 85 on the base 81 may deviate from the desired position.

[0029] Before grasping the workpiece 85, the robot device 5 captures an image of the workpiece 85 with the camera 6. In this embodiment, the distance from the camera 6 to the top surface of the workpiece 85 is measured in advance. The operation correction unit 59 processes the image captured by the camera 6 to detect characteristic parts of the workpiece 85. For example, the outer edge 85a of the top surface of the workpiece 85 is detected. Then, based on the characteristic parts of the workpiece 85, the position of the workpiece 85 in the reference coordinate system 89 is detected. As the position of the workpiece 85, a predetermined position on the workpiece 85, such as the center of gravity of the outer edge 85a of the top surface, can be used.

[0030] The motion correction unit 59 corrects the position and posture of the robot for gripping the workpiece 85, which are described in the motion program 41, so as to match the position of the workpiece 85. The motion correction unit 59 sends a correction command to correct the position and posture of the robot to the motion control unit 43. The motion control unit 43 controls the position and posture of the robot based on the correction command. The robot device 5 then grips the workpiece 85. By performing such visual control based on the camera image, the work can be performed accurately even if the position of the workpiece 85 is misaligned.

[0031] In this embodiment, an image captured by the camera 6 as a visual sensor is referred to as a captured image. The processing unit 50 includes a steady-state image generating unit 52 that generates a steady-state image based on multiple images acquired from the camera 6 that captures images of the workpiece 85. In this embodiment, an image captured by steady-state light whose brightness is maintained during the image capturing period is referred to as a steady-state image. In other words, the brightness or brightness distribution may change during the image capturing period. In this case, an image captured when captured at the minimum illuminance (minimum brightness) at which brightness is maintained is referred to as a steady-state image.

[0032] The processing unit 50 includes an unstable image generating unit 53 that generates an unstable image based on a plurality of captured images acquired from the camera 6. In this embodiment, when the brightness or brightness distribution changes during the capturing period, an image generated by unstable light corresponding to the change in brightness is referred to as an unstable image. The illuminance of unstable light does not include the illuminance of stationary light. An unstable image is an image captured at an illuminance corresponding to the change in light when the illuminance of light or the distribution of illuminance of light changes.

[0033] The processing unit 50 includes a comparison image generation unit 54 that generates a comparison image in which a pixel value of each pixel of the image is set based on the ratio of the pixel value of the steady image to the pixel value of the unstable image for each pixel of the image. The processing unit 50 includes an evaluation unit 55 that evaluates, in the comparison image generated by the comparison image generation unit 54, a change in the illuminance of the light illuminating the workpiece and a change in the distribution of the illuminance of the light illuminating the workpiece.

[0034] The processing unit 50 includes an acquisition unit 51 that acquires pixel values ​​of each pixel in the captured image captured by the camera 6. In this embodiment, the acquisition unit 51 acquires a luminance value as the pixel value of each pixel in the captured image. The luminance value corresponds to the illuminance of the lighting, the brightness of the pixel, or the density of the pixel. The acquisition unit 51 can also acquire pixel values ​​of images generated by the processing unit 50, such as steady images and unstable images.

[0035] Each of the above-mentioned processing unit 50 and motion correction unit 59 corresponds to a processor that operates according to an operation program. Also, each of the steady-state image generation unit 52, unstable image generation unit 53, comparison image generation unit 54, evaluation unit 55, and acquisition unit 51 corresponds to a processor that operates according to an operation program. The processor reads the operation program and performs the control defined in the operation program, thereby functioning as each unit.

[0036] 3 is a schematic diagram illustrating an example of a lighting device for a building. In this embodiment, an LED light 7 that illuminates a workpiece 85 is disposed near the camera 6. The LED light 7 is disposed so as to uniformly illuminate the surface of the workpiece 85 that is imaged by the camera 6. The light from such an LED light 7 is included in stationary light that does not change with time.

[0037] On the other hand, mercury lamps 86 or fluorescent lamps 87 may be installed on the ceiling of a building. Fluorescent lamps 87 and mercury lamps 86 may cause flickering, in which the illuminance changes according to the electrical frequency. For example, the lamps may become brighter or darker at very short intervals in accordance with the electrical frequency. Alternatively, lighting devices with variable illuminance may be installed. For example, lighting devices that turn off when the robot device is operating without a worker present may be installed.

[0038] Furthermore, the building where work is performed may have windows 88 through which light enters from outside. The brightness of the light entering the building through the windows 88 varies between morning, noon, and night. The direction from which the light enters also changes depending on the time of day. The brightness of the light entering through the windows 88 also changes depending on the weather.

[0039] In this way, due to the characteristics of the building's lighting equipment or the influence of external light, the illuminance of the light illuminating the surface of the workpiece 85 may change over time, a distribution of the illuminance of the light may occur, or the distribution of the illuminance of the light may change.

[0040] In this embodiment, light corresponding to changes in brightness when illuminance changes or distribution occurs is referred to as unstable light. Unstable light is included in light that shines through windows in buildings, light from lighting devices that flicker, and light from lighting devices whose illuminance changes. These types of light may also include steady light whose illuminance does not change. In this way, the workpiece 85 is illuminated by light that is a combination of steady light and unstable light.

[0041] The image processing device of this embodiment captures multiple images of a workpiece and creates from the multiple images a steady image corresponding to the image captured under steady light and an unstable image corresponding to the image captured under unstable light. Furthermore, based on the ratio of pixel values ​​of the steady image and the unstable image, a comparison image is generated to determine the degree of influence of unstable light and the location of unstable light. The steady image, unstable image, and comparison image are not images actually captured by a camera, but images obtained by performing image processing on each pixel of the captured image.

[0042] Fig. 4 shows a flowchart of the control of the image processing device in this embodiment. Fig. 5 shows an explanatory diagram of a captured image, a steady image, and an unstable image in this embodiment. With reference to Figs. 4 and 5, in steps 91 to 94, a steady image 71a is generated based on captured images 61a to 61c. In this embodiment, camera 6 captures a plurality of images of workpiece 85. In this embodiment, camera 6 captures an image of the top surface of workpiece 85.

[0043] In step 91, in one of a plurality of states in which the workpiece 85 is illuminated, the camera 6 captures a first captured image 61a of the workpiece 85. The first captured image 61a includes an image 61aa corresponding to an area surrounded by an outer edge 85a of the workpiece 85 and an image 61ab corresponding to the recessed portion 85b. In this embodiment, the image of the area surrounded by the outer edge 85a but outside the recessed portion 85b is referred to as the "image corresponding to the outer edge 85a." In this embodiment, the brightness is constant in the area corresponding to the outer edge 85a (the area surrounded by the outer edge 85a but outside the recessed portion 85b), and the brightness is constant in the area corresponding to the recessed portion 85b.

[0044] In this embodiment, the brightness value of each pixel is determined as a value of 256 gradations. That is, the brightness value of the image (a value corresponding to brightness and darkness) is used as the pixel value. For each pixel, a pixel value of 0 is the darkest (black), and a pixel value of 255 is the brightest (white).

[0045] Next, in step 92, the acquisition unit 51 acquires pixel values ​​of pixels in the first captured image 61a. Here, the pixel value (number of gradations) of the pixel in the image 61aa is 254, and the pixel value of the pixel in the image 61ab is 200.

[0046] Next, in step 93, pixel values ​​of pixels in the steady image are set to create the steady image. The steady image generation unit 52 generates the steady image 71a based on the pixel values ​​of each pixel in the multiple images. More specifically, the steady image generation unit 52 generates the steady image by selecting the darkest pixel value from the multiple images for each image. In the first imaging, since there is only one captured image, the pixel values ​​of each pixel in the first captured image 61a become the pixel values ​​of the steady image. In other words, at this stage, the steady image is identical to the first captured image 61a. Then, the steady image generation unit 52 determines whether the steady image will be updated by repeating imaging.

[0047] Next, in step 94, the steady-state image generating unit 52 determines whether or not to end photographing the workpiece. In this example, control proceeds to step 91 in order to capture three images with varying lighting brightness. Conditions for capturing multiple images can be determined in advance. For example, the number of images to be captured, the time interval between images, the time at which the images are captured, the operating status of the lighting device that illuminates the workpiece, and the status of the window curtains can be determined in advance. In addition, the worker may determine whether or not to perform the next photographing.

[0048] In step 91, the camera 6 captures a second captured image 61b in an environment that illuminates the workpiece differently from the environment in which the previous image was captured. The second captured image 61b is an image that is darker overall than the first captured image 61a.

[0049] Next, in step 92, the acquisition unit 51 acquires pixel values ​​of pixels in the second captured image 61b. The acquisition unit 51 acquires pixel values ​​of all pixels. The image 61ba corresponding to the outer edge 85a has a pixel value of 200, and the image 61bb corresponding to the recess 85b has a pixel value of 154. Next, in step 93, the steady-state image generation unit 52 compares the pixel value of the current steady-state image with the pixel value of the second captured image 61b for each pixel in the image. In order to select the darkest pixel value for each pixel, the steady-state image generation unit 52 sets the smaller pixel value as the pixel value of the new steady-state image. For example, for the pixel in the region corresponding to the outer edge 85a, the pixel value of the first captured image 61a is 254, and the pixel value of the second captured image 61b is 200. The smaller pixel value, 200, is selected as the pixel value of the steady-state image. Similarly, in the region of the recess 85b, the pixel value adopted is the pixel value of the pixel included in the second captured image 61b, 154. At this stage, the steady-state image is the same as the second captured image 61b.

[0050] Next, control proceeds to step 94. In this example, control returns to step 91 to capture another image. In step 91, the camera 6 captures a third captured image 61c in an environment that illuminates the workpiece differently from the environment in which the previous image was captured. The third captured image 61c is darker than the second captured image 61b.

[0051] In step 92, the acquiring unit 51 acquires the pixel values ​​of the pixels included in the third captured image 61c. The pixel value of the pixel included in the image 61ca corresponding to the outer edge 85a is 154, and the pixel value of the pixel included in the image 61cb corresponding to the recess 85b is 100.

[0052] Next, in step 93, the steady-state image generating unit 52 compares the pixel values ​​of the pixels in the current steady-state image with the pixel values ​​of the pixels in the third captured image 61c. For each pixel, the smaller pixel value is selected. In this example, the pixel value of the image 71aa corresponding to the outer edge 85a of the steady-state image 71a to be finally generated becomes the pixel value of the image 61ca of the third captured image 61c. Furthermore, the pixel value of the image 71ab corresponding to the recess 85b becomes the image value of the image 61cb of the third captured image.

[0053] In this way, the still image generating unit 52 selects the darkest pixel value for each pixel among the plurality of images to generate a still image. Note that in the present embodiment, the pixel value of the captured image is compared with the pixel value of the current still image each time an image is captured, but this is not limiting. After capturing a plurality of images, the still image generating unit 52 may set the smallest pixel value for each pixel to generate a still image.

[0054] Next, in steps 95 to 98, the unstable image generation unit 53 generates an unstable image. The unstable image generation unit 53 generates the unstable image based on the pixel values ​​of each pixel in the multiple images and the pixel values ​​of the steady image. In this example, the unstable image generation unit 53 generates the unstable image using pixel values ​​obtained by subtracting the pixel value of the steady image from the brightest pixel value for each pixel.

[0055] In step 95, the unstable image generating unit 53 selects one captured image from the plurality of captured images 61a to 61c. In this example, the unstable image generating unit 53 selects the first captured image 61a.

[0056] In step 96, the unstable image generator 53 calculates the difference between the pixel value of the first captured image 61a and the pixel value of the steady-state image 71a for each pixel. In this example, the pixel value of the steady-state image is subtracted from the pixel value of the captured image. In the region corresponding to the outer edge 85a, 100 is calculated as the difference between the pixel value of the image 61aa in the first captured image 61a (254) and the pixel value of the image 71aa in the steady-state image 71a (154). Similarly, in the region corresponding to the recess 85b, 100 is calculated as the difference between the pixel value of the image 61ab in the first captured image 61a (200) and the pixel value of the image 71ab in the steady-state image 71a (100).

[0057] Next, in step 97, the unstable image generation unit 53 selects pixel values ​​for the pixels of the unstable image. For each pixel, the unstable image generation unit 53 compares the difference in pixel value calculated this time with the pixel value of the current unstable image and selects the larger value. Here, since this is the first time an unstable image is being created, the difference of 100 is selected as the pixel value for the region corresponding to the outer edge 85a. The difference of 100 is set as the pixel value for the region corresponding to the recess 85b.

[0058] Next, in step 98, the unstable image generation unit 53 determines whether or not all captured images have been selected. If all captured images have not been selected, control returns to step 95. Here, control returns to step 95.

[0059] In step 95, the unstable image generator 53 selects the second captured image 61b. In step 96, the unstable image generator 53 calculates the difference between the pixel value of each pixel in the second captured image 61b and that of the steady image 71a. Here, in the region corresponding to the outer edge 85a, the difference between the pixel value of 200 in the image 61ba and the pixel value of 154 in the image 71aa is calculated as 46. In addition, in the region corresponding to the recess 85b, the difference between the pixel value of 154 in the image 61bb and the pixel value of 100 in the image 71ab is calculated as 54.

[0060] Next, in step 97, the unstable image generation unit 53 selects the pixel value with the largest difference for each pixel. For each pixel, the unstable image generation unit 53 compares the difference between the current second captured image 61b and the steady image with the pixel value of the current unstable image. Then, for each pixel, the larger value is set as the pixel value of the unstable image. Here, since the difference calculated using the first captured image 61a is larger than the difference calculated using the second captured image 61b, the pixel value of 100 is maintained in the region corresponding to the outer edge 85a. The pixel value of 100 is maintained in the region corresponding to the recess 85b.

[0061] Next, in step 98, it is determined whether all captured images have been selected. In this case, since there are captured images that have not been selected, control proceeds to step 95. In step 95, the unstable image generation unit 53 selects the third captured image 61c.

[0062] In step 96, the unstable image generator 53 calculates the difference in pixel value between the third captured image 61c and the steady-state image 71a for each pixel. In the region corresponding to the outer edge 85a, a difference of 0 is calculated. In the region corresponding to the recess 85b, a difference of 0 is calculated.

[0063] In step 97, the unstable image generating unit 53 selects pixel values ​​of the unstable image. In both the region corresponding to the outer edge 85 a and the region corresponding to the recess 85 b, the difference with respect to the third captured image 61 c is smaller than the pixel values ​​of the current unstable image, so the pixel values ​​of the current unstable image are selected.

[0064] In this way, the unstable image generator 53 compares the pixel value differences between the steady image and all captured images, and generates the unstable image 71b using the largest difference as the pixel value. In the unstable image 71b, the pixel value of the image 71ba corresponding to the outer edge 85a is 100, and the pixel value of the image 71bb corresponding to the recess 85b is 100.

[0065] In the above embodiment, the unstable image generating unit compares the difference in pixel values ​​of the captured image with the pixel value of the current unstable image each time a captured image is selected, but this is not limited to this. The unstable image generating unit may select the largest pixel value for each pixel from the pixel values ​​of all captured images, and set the difference between the largest pixel value and the pixel value of the steady image as the pixel value of the unstable image.

[0066] In addition, in the present embodiment, the unstable image is generated after the steady image is completed, but this is not a limitation. The pixel values ​​of the steady image and the unstable image may be selected for each captured image. For example, after capturing one captured image, the steady image generating unit compares the pixel values ​​of the current steady image with the pixel values ​​of the captured image, and sets the smaller pixel value as the pixel value of the steady image.

[0067] Next, after capturing one captured image, the unstable image generation unit compares the pixel values ​​of the current steady image with the pixel values ​​of the currently captured image and generates an image with the maximum pixel value using the larger pixel value.The unstable image generation unit calculates a difference for each pixel by subtracting the pixel value of the currently captured image from the pixel value of the image with the maximum pixel value.The unstable image generation unit then compares the difference calculated this time with the pixel value of the current unstable image and sets the larger value as the pixel value of the unstable image.

[0068] Next, in step 99, the comparison image generation unit 54 generates a comparison image in which pixel values ​​are set for each pixel of the image based on the ratio between the pixel value of the steady image and the pixel value of the unstable image. In this embodiment, the comparison image generation unit 54 calculates the ratio of the pixel value of the unstable image to the pixel value of the steady image for each pixel. The comparison image generation unit 54 calculates the pixel value of each pixel of the comparison image by multiplying this ratio by a predetermined pixel value.

[0069] 6 is an explanatory diagram of the process of generating a comparison image from a steady image and an unstable image. Referring to FIGS. 2, 4, and 6, comparison image generation unit 54 selects one pixel. In this embodiment, comparison image generation unit 54 divides the pixel value of the unstable image by the pixel value of the steady image, and multiplies the result by 128, which is the median of the pixel values.

[0070] For example, in the region corresponding to the outer edge 85a, the pixel value of 100 in the image 71ba of the unstable image 71b is divided by the pixel value of 154 in the image 71aa of the steady image 71a. This value is then multiplied by the intermediate pixel value of 128. As a result, the pixel value of the image 71ca of the comparison image 71c is set to 83 (= 100 / 154 × 128). Also, in the region corresponding to the recess 85b of the workpiece 85, the pixel value of 100 in the image 71bb of the unstable image 71b is divided by the pixel value of 100 in the image 71ab of the steady image 71a, and multiplied by the intermediate pixel value of 128. As a result, the pixel value of the image 71cb of the comparison image 71c is set to 128 (= 100 / 100 × 128). By calculating the pixel values ​​of each pixel in this manner, the comparison image 71c can be generated. If the pixel value of the comparison image exceeds 255, which is the maximum number of gradations, the pixel value can be set to the maximum value.

[0071] Next, in step 100, the steady image, unstable image, and comparison image are evaluated. By looking at the generated steady image 71 a, unstable image 71 b, and comparison image 71 c, the operator can visually determine the effect of unstable light on steady light.

[0072] In this embodiment, the steady-state image 71a corresponds to an image of steady-state light when the illuminance of light is at its lowest value in each pixel. The unstable image 71b corresponds to an image captured with the illuminance of unstable light added to the steady-state light. By viewing the unstable image 74b, the worker can grasp the intensity of the unstable light (amount of change in light). Furthermore, the worker can compare the illuminance of the steady-state light (brightness of the image) with the illuminance of the unstable light (brightness of the image). The magnitude of the illuminance of the unstable light relative to the illuminance of the stable light can be grasped.

[0073] On the other hand, by looking at the comparison image 71c, it is possible to estimate the areas where the influence of unstable light is greater than that of stationary light. Also, it is possible to grasp the magnitude of the illuminance of unstable light relative to the illuminance of stable light. In this embodiment, it is possible to determine that areas with large pixel values ​​(bright areas) in the comparison image 71c are greatly influenced by unstable light. On the other hand, it is possible to determine that areas with small pixel values ​​(dark areas) are little influenced by unstable light.

[0074] In this example, the ratio of the pixel values ​​of the unstable image to the pixel values ​​of the steady image is multiplied by the median of the pixel values, so if the pixel value is brighter than the median value, it can be determined that the influence of unstable light is greater than that of stable light.

[0075] In the above embodiment, the ratio of pixel values ​​of the unstable image to pixel values ​​of the steady image is calculated to generate the comparison image, but this is not limiting. It is also possible to calculate the ratio of pixel values ​​of the steady image to pixel values ​​of the unstable image. In this case, the darker the comparison image is, the greater the influence of unstable light.

[0076] In this way, in this embodiment, it is possible to visually grasp the magnitude of the influence of unstable light on an image. That is, it is possible to grasp the degree to which unstable light changes relative to stationary light. Furthermore, it is possible to estimate areas where the influence of unstable light is significant. In particular, in the case of a complex image containing bright and dark areas in a captured image, it is possible to more clearly determine areas where the influence of unstable light is significant by generating a comparison image.

[0077] In this embodiment, the operator determines the magnitude of the influence of unstable light by viewing the steady image, unstable image, and comparison image, but this is not limiting. The evaluation unit 55 of the processing unit 50 may determine whether the influence of unstable light is large or not based on the pixel values ​​of the steady image, unstable image, and comparison image.

[0078] For example, a portion of the ratio image where the pixel value exceeds a predetermined judgment value may be judged as a portion where the influence of unstable light on stationary light is large. Such a judgment can be performed for each pixel. For example, the pixel value multiplied by the above ratio can be used as the judgment value. When the pixel value is larger than the pixel value multiplied by the ratio, the evaluation unit can judge that the influence of unstable light is large. The control device can perform control to notify the operator. For example, the evaluation unit can perform control to display a warning on the display unit of the control device. On the other hand, when the pixel value is equal to or smaller than the pixel value multiplied by the ratio, the evaluation unit can judge that the influence of unstable light is small.

[0079] If it is determined that the influence of unstable light is large, the worker can take steps to reduce the illuminance of the unstable light. For example, if the influence of light from a window is large, curtains can be installed to prevent the light from entering from the window. Alternatively, if it is determined that there is an influence from flickering of ceiling lights, a roof can be installed to prevent the ceiling lights from shining on the surface of the workpiece. Similar measures can be taken when there is ceiling lighting with variable illuminance. Alternatively, the lighting can be replaced with lighting that does not cause flickering. In other words, the equipment can be changed so that unstable light does not shine on the workpiece.

[0080] Alternatively, the illuminance of the lighting device that emits constant light can be increased. In this embodiment, the illuminance of the LED lighting 7 can be increased. Either of these methods can reduce the influence of unstable light.

[0081] In this embodiment, the evaluation of unstable light can be performed before the robot actually performs work. Then, after establishing a state in which the influence of unstable light on stationary light is small, the robot device can perform actual work. As a result, erroneous determinations in image processing of workpiece images are suppressed, and the robot device can perform work efficiently.

[0082] The conditions for capturing multiple images with a camera can be changed depending on the type of unstable light. For example, if the unstable light depends on the frequency of electricity, such as flickering, the camera can capture a number of images according to the frequency. For example, if the electrical frequency is 60 Hz, 30 images can be captured in a one-second capture period. If the illuminance of the lighting device changes, images can be captured in multiple states when the illuminance of the lighting device changes. Alternatively, the illuminance of the lighting device can be increased above normal illuminance in anticipation of cases where the influence of unstable light is greater. If the unstable light is light from a window, three or more images can be captured in the morning, afternoon, and evening. Alternatively, images can be captured with the curtains open and with the curtains closed.

[0083] The visual sensor in this embodiment is a camera that captures grayscale images, but is not limited to this. A camera that captures color images can also be used as the visual sensor. In this case, the control of this embodiment can be performed on the pixel values ​​of red, green, and blue pixels to generate a steady image, an unstable image, and a comparison image.

[0084] The image processing device of this embodiment is disposed in a robot control device, but is not limited to this. The image processing device of this embodiment can be disposed in any device equipped with a camera. For example, the image processing device of this embodiment can be disposed in an inspection device that detects the contour of a workpiece and inspects the dimensions of the workpiece.

[0085] Second Embodiment An image processing device according to a second embodiment will be described with reference to Figures 7 and 8. The configuration of the robot device 5 according to this embodiment is the same as that of the first embodiment (see Figures 1 and 2). The control of image processing by the control device 4 is also the same as that of the first embodiment (see Figure 4). In the second embodiment, a brightness distribution occurs in the captured image.

[0086] FIG. 7 illustrates an explanatory diagram of a captured image, a steady image, and an unstable image in this embodiment. In the second embodiment, three captured images 62a, 62b, and 62c are also captured. In the first captured image 62a, the brightness gradually changes in the direction indicated by the arrow 111 in the image 62aa corresponding to the outer edge 85a. The brightness of the image 62ab corresponding to the recess 85b is constant. In the second captured image 62b and the third captured image 62c, there is no change in brightness in the regions corresponding to the outer edge 85a and the recess 85b, and the brightness is constant in each region. Since the steady image 72a is an image in which the darkest pixel value is selected for each pixel, in this example, it is identical to the third captured image 62c.

[0087] In the unstable image 72b, the brightness of the image 72bb corresponding to the recess 85b is constant, but the brightness of the image 72ba corresponding to the outer edge 85a gradually increases as indicated by an arrow 111.

[0088] 8 is an explanatory diagram illustrating a comparative image generated from a steady image and an unstable image in the second embodiment. As described above, the brightness of image 72ba of unstable image 72b gradually changes. In image 72ca of comparative image 72c, the brightness gradually increases as indicated by arrow 111. On the other hand, the brightness of image 72cb corresponding to recess 85b remains constant. As such, it can be seen that the influence of unstable light is greater in areas other than recess 85b.

[0089] In this way, even when a luminance distribution occurs in at least some of the captured images, the image processing control of the first embodiment can be performed. In this embodiment, too, the magnitude of the influence of unstable light can be evaluated using a steady image, an unstable image, and a comparison image.

[0090] The other configurations, operations, and effects are the same as those of the first embodiment, and therefore will not be described again here.

[0091] (Third embodiment) An image processing device according to a third embodiment will be described with reference to Fig. 9. The configuration of the robot device 5 according to this embodiment is the same as that of the first embodiment (see Figs. 1 and 2). The control of image processing by the control device 4 is also the same as that of the first embodiment (see Fig. 4). In the third embodiment, the brightness of the captured image varies partially. That is, uneven brightness occurs in the captured image.

[0092] 9 shows an explanatory diagram of a captured image, a steady image, an unstable image, and a comparative image in this embodiment. In this embodiment, two captured images 63a and 63b are captured. In the first captured image 63a, the brightness of an image 63aa corresponding to the outer edge 85a gradually increases in the direction indicated by the arrow 111.

[0093] The image 63ab corresponding to the recess 85b includes an image 63ac where the brightness is locally increased. The image 63ac corresponds to, for example, a spot area hit by linearly traveling light. The brightness is constant in the image 63ab except for the image 63ac. The second captured image 63b does not have a brightness distribution, and the brightness is constant in the area corresponding to the outer edge 85a and the area corresponding to the recess 85b.

[0094] In this example, the steady-state image 73a is identical to the second captured image 63b. In the unstable image 73b, an image 73bb corresponding to the recess 85b includes an image 73bc as a spot region where the brightness increases. An image 73ba corresponding to the outer edge 85a other than the recess 85b has a gradual change in brightness.

[0095] In this embodiment, a comparative image 73c is also generated from a steady image 73a and an unstable image 73b. In the comparative image 73c, the brightness gradually increases in an image 73ca corresponding to the outer edge 85a, as indicated by an arrow 111. An image 73cb corresponding to the recess 85b includes an image 73cc as a spot region that is locally bright. The brightness increases in the image 73cc.

[0096] In this way, even when an image with locally varying brightness is captured, the image processing control of the first embodiment can be performed. In this embodiment, too, the magnitude of the influence of unstable light can be evaluated using a steady image, an unstable image, and a comparison image.

[0097] In the present embodiment, an image in which the luminance increases (becomes brighter) locally has been described as an example, but the present invention is not limited to this. The control of the present embodiment can also be applied to an image in which the luminance decreases (becomes darker) locally. The control of the present embodiment can also be applied to an image in which the luminance changes irregularly.

[0098] The other configurations, operations, and effects are similar to those of the first and second embodiments, and therefore will not be described repeatedly here.

[0099] (Fourth embodiment) An image processing device in a fourth embodiment will be described with reference to Fig. 10. The configuration of the robot device 5 in this embodiment is the same as that in the first embodiment (see Figs. 1 and 2). In this embodiment, the unstable image generation unit 53 generates a second unstable image by multiplying each pixel value of the first unstable image by a predetermined constant. Then, the comparison image generation unit 54 generates a comparison image in which, for each pixel of the image, the pixel value is set based on the ratio between the pixel value of the steady image and the pixel value of the second unstable image. Other image processing controls are the same as those in the first embodiment (see Fig. 4).

[0100] 10 is an explanatory diagram of a steady image, a first unstable image, a second unstable image, and a comparison image in this embodiment. The steady image generating unit 52 generates a steady image 74a based on a plurality of captured images. The unstable image generating unit 53 generates a first unstable image 74b based on the plurality of captured images and the steady image 74a.

[0101] In the first unstable image 74b, the brightness gradually increases in the direction indicated by the arrow 111 in the area corresponding to the outer edge 85a. In the area corresponding to the outer edge 85a, the pixel value changes from 26 to 127. In the area corresponding to the recess 85b, the pixel value remains constant at 77. Here, since the first unstable image 74b is an image in which the pixel value is set to reflect the increase in pixel value due to unstable light, the pixel value may decrease overall. In other words, the image may become dark. As a result, the unstable image and the comparison image may become difficult to distinguish.

[0102] In this embodiment, the unstable image generation unit 53 multiplies each pixel value of the first unstable image 74b by a predetermined constant greater than 1 to generate the second unstable image 74d. Here, each pixel value of the first unstable image 74b is multiplied by 2. In the area corresponding to the outer edge 85a, the pixel value changes from 52 to 254. In the area corresponding to the recess 85b, the pixel value is 154. The second unstable image 74d is brighter than the first unstable image 74b. Note that it is preferable to set the constant so that the pixel value of the second unstable image does not exceed the maximum number of gradations. However, if the pixel value of the second unstable image exceeds the maximum number of gradations, the pixel value may be set to the maximum value.

[0103] The comparison image generator 54 generates a comparison image 74c based on the steady image 74a and the second unstable image 74d. The steady image 74a and the first unstable image 74b of the present embodiment are the same as the steady image 72a and the unstable image 72b of the second embodiment (see FIG. 8). When the comparison image 74c of the present embodiment is compared with the comparison image 72c of the second embodiment, it can be seen that the difference between the bright and dark areas is clear.

[0104] In this way, when an unstable image is dark overall, by generating an image in which the pixel value of each pixel of the unstable image is multiplied by a constant greater than 1, it is possible to more clearly determine the areas where the influence of unstable light is significant.

[0105] As a method for determining whether an unstable image is dark, for example, if the unstable image is darker overall than the steady image, the pixels of the unstable image can be multiplied by a predetermined constant. For example, if the average pixel value of all pixels of the unstable image is smaller than the average pixel value of all pixels of the steady image, the unstable image can be determined to be darker than the steady image. Alternatively, if the average pixel value of the unstable image is smaller than a predetermined judgment value, the unstable image can be determined to be dark. Alternatively, a histogram of each pixel value can be generated, and the median value can be used instead of the average pixel value.

[0106] The control of this embodiment may be performed by the evaluation unit 55 of the processing unit 50. That is, the evaluation unit 55 determines whether the first unstable image is dark or not in the same manner as the operator determines. If the evaluation unit determines that the unstable image is dark, the unstable image generation unit can generate a second unstable image by multiplying all pixel values ​​of the first unstable image by a constant greater than 1.

[0107] The other configurations, operations, and effects are the same as those of the first to third embodiments, and therefore will not be described repeatedly here.

[0108] Fifth Embodiment An image processing device according to a fifth embodiment will be described with reference to Fig. 11 . The configuration of the robot device 5 according to this embodiment is the same as that of the first embodiment (see Figs. 1 and 2). In this embodiment, a comparison image generating unit 54 generates a binarized image by binarizing the pixel values ​​of each pixel in the comparison image. Other image processing controls are the same as those of the first embodiment (see Fig. 4).

[0109] 11 shows an explanatory diagram of a comparison image and a binarized image according to the present embodiment. Comparison image generator 54 generates comparison image 75c based on the steady image and the unstable image. In comparison image 75c according to the present embodiment, the brightness gradually increases in the direction indicated by arrow 111.

[0110] The comparison image generating unit 54 generates a binarized image 75d by binarizing the pixel values ​​of each pixel based on the pixel values ​​of the pixels. The comparison image generating unit 54 obtains a judgment value for the pixel value. The judgment value for the pixel value can be determined in advance. The comparison image generating unit 54 then sets a pixel having a pixel value greater than the judgment value to a first fixed pixel value. A large pixel value (bright pixel value) can be set as the first fixed pixel value.

[0111] On the other hand, the comparison image generating unit 54 sets pixels having pixel values ​​equal to or less than the determination value to a second fixed pixel value. The second fixed pixel value can be set to a small pixel value (a dark pixel value). In this way, the comparison image generating unit 54 can generate the binarized image 75d.

[0112] Alternatively, the comparison image generating unit 54 may adopt a first fixed pixel value when the ratio of the pixel values ​​of the pixels in the unstable image to the pixel values ​​of the pixels in the steady image is greater than a predetermined judgment value, and may adopt a second fixed pixel value when the ratio is equal to or less than the predetermined judgment value. In this case, the judgment value for the ratio may be, for example, 0.2.

[0113] The binarized image 75d can be separated into two regions: a bright region and a dark region. By performing this control, it is possible to roughly grasp the region where the influence of unstable light is large. That is, it is possible to determine that the bright region is a region where the influence of unstable light is large. It is possible to determine that the dark region is a region where the influence of unstable light is small. Furthermore, the operator can take measures to reduce unstable light or increase the illuminance of stable light so that the entire binarized image is made up of pixels with bright luminance.

[0114] The other configurations, operations, and effects are the same as those of the first to fourth embodiments, and therefore will not be described repeatedly here.

[0115] At least one embodiment described above is capable of generating steady and unstable images that allow assessment of the effects of unstable light.

[0116] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0117] The following supplementary notes are disclosed regarding the above-described embodiment and modifications.

[0118] (Supplementary Note 1) An image processing device comprising: a steady-state image generating unit that generates a steady-state image as an image created by steady-state light whose brightness is maintained based on a plurality of images acquired from a visual sensor that captures an image of an object; and an unstable image generating unit that generates an unstable image as an image created by unstable light that corresponds to a change in brightness based on the plurality of images acquired from the visual sensor.

[0119] (Supplementary Note 2) An image processing device as described in Supplementary Note 1, wherein the steady-state image generation unit generates a steady-state image based on pixel values ​​of each pixel in the plurality of images, and the unstable image generation unit generates an unstable image based on pixel values ​​of each pixel in the plurality of images and pixel values ​​of the steady-state image.

[0120] (Appendix 3) An image processing device as described in Appendix 2, wherein the steady-state image generation unit generates a steady-state image by setting the darkest pixel value from the plurality of images for each pixel, and the unstable image generation unit generates an unstable image by setting a pixel value for each pixel obtained by subtracting the pixel value of the steady-state image from the brightest pixel value from the plurality of images.

[0121] (Supplementary Note 4) The image processing device according to any one of Supplementary Notes 1 to 3, further comprising a comparison image generation unit that generates a comparison image in which pixel values ​​are set based on the ratio between the pixel values ​​of a steady image and the pixel values ​​of an unstable image for each pixel of the image.

[0122] (Supplementary Note 5) The image processing device according to Supplementary Note 4, wherein the comparison image generation unit calculates the pixel value of each pixel of the comparison image by multiplying, for each pixel, the ratio of the pixel value of the unstable image to the pixel value of the steady image by a predetermined pixel value.

[0123] (Appendix 6) An image processing device as described in Appendix 4, wherein the unstable image constitutes a first unstable image, the unstable image generation unit generates a second unstable image by multiplying each pixel value of the first unstable image by a predetermined constant, and the comparison image generation unit generates a comparison image in which pixel values ​​are set for each pixel of the image based on the ratio between the pixel value of the steady image and the pixel value of the second unstable image.

[0124] (Supplementary Note 7) The image processing device according to any one of Supplementary Notes 4 to 6, wherein the comparison image generating unit generates a binarized image by binarizing pixel values ​​of each pixel of the comparison image.

[0125] 4 Control device 6 Camera 42 Memory unit 50 Processing unit 51 Acquisition unit 52 Steady image generation unit 53 Unstable image generation unit 54 Comparison image generation unit 55 Evaluation unit 59 Movement correction unit 61a, 61b, 61c Captured images 61aa, 61ab, 61ba, 61bb, 61ca, 61cc Images 62a, 62b, 62c Captured images 62aa, 62ab Images 63a, 63b Captured images 63aa, 63ab, 63ac Images 71a, 72a, 73a Steady images 71aa, 71ab Images 71b, 72b, 73b, 74b, 74d Unstable images 71ba, 71bb, 72ba, 72bb Images 71c, 72c, 73c, 74c, 75c Comparison images 71ca, 71cb Images 75d Binarized image 85 Workpiece 85a Outer edge 85b Recess

Claims

1. An image processing device comprising: a steady-state image generating unit that generates a steady-state image as an image created by steady-state light whose brightness is maintained based on a plurality of images acquired from a visual sensor that captures an image of an object; and an unstable image generating unit that generates an unstable image as an image created by unstable light that corresponds to a change in brightness based on a plurality of images acquired from the visual sensor.

2. An image processing device as described in claim 1, wherein the steady-state image generation unit generates the steady-state image based on the pixel values ​​of each pixel in a plurality of images, and the unstable image generation unit generates the unstable image based on the pixel values ​​of each pixel in a plurality of images and the pixel values ​​of the steady-state image.

3. The image processing device described in claim 2, wherein the steady-state image generation unit generates the steady-state image by setting the darkest pixel value from multiple images for each pixel, and the unstable image generation unit generates the unstable image by setting a pixel value for each pixel that is the brightest pixel value from multiple images minus the pixel value of the steady-state image.

4. An image processing device as described in any one of claims 1 to 3, comprising a comparison image generation unit that generates a comparison image in which pixel values ​​are set for each pixel of the image based on the ratio between the pixel values ​​of the steady image and the pixel values ​​of the unstable image.

5. An image processing device as described in claim 4, wherein the comparison image generation unit calculates the pixel value of each pixel of the comparison image by multiplying, for each pixel, the ratio of the pixel value of the unstable image to the pixel value of the steady image by a predetermined pixel value.

6. An image processing device as described in claim 4, wherein the unstable image constitutes a first unstable image, the unstable image generation unit generates a second unstable image by multiplying each pixel value of the first unstable image by a predetermined constant, and the comparison image generation unit generates a comparison image in which pixel values ​​are set for each pixel of the image based on the ratio between the pixel value of the steady image and the pixel value of the second unstable image.

7. An image processing device according to any one of claims 4 to 6, wherein the comparison image generating section generates a binarized image by binarizing the pixel values ​​of each pixel of the comparison image.

Images