Image processing device and picking system

Abstract

To provide an image processing device or the like with reduced cost and high versatility.SOLUTION: An image processing device used for detecting a detection object placed under a light source whose luminance changes at a predetermined cycle according to one aspect of the present invention includes: an image acquisition unit that acquires first image information obtained by imaging the detection object under a light source in a first luminance state and second image information obtained by imaging the detection object in a second luminance state that is different from the first luminance state; a generation unit that generates difference image information on a difference in luminance between the first image information and the second image information; and an output unit that outputs detection information indicating the detection results of the detection object based on the difference image information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and a picking system using the image processing device. [Background technology]

[0002] Traditionally, picking robots have been used in many production and manufacturing sites to pick up work objects or transported items (also called workpieces, hereafter referred to as "workpieces"), and in recent years, random picking, which picks bulk workpieces, has become more common. In random picking, a camera captures an image of the bulk workpieces before picking them up, and the robot recognizes the position and orientation of the workpieces from the captured image (for example, Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-146823 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, workpiece picking, including random picking, requires captured images of the workpieces. However, in the workplace where the picking robot operates, the brightness of the workplace can fluctuate due to scattered or reflected light from nearby processes (production lines) and ambient light, such as sunlight shining through exterior windows. This brightness fluctuation also causes fluctuations in the brightness between images required for picking, reducing the accuracy of workpiece recognition within the image. To mitigate the effects of this ambient light, it is possible to use an imaging device that is resistant to brightness fluctuations or a lighting device that cancels out brightness fluctuations. However, introducing high-performance equipment is costly, and because ambient light varies from workplace to workplace, a system built to suit one workplace cannot be applied to other workplaces.

[0005] The present invention has been made in view of the above problems, and has as its object to provide an image processing apparatus and the like that is cost-effective and highly versatile. [Means for solving the problem]

[0006] According to one aspect of the present invention, an image processing device used to detect a detection target placed under a light source whose brightness fluctuates at a predetermined cycle includes an image acquisition unit that acquires first image information of the detection target captured under the light source at a first brightness and second image information of the detection target captured at a second brightness different from the first brightness, a generation unit that generates differential image information of the difference in brightness between the first image information and the second image information, and an output unit that outputs detection information indicating the detection result of the detection target based on the differential image information.

[0007] In an image processing device according to one aspect of the present invention, the image acquisition unit acquires first image information and second image information from an imaging device that captures images at predetermined time intervals, and the generation unit may generate differential image information in which, of the multiple images captured by the imaging device, information regarding the image with the highest brightness is the first image information and information regarding the image with the lowest brightness is the second image information.

[0008] In an image processing device according to one aspect of the present invention, when a light source frequency, which is the reciprocal of the time between luminance peaks in the light source, is f1 and an imaging frequency, which is the reciprocal of a predetermined time interval in the imaging device, is f2, the ratio f1:f2 may be 1:0.5 to 1:4 (excluding 1:1).

[0009] In the image processing device according to an aspect of the present invention, when the difference between the first luminance and the second luminance is below a predetermined threshold, at least one of the light source frequency and the imaging frequency may be changed.

[0010] In the image processing device according to an aspect of the present invention, when the difference between the first luminance and the second luminance is below a predetermined threshold, the timing at which the imaging device captures an image of the detection target may be changed.

[0011] A picking system according to one aspect of the present invention includes an image processing device and a picking device, and the picking device includes a placement section for the detection target, a picking section that picks the detection target from the placement section, and a picking control section that controls the picking section based on detection information. [Effects of the Invention]

[0012] An image processing device according to one aspect of the present invention is an image processing device used to detect a detection target placed under a light source that generates flicker, where the luminance fluctuates at a predetermined cycle, and is characterized by including an image acquisition unit that acquires first image information obtained by capturing the detection target under the light source at a first luminance and second image information obtained by capturing the detection target at a second luminance different from the first luminance, a generation unit that generates differential image information representing the difference in luminance between the first image information and the second image information, and an output unit that outputs detection information indicating the detection result of the detection target based on the differential image information. This makes it possible to provide an image processing device and the like that is cost-effective and highly versatile. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic configuration diagram of a picking system according to one embodiment of the present invention. [Figure 2] 1 is a functional diagram of an image processing device according to an aspect of the present invention; [Figure 3] 1A and 1B are diagrams illustrating an overview of an image processing device according to an embodiment of the present invention. [Figure 4] 1A to 1C are diagrams illustrating an outline of an image processing device according to an embodiment of the present invention. [Figure 5] 1A to 1C are diagrams illustrating an outline of an image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] An image processing device according to one embodiment of the present invention will be described in detail below with reference to the drawings. Note that the drawings are merely examples, and the image processing device of the present invention is not limited to those shown. Also, the drawings are schematic, and the size ratios, number, positional relationships, graphs, etc. of the components of the picking system in the drawings are not strict.

[0015] <Picking system> FIG. 1 is a schematic diagram of a picking system according to one embodiment of the present invention. The picking system 100 includes at least an image processing device 200 and a picking device 30. The picking device 30 includes a picking arm (picking unit) 31 and a placement unit 40, and picks up and grips a workpiece 50 placed on the placement unit 40, or moves the workpiece 50 to another process. Although FIG. 1 shows the workpiece 50 as a single unit, the workpiece 50 may be stacked in bulk. Furthermore, the workpiece 50 may be any object that can be picked up by the picking arm 31. Furthermore, the placement unit 40 is not limited to a belt conveyor-like structure as shown in FIG. 1, but may be a box-like structure on which the workpieces 50 are stacked in bulk.

[0016] The image processing device 200 acquires an image of the workpiece 50, which is the detection target, captured by the imaging device 20, from the imaging device 20 and performs image recognition processing. The imaging device 20 may be a known camera such as a CCD camera, an RGB camera, or a stereo camera. The image processing device 200 is also connected to the picking device 30, recognizes the state (orientation, shape, etc.) of the workpiece 50 from the image captured by the imaging device 20, and transmits information regarding the state of the workpiece 50 to the picking device 30. Note that although the image processing device 200 and the picking device 30 are shown separately in FIG. 1, the image processing device 200 may be incorporated into the picking device 30.

[0017] In the picking system 100, as shown in FIG. 1, the brightness of the image captured by the imaging device 20 can change due to ambient light 70, including sunlight 60 shining through a window at the work site and light reflected from other processes. Therefore, a picking system according to one embodiment of the present invention performs image processing utilizing the characteristics of the light source 10 at the work site. As will be described in detail later, the light source 10 at the work site may be a light source whose brightness fluctuates at a predetermined cycle. For example, the light source 10 may be a conventional light source that generates flicker, such as a fluorescent lamp, mercury lamp, incandescent lamp, or LED. In other words, the light source 10 may not have a so-called flickerless function. Note that flicker is caused by frequency fluctuations in the incoming voltage, and is a phenomenon in which light and dark alternate at a cycle of 60 Hz in western Japan and 50 Hz in eastern Japan.

[0018] In the following, we will use a light source that generates flicker as an example of a light source whose brightness fluctuates at a predetermined cycle, but the present invention is not limited to this, and any lighting can be used as long as the speed at which the brightness fluctuates (i.e., the speed at which light changes from bright to dark) can be adjusted.

[0019] <Image processing device> Next, the functional configuration of the image processing device 200 and the operation of the image processing device 200 will be described with reference to Figs. 2 and 3. As shown in Fig. 2, the image processing device 200 includes at least an image acquisition unit 210, an image processing unit 220, and a detection result output unit 230. The image acquisition unit 210 acquires an image captured by the imaging device 20. At this time, the image acquisition unit 210 acquires first image information of the workpiece to be detected captured when the light source 10 is at a first luminance, and second image information of the workpiece captured when the light source 10 is at a second luminance different from the first luminance.

[0020] Here, the operation of the image processing device 200 according to one embodiment of the present invention will be described with reference to FIG. 3. FIG. 3(a) is a graph showing changes in luminance under the influence of ambient light 70 at the work site of the picking system 100 shown in FIG. 1. In FIG. 3(a), the horizontal axis represents time, the vertical axis represents luminance, a dashed line 11 represents the change in luminance of the light source 10 over time, and a dashed-dotted line 71 represents the change in luminance of the ambient light 70 over time. As indicated by the dashed line 11 in FIG. 3(a), the light source 10 flickers, in which the luminance fluctuates at a predetermined cycle. In the example of FIG. 3(a), the light source 10 alternates between bright and dark at a cycle T1 [s] (i.e., light source frequency f1 = 1 / T1 [Hz]). In contrast, as indicated by the dashed-dotted line 71, the ambient light 70 exhibits a more gradual change in luminance over time than the light source 10. The light source 10 and the ambient light 70 are superimposed, and the brightness fluctuation at the work site of the picking system 100 is shown in the graph by a solid line 81.

[0021] The imaging device 20 captures an image of the workpiece 50 under the luminance fluctuations shown by the solid line 81, and therefore captures images at different luminances depending on the timing. For example, as shown in Fig. 3(a), the luminance of an image captured at time t1 is L1 (first luminance), and the luminance of an image captured at time t0 is L0 (second luminance), and images at different luminances can be captured.

[0022] The generation unit 221 generates a difference image by subtracting the difference in luminance between the first image and the second image. Here, the properties of the difference image generated from the images captured under the first luminance and the second luminance will be described with reference to Fig. 3(b).

[0023] In FIG. 3(b), the first image 21 is a high-brightness image captured at a first luminance (L1), which is a relatively high luminance in FIG. 3(a), and the second image 22 is a low-brightness image captured at a second luminance (L0), which is a relatively low luminance. As shown in FIG. 3(b), noise 72 occurs in the first image 21 and the second image 22 due to the influence of disturbance light 70. Here, because the time variation in the luminance of the disturbance light 70 is gradual as shown in FIG. 3(a), there is little change in the noise 72 between the first image 21 and the second image 22. Therefore, the difference image 23 of the first image 21 and the second image 22 generated by the generation unit 221 can be an image that does not include components of the noise 72.

[0024] The image processing unit 220 uses a known image recognition technique on the differential image 23 to detect the workpiece 50 included in the differential image 23. Note that the image recognition may be realized by machine learning or AI (Artificial Intelligence).

[0025] The detection result output unit 230 transmits the detection result by the image processing unit 220 to the pickup device 30. The detection result may include information about the state of the workpiece 50, such as the position and orientation of the workpiece 50. A control unit (not shown) of the pickup device 30 may output a control signal to the picking arm 31 to pick up the workpiece 50, based on the information about the state of the workpiece 50 included in the detection result.

[0026] In this way, the image processing device according to one aspect of the present invention can eliminate the effects of ambient light by utilizing fluctuations in the luminance of the light source. Therefore, a low-cost picking system can be realized without requiring high functionality from the light source or imaging device.

[0027] The imaging device 20 may be an imaging device that captures images at predetermined time intervals, and the image acquisition unit 210 may acquire multiple images from the imaging device 20. For example, in the example of FIG. 3, the imaging device 20 starts capturing images from time t1 at a period T2 [s] (i.e., an imaging frequency f2 = 1 / T2 [Hz]), and multiple images are captured at brightnesses indicated by "◯" on the periodically changing curve in the figure. In this case, the generation unit 221 may generate a difference image by using the image with the maximum brightness among the multiple images captured by the imaging device 20 as a first image (in the example of the figure, the image at time t1 and brightness L1) and the image with the minimum brightness as a second image (in the example of the figure, the image at time t0 and brightness L0).

[0028] In this way, the image processing device 200 according to one aspect of the present invention may generate a difference image by selecting an image with a large difference in brightness from among the multiple images acquired by the image acquisition unit 210. Therefore, the brightness level of the difference image can be increased, thereby improving the detection accuracy of the detection target.

[0029] In the above description, an example has been described in which a differential image is generated regardless of the order in which images are captured by the imaging device 20. However, there may be cases in which the state of the detection target changes over time, such as when the workpiece 50 moves on a conveyor. In this case, a differential image may be generated between captured images that are adjacent on the time axis.

[0030] Fig. 4(b) shows a graph of the luminance fluctuation of the difference image 23 obtained when the imaging device 20 captures images at a predetermined time interval and takes the difference between adjacent images on the time axis. Note that Fig. 4(a) is the same as Fig. 3(a), but for simplicity, the dashed line 11 indicating the luminance fluctuation of the light source 10 is omitted, and only the dashed-dotted line 71 indicating the luminance fluctuation of the ambient light 70 and the solid line 81 indicating the luminance fluctuation of the location where the picking system 100 operates using the light source 10 and ambient light 70 are shown.

[0031] When the imaging device 20 captures images at a period T2 (T2=T1 / 2 in the example shown in FIG. 4(a)), the image acquisition unit 210 acquires a plurality of first luminance images captured at a first luminance indicated by a "●" in the figure, which are relatively high in luminance, and a plurality of second luminance images captured at a second luminance indicated by a "■" in the figure, which are relatively low in luminance. The generation unit 221 may generate a difference image from the first luminance image and the second luminance image that are adjacent on the time axis. In this case, image frames F11, F12, etc. with the same luminance are obtained, as shown in FIG. 4(b). The image processing unit 220 may perform recognition processing on the images of each image frame F11, F12, etc.

[0032] However, compared to the original image (especially high-brightness images), each of the image frames F11, F12, etc. in FIG. 4(b) emphasizes shot noise and thermal noise, resulting in a poor S / N ratio (signal-to-noise ratio). The S / N ratio here refers to the ratio of noise to the signal representing the detection target. Therefore, as shown in FIG. 4(c), the image processing unit 220 may perform brightness averaging between each image frame (between three frames F21 or F22 in the illustrated example). This reduces shot noise and thermal noise and improves the S / N ratio. Note that performing averaging between image frames reduces the number of image frames, which may result in a decrease in detection accuracy when the detection target is moving, for example. Therefore, the number of image frames to be averaged may be set according to the moving speed of the detection target, etc.

[0033] Note that the example of FIG. 4 describes an ideal case in which the difference between the first luminance and the second luminance is maximized when the imaging device 20 captures images at an imaging period T2 (=T1 / 2). However, as shown in FIG. 5(a), depending on the imaging timing, i.e., the relative time lag between the light source period T1 of the light source 10 and the imaging period T2 of the imaging device 20, the luminance difference between images captured by the imaging device 20 may be very small. In the example of FIG. 5(a), the luminances of images captured by the imaging device 20 at imaging period T2 and times t3, t4, t5, and t6 are L3, L4, L5, and L6, respectively, indicating that the difference between the captured image frames is small. In such a case, the luminance of the difference image generated by the generation unit 221 becomes very small, which may make it difficult to detect the target object.

[0034] Therefore, according to the image processing device 200 according to one aspect of the present invention, when the light source frequency, which is the reciprocal of the time between luminance peaks in the light source 10, is f1 (=1 / T1), and the imaging frequency, which is the reciprocal of the predetermined time interval (T2) in the imaging device 20, is f2 (=1 / T2), f1:f2 = 1:0.5 to 1:4 (excluding 1:1) (Equation 1) The imaging period or the light source period may be set so as to satisfy the above.

[0035] For example, Figure 5(b) shows a case where the shooting frequency f2' (= 1 / T2') is four times the light source frequency f1. In this case, the luminances of images captured at times t7, t8, t9, and t10 are L7, L8, L9, and L10, respectively, and some degree of difference in luminance can be generated between some of the captured image frames. Therefore, in the example of Figure 5(b), the image captured at luminance L8 is designated as the first luminance image, and the image captured at luminance L9 is designated as the second image, and a difference image can be generated.

[0036] Equation (1) shows a preferred relationship between the imaging frequency and the light source frequency, and FIG. 5(c) shows a preferred magnification of the imaging frequency to the light source frequency. If the imaging frequency is 0.5 or 1 times the light source frequency, the difference in luminance between the first luminance image and the second luminance image will become small, as described above, which is undesirable. Therefore, except for 0.5 and 1 times, if the imaging frequency is 4 times the light source frequency, a detectable difference image can be generated. While an imaging frequency exceeding 4 times the light source frequency is of course possible, this would require the imaging device 20 to have high functionality, so the imaging frequency should be determined taking into account cost.

[0037] As shown in Fig. 5(a), when the difference in luminance between captured images is small due to the relationship between the imaging cycle and the light source cycle, i.e., when the difference between the first luminance and the second luminance is below a predetermined threshold, at least one of the light source frequency f1 and the imaging frequency f2 may be changed. For example, by increasing the imaging frequency f2 from the state shown in Fig. 5(a), it is possible to transition to the state shown in Fig. 5(b). The light source frequency f1 of the light source 10 may be changed by an inverter device or the like.

[0038] Although the above describes a mode in which either the light source frequency f1 or the imaging frequency f2 is changed, if the difference between the first luminance and the second luminance falls below a predetermined threshold, the timing at which the imaging device 20 captures an image of the detection target may be changed. For example, from the state of Fig. 5(a), imaging by the imaging device 20 may be stopped once, and imaging may be started after shifting the imaging timing.

[0039] These methods can increase the brightness of the difference image and improve the detection accuracy.

[0040] The picking system 100 including the image processing device 200 according to one embodiment of the present invention and the picking device 30 may utilize the flicker of the conventionally used light source 10 and set the imaging frequency of the imaging device 20 according to the frequency of the flicker. Therefore, a low-cost, highly versatile picking system can be realized without requiring a special lighting device or a high-performance imaging device. Furthermore, the light source frequency of the light source 10 may be adjusted relative to the imaging frequency of the imaging device 20. In this case, there is no need to prepare a new imaging device 20 to build the picking system 100, thereby reducing costs.

[0041] Although the present invention has been described based on the drawings and examples, it should be noted that various modifications and alterations can be easily made by those skilled in the art based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present invention. [Explanation of symbols]

[0042] 10 light source 20 Captured images 30 Pickup device 31 Pickup arm 40 Placement section 50 Workpieces (detection target) 60 sun 70 Ambient light 100 Picking System 200 Image processing device 210 Image acquisition unit 220 Image Processing Unit 221 Generation part 230 Detection result output unit

Claims

1. An image processing device used to detect a detection target placed under a light source whose luminance fluctuates at a predetermined cycle, an image acquisition unit that acquires first image information obtained by capturing an image of the detection target under the light source at a first luminance and second image information obtained by capturing an image of the detection target under a second luminance different from the first luminance; a generating unit that generates differential image information representing a difference in luminance between the first image information and the second image information; an output unit that outputs detection information indicating a detection result of the detection object based on the differential image information; Equipped with the image acquisition unit acquires the first image information and the second image information from an imaging device that captures images at predetermined time intervals; the generation unit generates difference image information in which information on an image with the highest luminance among a plurality of images captured by the imaging device is the first image information and information on an image with the lowest luminance is the second image information, When a light source frequency, which is the reciprocal of the time between luminance peaks in the light source, is f1, and an imaging frequency, which is the reciprocal of the predetermined time interval in the imaging device, is f2, f1:f2 = 1:0.5 to 1:4 (excluding 1:1) Meet the image processing device.

2. When the difference between the first luminance and the second luminance is less than a predetermined threshold, at least one of the light source frequency and the imaging frequency is changed.

2. The image processing device according to claim 1, wherein:

3. When the difference between the first luminance and the second luminance is below a predetermined threshold, the imaging device changes a timing at which it captures an image of the detection target.

2. The image processing device according to claim 1, wherein:

4. A picking system comprising the image processing device according to any one of claims 1 to 3 and a picking device, The picking device is a placement portion for the detection target; a picking unit that picks the detection target from the placement unit; a picking control unit that controls the picking unit based on the detection information; A picking system comprising:

Images