Image acquisition method, image acquisition device, and image acquisition program

Abstract

According to the present invention, an image processing method includes: acquiring a captured image obtained by imaging an object; detecting glossy pixels having a glossy appearance among pixels included in the captured image; generating a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image; executing, a plurality of times, first processing for determining an illumination intensity of illumination of the next frame and controlling the illumination, on the basis of the captured image and the gloss image; and, while the first processing is being executed a plurality of times, executing second processing for generating a fused image obtained by fusing the plurality of captured images, on the basis of the plurality of gloss images generated by the first processing.

Description

Image acquisition method, image acquisition device, and image acquisition program

[0001] The present disclosure relates to an image acquisition method, an image acquisition device, and an image acquisition program.

[0002] Patent Document 1 describes an image processing method that uses an imaging unit placed at a predetermined position to capture multiple images of an object to be inspected moving in a predetermined direction while illuminating the field of view of the imaging unit with an illumination light source placed at a fixed relative position to the imaging unit, and sequentially searches for an area showing at least a part of the object to be inspected between two of the multiple images acquired by the series of images captured by the imaging unit, thereby aligning the multiple images using the object to be inspected as a reference, and for each unit area of ​​the aligned multiple images, calculates image information that represents the unit area from the image information held by each area in the aligned multiple images that corresponds to the unit area, thereby generating a composite image.

[0003] Japanese Patent Application Publication No. 2011-163766

[0004] The present disclosure has been devised in view of the conventional circumstances, and aims to provide an image acquisition method, an image acquisition device, and an image acquisition program for acquiring a captured image with reduced gloss.

[0005] The present disclosure provides an image processing method performed by an image processing device capable of communicating between a camera capable of capturing an image of an object and lighting that illuminates the object, the image processing method comprising: acquiring an image of the object; detecting glossy pixels among the pixels included in the captured image; generating a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image; performing a first process multiple times to determine the illumination intensity of the lighting for the next frame and control the illumination based on the captured image and the gloss image; and, while performing the first process multiple times, performing a second process to generate a fused image by fusing a plurality of the captured images based on the plurality of gloss images generated in the first process.

[0006] The present disclosure also provides an image processing device capable of communicating between a camera capable of capturing an image of an object and lighting that illuminates the object, the image processing device comprising: a first control unit that executes a first process multiple times to acquire an image of the object, detect glossy pixels among the pixels included in the image, generate a gloss image indicating the presence or absence and degree of gloss of each pixel included in the image, and determine the illumination intensity of the lighting for the next frame based on the image and the gloss image to control the illumination; and a second control unit that executes a second process while executing the first process multiple times to generate a fusion image by fusing a plurality of the imaged images based on the plurality of gloss images generated in the first process.

[0007] The present disclosure also provides an image acquisition program executed by a processor capable of communicating between a camera capable of capturing an image of an object and lighting that illuminates the object, the image acquisition program realizing the following steps: causing the processor to acquire an image of the object; detecting glossy pixels among the pixels included in the captured image, generating a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image, and executing a first process multiple times to determine the illumination intensity of the lighting for the next frame and control the illumination based on the captured image and the gloss image; and executing a second process, while executing the first process multiple times, to generate a fused image by fusing a plurality of the captured images based on the plurality of gloss images generated in the first process.

[0008] According to the present disclosure, it is possible to acquire a captured image with reduced gloss.

[0009] FIG. 1 is a diagram for explaining an example of the configuration of a picking system according to the first embodiment; FIG. 2 is a block diagram showing an example of the internal configuration of an image processing device; FIG. 3 is a diagram showing a first example of processing of the picking system; FIG. 4 is a diagram showing an example of a captured image and a gloss image; FIG. 5 is a diagram showing an example of control of illumination intensity;

[0010] (Background to the present disclosure) In recent years, production processes in factories have become common in which a picking robot and a worker work together to pick objects transported on a conveyor belt. The picking robot is equipped with a camera, lighting, and the like, and captures images of the objects being transported. By performing image recognition of the objects captured in the captured images, the positions of the objects moving on the conveyor belt can be tracked in real time. However, images captured in this manner can have glossy, washed-out highlights on the surface of the objects due to the material or shape of the objects to be picked, or the position or intensity of the lighting, which can make the images unsuitable for image recognition.

[0011] One possible solution is to capture an image with reduced gloss by controlling the brightness of the lighting and capturing an image of the object. However, if a method of varying the brightness of the lighting is adopted in a space where humans (workers) work together, the humans (workers) will perceive light fluctuations of 60 Hz or less, and the workers working together with the picking robot may be stressed by the fluctuations in the brightness of the lighting.

[0012] The above-mentioned Patent Document 1 discloses a technology for recognizing and tracking inspection objects on a belt conveyor using multiple images captured by a camera installed at a fixed location that cannot be moved relative to a production line, etc. However, when this technology is applied, it is not possible to simultaneously inspect and pick objects, so picking by a picking robot is not performed and it is necessary to set up a dedicated inspection section on the belt conveyor just to capture images of the objects to be inspected, which requires an expansion of the production line.

[0013] Therefore, in the following embodiments, examples of an image acquisition method, an image acquisition device, and an image acquisition program for acquiring a captured image with reduced gloss will be described.

[0014] Hereinafter, with appropriate reference to the accompanying drawings, detailed descriptions of embodiments specifically disclosing an image acquisition method, an image acquisition device, and an image acquisition program according to the present disclosure will be provided. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0015] (Embodiment 1) A picking system 100 according to embodiment 1 will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram illustrating an example of the configuration of the picking system 100 according to embodiment 1. Fig. 2 is a block diagram illustrating an example of the internal configuration of the picking system 100.

[0016] This disclosure provides an example use case in which, for example, during a production process in a factory, an image is acquired of a commodity, product, or part (hereinafter referred to as "object") that is to be picked up by an end effector such as a robotic hand, and the process of controlling the illumination intensity for illuminating the object based on the gloss area of ​​the captured image is repeated to capture the object, thereby generating a gloss-suppressed image in which the gloss of the captured image used to recognize the object is suppressed.

[0017] In the following description, an object Tg is a picking target that is picked by an end effector EF of a picking system 100 installed in a factory. After being picked, the object may be moved to another lane (production line) or stored in a box such as a cardboard box.

[0018] The picking system 100 according to the first embodiment includes an actuator AC, a camera CM, a lighting LT, an image processing device P1, a fusion image database DB, and an operation device INP. The actuator AC and the image processing device P1, the camera CM and the image processing device P1, the lighting LT and the image processing device P1, the fusion image database DB and the image processing device P1, and the operation device INP and the image processing device P1 are connected to each other so as to enable input and output (transmission and reception) of data signals.

[0019] The actuator AC controls the lighting LT and the camera CM so that they can move three-dimensionally, and controls so that the positional relationship between the object Tg moving on the belt conveyor BC, the end effector EF that picks the object Tg, and the lighting LT and camera CM fixedly installed on the end effector EF can be changed.

[0020] The actuator AC controls the end effector EF, the light LT, and the camera CM provided on the end effector EF so that they can be moved three-dimensionally using multiple axes. In other words, the actuator AC recognizes the three-dimensional positions (coordinates) of the light LT and the camera CM and can maintain or change the positions.

[0021] The end effector EF is, for example, a robot hand provided at the tip of a robot arm deployed in correspondence with the picking system 100, and approaches the target object Tg and picks up the target object Tg under the control of the actuator AC.

[0022] The camera CM is disposed near the end effector EF and moves integrally with the end effector EF under the control of the actuator AC to capture an image of the object Tg illuminated by the light LT. The camera CM captures images of the object Tg at a predetermined frame rate (e.g., 1000 frames per second (hereinafter referred to as "fps")) and transmits each captured image of the object Tg to the image processing device P1.

[0023] The illumination LT has its illumination intensity controlled by the processor 11 and illuminates the object Tg, which is the image capture target of the camera CM. The illumination LT is configured to have one or more light-emitting elements, such as a light-emitting diode (LED). The illumination LT may be configured as, for example, a surface light source, a point light source, or a linear light source, or may be configured by combining a plurality of different surface light sources, point light sources, or linear light sources.

[0024] The image processing device P1 repeatedly executes a first process of determining the illumination intensity of the lighting LT when capturing the next frame based on the gloss state of the captured image of the object Tg transmitted from the camera CM and performing illumination control, and a second process of generating a gloss-suppressed image based on the captured image obtained by the first process and performing object tracking based on the movement amount of the object using the object appearing in the gloss-suppressed image, until the end effector EF picks up the object Tg.

[0025] The image processing device P1 may be, for example, a personal computer (hereinafter referred to as "PC"), or may be realized by a plurality of dedicated hardware devices corresponding to the processing speeds of the first process and the second process described above. The image processing device P1 performs the first process and the second process described above, thereby recognizing the position and orientation of the target object Tg picked by the end effector EF. The image processing device P1 includes a communication unit (not shown), a processor 11, and a memory 12. The image processing device P1 may be integrated with a fusion image database DB.

[0026] The communication unit (not shown) is connected to the actuator AC, the camera CM, the lighting LT, the fusion image database DB, and the operation device INP so that data can be communicated between them, and transmits and receives data. The communication unit (not shown) outputs the captured image transmitted from the camera CM and the control command transmitted from the operation device INP to the processor 11. The communication unit (not shown) transmits the control command output from the processor 11 to the lighting LT or the actuator AC.

[0027] The processor 11 is configured using, for example, a Central Processing Unit (CPU) or a Field Programmable Gate Array (FPGA), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 references the programs and data stored in the memory 12 and executes the programs to realize the functions of the high-speed processing unit 11A and the low-speed processing unit 11B.

[0028] The high-speed processing unit 11A controls the illumination intensity of the illumination LT in accordance with the frame rate of the camera CM so that the total amount of light emitted by the illumination LT during N frames (N: an integer equal to or greater than 3) is constant, acquires captured images captured by the camera CM in each frame, and executes a first process to acquire an aligned captured image and a gloss image obtained by aligning the captured image captured in the first frame of each cycle. The high-speed processing unit 11A executes the first process at a speed corresponding to the frame rate of the camera CM, i.e., at a cycle of 1000 fps. The high-speed processing unit 11A includes a gloss detection unit 111, an image alignment unit 112, an illumination intensity determination unit 113, and an illumination control unit 114, and realizes the functions of these units to realize the first process.

[0029] The gloss detection unit 111 detects glossy pixels by emphasizing glossy (blown-out highlight) pixels (areas) from a captured image (a color image, for example, captured images IMG11 and IMG1M shown in FIG. 3 ) captured by the camera CM. Based on the detected glossy pixels, the gloss detection unit 111 generates a gloss image (a black-and-white image, for example, gloss images IMG21 and IMG2M shown in FIG. 3 ) that indicates the presence or absence and the degree of gloss.

[0030] The image alignment unit 112 performs alignment between the captured image captured in the first frame of the corresponding cycle and the captured images captured in the second frame and thereafter, based on the captured image captured by the camera CM, the gloss image generated by the gloss detection unit 111 and corresponding to the captured image, and the movement amount of the object Tg per 1 ms output from the actuator control unit 117. The image alignment unit 112 generates a set of fusion data that associates the captured image and gloss image after alignment with the illumination intensity of the illumination LT when the captured image was captured, and stores (preserves) the data in the fusion image database DB.

[0031] The illumination intensity determination unit 113 determines the gloss area and gloss intensity based on the captured image and the gloss image. The illumination intensity determination unit 113 determines the illumination intensity of the illumination LT to be used for capturing the next frame based on the gloss area and gloss intensity, and outputs the illumination intensity to the illumination control unit 114.

[0032] The lighting control unit 114 acquires the lighting intensity information output from the lighting intensity determination unit 113. The lighting control unit 114 controls the lighting intensity of the lighting LT to the lighting intensity output from the lighting intensity determination unit 113. When capturing the first frame, the lighting control unit 114 sets the lighting intensity of the lighting LT to a default lighting intensity that was set in advance based on the operator's operation of the operation device INP. The lighting control unit 114 also outputs lighting intensity information for the current frame to the image fusion unit 115.

[0033] The low-speed processing unit 11B fuses the M captured images and gloss images obtained by the first processing to generate a gloss-reduced image IMG (see FIG. 7 ), and then performs a second processing in which the movement amount of the object Tg is calculated based on the generated gloss-reduced image IMG to control the movement amount of the actuator AC. The low-speed processing unit 11B performs the second processing at a speed corresponding to one cycle (=N frames) of the high-speed processing unit 11A, for example, at a cycle of 60 fps (i.e., 16 ms). Note that the speed of the second processing performed by the low-speed processing unit 11B may be changed as desired based on, for example, the number of captured images and gloss images used to generate the gloss-reduced image IMG. The low-speed processing unit 11B includes an image fusion unit 115, a tracking unit 116, and an actuator control unit 117.

[0034] When M sets of fusion data are stored (saved) in the fusion image database DB, the image fusion unit 115 reads out the M sets of fusion data. The image fusion unit 115 fuses the captured images and gloss images of the M sets of fusion data to generate a gloss-reduced image IMG (see FIG. 7 ) from which gloss has been removed.

[0035] The tracking unit 116 calculates the movement amount of the object Tg per 16 ms based on the gloss reduction image IMG output from the image fusion unit 115. The tracking unit 116 outputs the calculated movement amount of the object Tg per 16 ms to the actuator control unit 117, and outputs the movement amount of the object Tg per 1 ms to the image alignment unit 112.

[0036] The actuator control unit 117 controls the actuators AC to track the object Tg based on the movement amount of the object Tg per 16 ms calculated by the tracking unit 116. The actuator control unit 117 controls the actuators AC to track the object Tg until the end effector EF picks up the object Tg.

[0037] The memory 12 includes, for example, a random access memory (RAM) as a work memory used when executing each process of the processor 11, and a read only memory (ROM) that stores programs and data that define the operation of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM stores programs that define the operation of the processor 11.

[0038] The fusion image database DB is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a dynamic random access memory (DRAM), etc. The fusion image database DB stores (registers) fusion data of the object Tg to be picked, arranged in chronological order of when the captured images were captured.

[0039] The operation device INP is an interface that detects inputs from an operator and is configured with, for example, a mouse, a keyboard, or a touch panel. Upon receiving an operation from the operator, the operation device INP generates an electrical signal based on the operator's operation and transmits it to the image processing device P1.

[0040] Next, the first process and the glossy image generated by the first process will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a diagram showing an example of the first process of the picking system 100. Fig. 4 is a diagram showing an example of a captured image IMG11 and a glossy image IMG21. Note that in the glossy image IMG21 shown in Fig. 4, only a portion of the glossy area is illustrated and labeled for ease of understanding.

[0041] The high-speed processing unit 11A controls the illumination LT N times in accordance with the frame rate (=1000 fps) of the camera CM, with N frames being one cycle. The high-speed processing unit 11A executes first illumination control for capturing an image of the object Tg in the first M frames of the N frames (one cycle), and executes second illumination control for adjusting the total light amount of the illumination LT irradiated in the first processing and suppressing flickering of the illumination LT in the remaining (N-M) frames.

[0042] In one cycle of the first processing, the high-speed processing unit 11A executes a process of acquiring a captured image captured by the camera CM, a gloss detection process of detecting glossy areas (pixels) from the acquired captured image and generating a glossy image, a process of aligning the captured image and the glossy image based on the movement amount of the actuator AC per 1 ms acquired based on the encoder value of the actuator AC (image alignment process), and a process of controlling the illumination intensity of the illumination LT based on the glossy area and gloss intensity of the glossy image. Note that while the high-speed processing unit 11A executes one cycle of the first processing, the actuator AC executes movement control to track the target Tg based on the result of the second processing.

[0043] In the example shown in FIG. 3, in the first processing, that is, in the first processing executed in the first frame, the illumination LT has an illumination intensity L １ The object Tg is illuminated with an illumination intensity L １ An image of the object Tg illuminated with .gamma.=0.1 is captured, and the captured image is output to the high-speed processing unit 11A.

[0044] In the first processing, the high-speed processing unit 11A generates a glossy image IMG21 (black-and-white image) based on the captured image IMG11 (color image). The high-speed processing unit 11A also generates a first set of fusion data by associating the illumination intensity (default illumination intensity) of the first illumination LT with the captured image IMG11 and the glossy image IMG21, and stores (preserves) the data in the fusion image database DB.

[0045] Specifically, the high-speed processing unit 11A detects glossy pixels from the captured image IMG11 and generates a glossy image IMG21 that indicates the presence or absence and degree of glossiness of each pixel included in the captured image IMG11. The glossy image IMG21 is divided into glossy areas AR21A, AR21B, and AR21C that indicate glossy pixels where glossiness is detected, and an outside glossy area AR210 that indicates non-glossy pixel light where glossiness is not detected. The high-speed processing unit 11A calculates the illumination intensity of the illumination LT in the next frame based on the number of pixels included in the glossy areas AR21A to AR21C relative to the angle of view (total number of pixels) of the captured image IMG11 (i.e., the size of the glossy area), the gloss intensity of the glossy image IMG21 in the captured image IMG11, an area in the captured image IMG11 that corresponds to the outside glossy area AR210 of the glossy image IMG21, and the intensity of the pixels included in that area. ２ = 0.3. The high-speed processing unit 11A determines the illumination intensity L ２ The illumination LT is controlled to 0.3 to illuminate the object Tg.

[0046] In the first process executed in the second first process, the illumination LT has an illumination intensity L ２ The object Tg is illuminated with an illumination intensity L ２ An image of the object Tg illuminated at .gtoreq.0.3 is captured, and the captured image is output to the high-speed processing unit 11A.

[0047] In the second first processing, the high-speed processing unit 11A generates a gloss image (not shown, black-and-white image) based on the captured image (not shown, color image). The high-speed processing unit 11A reads the movement amount of the actuator AC per 1 ms before the captured image was captured, based on the encoder value of the actuator AC. The high-speed processing unit 11A aligns the captured image and gloss image acquired in the first first processing with the captured image and gloss image acquired in the second first processing, based on the movement amount of the actuator AC per ms. The high-speed processing unit 11A associates the illumination intensity of the illumination LT in the second first processing with the aligned captured image and gloss image, generates a second set of fusion data, and stores (saves) it in the fusion image database DB.

[0048] The high-speed processing unit 11A repeatedly executes the first process described above, and in the first process executed in the Mth iteration, the illumination LT is set to an illumination intensity L Ｍ The object Tg is illuminated with an illumination intensity L Ｍ An image of the object Tg illuminated with .gamma.=1.0 is captured, and the captured image is output to the high-speed processing unit 11A.

[0049] In the Mth iteration of the first processing, the high-speed processing unit 11A generates a gloss image (not shown, black-and-white image) based on a captured image (not shown, color image). The high-speed processing unit 11A reads the movement amount of the actuator AC per 1 ms before the captured image was captured, based on the encoder value of the actuator AC. The high-speed processing unit 11A aligns the captured image and gloss image acquired in the Mth iteration of the first processing, the captured image and gloss image acquired in the first iteration of the first processing, and the captured image and gloss image acquired in the second iteration of the first processing, based on the movement amount of the actuator AC per ms. The high-speed processing unit 11A associates the illumination intensity of the illumination LT in the Mth iteration of the first processing with the aligned captured image and gloss image, thereby generating the Mth set of fusion data, and stores (preserves) the data in the fusion image database DB.

[0050] After executing the first process M times, the low-speed processing unit 11B generates a gloss suppression image IMG (see FIG. 7) based on the M sets of fusion data. The low-speed processing unit 11B controls the actuator AC by image recognition or tracking using the generated gloss suppression image IMG.

[0051] Furthermore, the high-speed processing unit 11A determines the illumination intensity of the illumination LT from the (M+1)th frame to the Nth frame so that the total light amount of the illumination LT in N frames (one cycle) becomes a predetermined value (2.5 in this case), and controls the illumination intensity of the illumination LT. Furthermore, from the (M+1)th frame to the Nth frame, the high-speed processing unit 11A omits, in the first processing, the alignment processing of the captured image based on the movement amount of the actuator AC per 1 ms acquired based on the encoder value of the actuator AC (image alignment processing), and the control processing of the illumination intensity of the illumination LT based on the glossy area and gloss intensity of the glossy image.

[0052] Next, an example of controlling the illumination intensity of the illumination LT will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of controlling the illumination intensity. It goes without saying that the example of controlling the illumination intensity shown in Fig. 5 is just an example and is not limited to this. Furthermore, any value may be set for each threshold value in the description of Fig. 5.

[0053] When the average pixel value of the non-glossy area based on the pixel values ​​of the pixels included in the non-glossy area of ​​the captured image (hereinafter referred to as "non-glossy area intensity DI") and the number of pixels in the non-glossy area (hereinafter referred to as "non-glossy area range DA") is equal to or less than the threshold value a (DI / DA≦a), and the non-glossy area range DA is greater than the threshold value b (DA>b), the high-speed processing unit 11A determines whether the illumination intensity L of the previous frame is greater than the threshold value b (DA>b). ｉ―１ Based on the outer gloss intensity DI, the illumination intensity L of the previous illumination LT is calculated. ｉ―１ Illumination intensity L greater than ｉ is determined as the illumination intensity of the illumination LT for the next frame. Note that "i" is an integer, and 1≦i≦M. Also, "F" indicates the base of the illumination change.

[0054]

[0055] Furthermore, when the ratio of the non-glossy intensity DI of the captured image to the non-glossy area range DA is equal to or greater than a threshold c (DI / DA≧c), and the non-glossy area range DA is smaller than a threshold d (DA<d), and the magnitude of the change in the number of pixels in the glossy area (hereinafter referred to as the "glossy area range SA") relative to the change in the pixel value of the pixel included in the glossy area (hereinafter referred to as the "gloss intensity SI") is less than a threshold e (dSA / dSI<e), the high-speed processing unit 11A determines the illumination intensity L of the illumination LT of the previous frame. ｉ―１ Based on the non-glossy area range DA, the illumination intensity L of the illumination LT of the previous frame is calculated. ｉ―１ Illumination intensity L smaller than ｉ is determined as the illumination intensity of the illumination LT for the next frame.

[0056]

[0057] The high-speed processing unit 11A executes lighting control to gradually increase the lighting intensity of the lighting LT, and then executes lighting control in the second lighting control for the first week to maintain the lighting intensity of the lighting LT at a predetermined lighting intensity so that the total light amount for the first week becomes a predetermined value (constant value).

[0058] For example, in the example shown in FIG. 5, the high-speed processing unit 11A executes a first lighting control to control the lighting intensity of the lighting LT for each frame as shown in the lighting intensity graphs LGP1 and LGP2, and executes a second lighting control to adjust the lighting intensity of the lighting LT so that the total light amount of each of the N frames becomes a predetermined value.

[0059] The illumination intensity graph LGP1 is a graph showing the illumination intensity of the illumination LT in the first week of the first illumination control. The illumination intensity graph LGP1 shows an example of changes in illumination intensity when illumination control is executed to increase the illumination intensity of the illumination LT. The illumination intensity graph LGP2 is a graph showing the illumination intensity of the illumination LT in the second illumination control in the first week. Note that the illumination intensity graph LGP2 shown in FIG. 5 shows an example in which the illumination intensity is constant, but is not limited to this.

[0060] In the example shown in FIG. 5, the high-speed processing unit 11A executes a first lighting control in the first week to gradually increase the lighting intensity of the lighting LT, and then executes a second lighting control in the first week to maintain the lighting intensity of the lighting LT at a predetermined lighting intensity so that the total light amount of each of the N frames becomes a predetermined value.

[0061] Illumination intensity graph LGP3 is a graph showing the illumination intensity of the light LT in the first illumination control in the second week. Illumination intensity graph LGP1 shows an example of changes in illumination intensity when illumination control is executed to decrease the illumination intensity of the light LT. Illumination intensity graph LGP4 is a graph showing the illumination intensity of the light LT in the second illumination control in the second week.

[0062] In the example shown in Figure 5, the high-speed processing unit 11A performs lighting control in the first lighting control in the second week to gradually reduce the lighting intensity of the lighting LT, and then performs lighting control in the second lighting control in the second week to maintain the lighting intensity of the lighting LT at a predetermined lighting intensity so that the total light amount of each of the N frames becomes a predetermined value.

[0063] Next, the operational procedure of the first process executed by the high-speed processing section 11A will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the first process procedure of the image processing device P1.

[0064] The high-speed processing unit 11A sets a counter C indicating the number of times the captured image has been captured to C=0 (zero), a flag F indicating that the current frame is the first frame of the first process to F=1, and a flag S for executing fusion data to S=1 (St11). The high-speed processing unit 11A determines whether the currently set number of frames F is F=1 (St12).

[0065] If the high-speed processing unit 11A determines in step St12 that the currently set number of frames F is F=1 (St12, YES), it controls the illumination LT with the illumination intensity set as the default illumination intensity (St13).

[0066] On the other hand, if the high-speed processing unit 11A determines in step St12 that the currently set frame number F is not F = 1 (St12, NO), it executes lighting control of the lighting LT based on the lighting intensity obtained by determining the lighting intensity using the captured image and gloss image acquired in the previous frame (St14).

[0067] The high-speed processing unit 11A acquires an image captured by the camera CM (Step 15), and increments the counter C by 1 (Step 16). The high-speed processing unit 11A detects glossy pixels from the captured image (Step 17). The high-speed processing unit 11A determines whether the current value of the counter C is C<M (Step 18).

[0068] If the high-speed processing unit 11A determines in step St18 that the current counter C is C<M (YES in St18), it sets the counter S to S=1 (St19). The high-speed processing unit 11A determines the illumination intensity for the next frame so that the total amount of light irradiated by the illumination LT within N frames becomes a predetermined value (St20). The high-speed processing unit 11A proceeds to the process of step St26.

[0069] On the other hand, if the high-speed processing unit 11A determines in step St18 that the current counter C is not C<M (NO in St18), it sets the counter S to S=0 (zero) (St21). The high-speed processing unit 11A determines the illumination intensity of the illumination LT that will illuminate the object Tg in the next frame based on the glossy area range and glossy area intensity, and the non-glossy area range and non-glossy area intensity (St22).

[0070] The high-speed processing section 11A determines whether the current counter C is C=1 (St23).

[0071] If the high-speed processing unit 11A determines in step St23 that the current counter C is C=1 (St23, YES), it stores (saves) the illumination intensity of the illumination LT when the captured image was captured, and the captured image and the gloss image in the fusion image database DB (St25). Here, if the high-speed processing unit 11A determines in step St23 that the current counter C is C=1, there is no target for alignment of the object Tg appearing in the captured image and the gloss image (i.e., the captured image and the gloss image acquired in the previous frame), so the alignment process is omitted.

[0072] On the other hand, if the high-speed processing unit 11A determines in step St23 that the current counter C is not C=1 (NO in St23), it acquires the movement amount of the actuator AC per 1 ms based on the encoder value of the actuator AC, with the captured image and gloss image at the time when the counter C=1 are used as references. Based on the movement amount of the actuator AC, the high-speed processing unit 11A executes a position adjustment process to align the position of the object Tg appearing in the captured image and gloss image captured in the current frame with the position of the object Tg appearing in the captured image and gloss image captured in the frame at which the counter C=1 (St24). The high-speed processing unit 11A stores (saves) the illumination intensity of the illumination LT when the captured image was captured and the aligned captured image and gloss image in the fusion image database DB (St25).

[0073] The high-speed processing unit 11A sets the current flag F to F=0 (zero) (St26), and determines whether the current counter C is C≦N, that is, whether the first lighting control is completed (St27).

[0074] If the high-speed processing unit 11A determines in step St27 that the current counter C is not C≦N (St27, NO), it sets the current counter C to C=0 (zero) and determines whether the picking of the target object Tg by the actuator AC has been completed (St28).

[0075] On the other hand, if the high-speed processing unit 11A determines in step St27 that the current counter C is C≦N (St27, YES), it determines whether the work (e.g., picking) being performed on the object Tg by the actuator AC has been completed (St28).

[0076] If the high-speed processing unit 11A determines in the processing of step St28 that the work being performed on the object Tg by the actuator AC has been completed (St28, YES), it ends the operation procedure of the first processing shown in FIG. 6 .

[0077] On the other hand, if the high-speed processing section 11A determines in the processing of step St28 that the work being performed by the actuator AC on the object Tg has not been completed (St28, NO), it returns to the processing of step St12.

[0078] As described above, the image processing device P1 can adjust the illumination intensity of the illumination LT in accordance with the current state of gloss generation, based on the gloss intensity of the captured image and the gloss image, the number of gloss pixels, etc. Furthermore, by controlling the illumination intensity so that the total amount of light in one cycle (N frames) becomes a predetermined value, the image processing device P1 can realize illumination control for obtaining the captured image and gloss image for acquiring one gloss-reduced image, and illumination control for reducing the mental and physical burden on the worker due to fluctuations in illumination intensity.

[0079] Furthermore, as described above, the image processing device P1 can generate fusion data for generating a gloss-reduced image IMG (see FIG. 7 ) by generating M sets of captured images and gloss images, each captured with the same or different illumination intensities and with the position of the object Tg aligned. Furthermore, the image processing device P1 can calculate the movement amount of the actuator AC, i.e., the camera CM, every 1 ms based on the encoder value of the actuator AC, thereby enabling the captured images and gloss images to be aligned with higher precision.

[0080] Next, the operational procedure of the second process executed by the low-speed processing section 11B will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the second process procedure of the image processing device P1.

[0081] The low-speed processing unit 11B determines whether the currently set flag S is S=1 (St31).

[0082] If the low-speed processing unit 11B determines in step St31 that the currently set flag S is S = 1 (St31, YES), it reads and acquires M sets of fusion data (i.e., the illumination intensity when each captured image was captured, the captured image after alignment, and the gloss image) from the fusion image database DB (St32).

[0083] On the other hand, if the low-speed processing unit 11B determines in step St31 that the currently set flag S is not S=1 (St31, NO), the low-speed processing unit 11B returns to the processing of step St31.

[0084] The low-speed processing unit 11B generates a gloss suppression image IMG (see FIG. 7) based on the M sets of fusion data (St33).

[0085] The low-speed processing unit 11B uses the gloss-reduced image IMG to recognize an object Tg appearing in the gloss-reduced image IMG. Based on the position of the recognized object Tg, the low-speed processing unit 11B performs object tracking to track the object Tg that has moved during 16 ms (St34).

[0086] The low speed processing unit 11B calculates the amount of movement of the object Tg that has moved during 16 ms, determines the amount of movement of the actuator AC, and executes control to move the actuator AC by the determined amount of movement (St35).

[0087] The low-speed processing unit 11B determines whether the operation (for example, picking) performed by the actuator AC on the target object Tg has been completed (St36).

[0088] If the low-speed processing unit 11B determines in the processing of step St36 that the work being performed on the object Tg by the actuator AC has been completed (St36, YES), it terminates the operation procedure of the second processing shown in Figure 7.

[0089] On the other hand, if the low-speed processing unit 11B determines in the processing of step St36 that the work being performed by the actuator AC on the object Tg has not been completed (St36, NO), it returns to the processing of step St31.

[0090] As described above, the image processing device P1 can generate the gloss reduction image IMG. Furthermore, the image processing device P1 can object-recognize the target object Tg from the generated gloss reduction image IMG, and determine (calculate) with high accuracy the movement amount of the actuator AC that performs the work on the target object Tg, thereby realizing control of the actuator AC.

[0091] Next, the generation (fusion) process of the gloss reduction image IMG will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of generation of the gloss reduction image IMG. Note that the captured image and the gloss image shown in Fig. 8 are merely examples, and the present invention is not limited to these.

[0092] The low-speed processing unit 11B acquires each of M sets of captured images IMG31, IMG32, ..., IMG3M and each of glossy images IMG41, IMG42, ..., IMG4M stored in the fused image database DB. The low-speed processing unit 11B extracts and fuses non-glossy pixels from among pixels at the same position included in each of the captured images IMG31 to IMG3M to generate a gloss-reduced image IMG.

[0093] For example, the low-speed processing unit 11B determines whether or not a pixel at position p in each of the captured images IMG31 to IMG3M is a glossy pixel, and extracts non-glossy pixels from among the pixels at position p in the captured images IMG31 to IMG3M. The following (Equation 3) is a determination equation for determining whether or not a pixel at position p in the captured image captured in the i-th frame is a glossy pixel. Note that position p here is position information on the captured images IMG31 to IMG3M, and indicates the two-dimensional position of the pixel included in the captured images IMG31 to IMG3M.

[0094]

[0095] The low-speed processing unit 11B performs fusion by averaging the sum of the pixel values ​​of the non-glossy pixels at position p extracted from each of the captured images IMG31 to IMG3M (i.e., the numerator in (Equation 4)) by the sum of the illumination intensities at the time of capturing each of the captured images IMG31 to IMG3M (i.e., the denominator in (Equation 4)). The pixel value I of the pixel at position p after fusion is ｐ is calculated by (Equation 4).

[0096]

[0097] The low-speed processing unit 11B executes a calculation process of the pixel values ​​after fusion at all positions included in the captured images IMG31 to IMG3M using (Equation 3) and (Equation 4). The low-speed processing unit 11B generates the gloss-reduced image IMG based on the calculated pixel values ​​at all positions.

[0098] This allows the image processing device P1 to generate a gloss-reduced image IMG in which gloss is reduced or removed from a captured image containing glossy pixels. By reducing or removing gloss, the image processing device P1 can recognize the shape, posture, position, etc. of the object Tg captured in the gloss-reduced image IMG with higher accuracy.

[0099] (Variations of First Embodiment) The picking system 100 according to the first embodiment described above has shown an example in which the illumination intensity of the illumination LT is controlled based on the gloss of the captured image, and a plurality of captured images captured with the illumination intensity changed are combined with the gloss image to generate a gloss-suppressed image. In the picking system 100 according to the following variations of the first embodiment, polarizing plates are provided in the illumination LT and the camera CM, respectively, and a gloss-suppressed image is generated by removing a portion of the incident light that enters the lens and image sensor of the camera CM.

[0100] (Additional Notes) The above description of each embodiment discloses the following techniques.

[0101] (Technology 1) An image processing method performed by an image processing device P1 capable of communicating between a camera CM capable of capturing an image of an object Tg and a lighting LT that illuminates the object Tg, the image processing method comprising: acquiring a captured image of the object Tg, detecting glossy pixels among pixels included in the captured image, generating a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image, and executing a first process multiple times to determine the illumination intensity of the lighting LT for a next frame based on the captured image and the gloss image and control the lighting LT; and executing a second process while the first process is being executed multiple times to generate a fusion image (gloss-reduced image IMG) by fusing multiple captured images based on the multiple gloss images generated in the first process. This allows the image processing device P1 to adjust the illumination intensity of the lighting LT in response to a current gloss state based on the gloss intensity or the number of gloss pixels of the captured image and the gloss image, etc. Furthermore, the image processing device P1 can generate and acquire a gloss-suppressed image IMG (see FIG. 7) in which gloss is suppressed by fusing captured images captured with the same or different illumination intensities.

[0102] (Technology 2) The image processing method according to (Technology 1), wherein in the first process, an illumination intensity of the illumination LT for the next frame is determined based on the number of glossy pixels, the pixel values ​​of the glossy pixels, and the pixel values ​​and number of non-glossy pixels included in the captured image. This allows the image processing device P1 to acquire multiple captured images in which the object Tg is illuminated with various illumination intensities, and thereby perform illumination control using these captured images to obtain a captured image with reduced gloss.

[0103] (Technology 3) The image processing method according to (Technology 1) or (Technology 2), wherein in the first processing, an illumination intensity of the illumination LT for the next frame is determined based on the captured image, the gloss image, and the illumination intensity of the illumination LT when the captured image was captured. This allows the image processing device P1 to acquire multiple captured images in which the object Tg is illuminated with various illumination intensities, and therefore allows the image processing device P1 to generate and acquire a gloss-suppressed image IMG in which gloss is suppressed using these captured images.

[0104] (Technology 4) The image processing method according to any one of (Technology 1) to (Technology 3), wherein the camera CM is movable, and the first process includes: acquiring information about the movement of the camera CM; aligning the plurality of consecutively captured captured images and the glossy image based on the information about the movement of the camera CM; and recording the aligned captured images and the glossy image in chronological order. This enables the image processing device P1 to align the captured images and the glossy image with higher accuracy based on the amount of movement of the camera CM per frame (1 ms).

[0105] (Technology 5) The image processing method according to any one of (Technology 1) to (Technology 4), wherein the second process includes acquiring a plurality of aligned captured images and the gloss image, extracting non-glossy pixels determined to be non-glossy from among pixels included in the plurality of captured images based on the gloss image for each pixel of the captured images captured by the camera CM, and fusing the extracted non-glossy pixels for each pixel of the captured images to generate the fused image (gloss-reduced image IMG). This allows the image processing device P1 to generate and acquire the gloss-reduced image IMG in which gloss is reduced by averaging and fusing the sum of pixel values ​​of non-glossy pixels extracted from each of the captured images IMG31 to IMG3M (i.e., the numerator in (Equation 4)) at position p by the sum of illumination intensities when each of the captured images IMG31 to IMG3M was captured (i.e., the denominator in (Equation 4)).

[0106] (Technology 6) The image processing method according to any one of (Technology 1) to (Technology 5), wherein the first process includes first lighting control that controls the lighting intensity of the lighting LT that illuminates the object Tg imaged by the camera CM, and second lighting control that controls the lighting intensity of the lighting LT so that a total amount of light of the lighting LT irradiated during execution of the first process becomes a predetermined amount of light. As a result, the image processing device P1 controls the lighting intensity so that the total amount of light in one cycle (N frames) becomes a predetermined value, thereby realizing lighting control for obtaining captured images and gloss images for acquiring one gloss-reduced image, and lighting control for reducing the physical and mental burden on an operator due to fluctuations in lighting intensity.

[0107] (Technology 7) The image processing method according to (Technology 6), wherein the first processing is performed by executing the first lighting control and the second lighting control a total of N (N: an integer of 3 or more) times in accordance with the frame rate of the camera CM, executing the first lighting control M times until capturing M (M: an integer of 2 or more, M<N) images used in the fusion image (gloss-reduced image IMG), calculating at least one lighting intensity at which the total amount of light irradiated during the execution of the first processing becomes the predetermined light amount based on the lighting intensities at which each of the M captured images was captured, and executing the second lighting control based on the calculated lighting intensity. As a result, the image processing device P1 can control the lighting intensity so that the total amount of light in one cycle (N frames) becomes a predetermined value by executing the second lighting control for the remaining (N-M) frames based on the first lighting control executed for M frames. Therefore, the image processing device P1 can realize lighting control for capturing an image to obtain a single gloss suppression image and obtaining a gloss image, as well as lighting control for suppressing the physical and mental stress on the worker due to fluctuations in lighting intensity.

[0108] (Technology 8) An image processing device P1 capable of communication between a camera CM capable of capturing an image of an object Tg and a lighting LT that illuminates the object Tg, the image processing device P1 comprising: a first control unit (high-speed processing unit 11A) that executes a first process multiple times to acquire a captured image of the object Tg, detect glossy pixels among pixels included in the captured image, generate a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image, and determine an illumination intensity of the lighting LT for a next frame based on the captured image and the gloss image to control the lighting LT; and a second control unit (low-speed processing unit 11B) that executes a second process while executing the first process multiple times to generate a fusion image (gloss-reduced image IMG) by fusing a plurality of the captured images based on the plurality of gloss images generated in the first process. This allows the image processing device P1 to adjust the illumination intensity of the lighting LT in response to a current gloss state based on the gloss intensity or the number of gloss pixels of the captured image and the gloss image, etc. Furthermore, the image processing device P1 can generate and acquire a gloss-suppressed image IMG (see FIG. 7) in which gloss is suppressed by fusing captured images captured with the same or different illumination intensities.

[0109] (Technology 9) An image acquisition program executed by a processor 11 capable of communicating between a camera CM capable of capturing an image of an object Tg and a lighting LT that illuminates the object Tg, the image acquisition program causing the processor 11 to: acquire a captured image of the object Tg, detect glossy pixels from among pixels included in the captured image, generate a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image, and execute a first process multiple times to determine the illumination intensity of the lighting LT for a next frame and control the lighting LT based on the captured image and the gloss image, and execute a second process while the first process is executed multiple times to generate a fusion image (gloss-reduced image IMG) by fusing a plurality of the captured images based on the plurality of gloss images generated in the first process. This allows an image processing device P1 to adjust the illumination intensity of the lighting LT in response to a current gloss generation state based on the gloss intensity or the number of gloss pixels of the captured image and the gloss image, etc. Furthermore, the image processing device P1 can generate and acquire a gloss-suppressed image IMG (see FIG. 7) in which gloss is suppressed by fusing captured images captured with the same or different illumination intensities.

[0110] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention.

[0111] This application is based on a Japanese patent application (Patent Application No. 2024-085844) filed on May 27, 2024, the contents of which are incorporated herein by reference.

[0112] The present disclosure is useful as an image acquisition method, an image acquisition device, and an image acquisition program for acquiring a captured image with reduced gloss.

[0113] 11 Processor 11A High-speed processing unit 11B Low-speed processing unit 12 Memory 100 Picking system 111 Gloss detection unit 112 Image alignment unit 113 Illumination intensity determination unit 114 Illumination control unit 115 Image fusion unit 116 Tracking unit 117 Actuator control unit AC Actuator AR21A, AR21B, AR21C Gloss area AR210 Outside gloss area BC Belt conveyor CM Camera DA Outside gloss area range DB Fusion image database EF End effector IMG Gloss suppression image IMG11, IMG1M, IMG31, IMG32, IMG3M Captured image IMG21, IMG2M, IMG41, IMG42, IMG4M Gloss image LT Illumination P1 Image processing device Tg Object

Claims

1. An image processing method performed by an image processing device capable of communicating between a camera capable of capturing an image of an object and a light that illuminates the object, the image processing method comprising: acquiring an image of the object; detecting glossy pixels among the pixels included in the captured image; generating a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image; determining the illumination intensity of the illumination for the next frame based on the captured image and the gloss image and controlling the illumination, and while the first process is being performed multiple times, performing a second process to generate a fused image by fusing multiple captured images based on the multiple gloss images generated in the first process.

2. The image processing method of claim 1, wherein in the first process, the illumination intensity of the illumination for the next frame is determined based on the number of glossy pixels, the pixel values ​​of the glossy pixels, and the pixel values ​​and number of non-glossy pixels among the pixels included in the captured image.

3. The image processing method of claim 1, wherein in the first processing, the illumination intensity of the illumination for the next frame is determined based on the captured image and the glossy image, and the illumination intensity of the illumination when the captured image was captured.

4. The image processing method of claim 1, wherein the camera is movable, and in the first process, information about the camera is acquired, and based on the information about the camera, alignment of the multiple captured images and the glossy image captured in succession is performed, and the aligned captured images and the glossy images are recorded in chronological order.

5. The image processing method of claim 1, wherein the second process acquires a plurality of aligned captured images and glossy images, extracts non-glossy pixels determined to be non-glossy from among the pixels contained in the plurality of captured images based on the glossy images, and performs a process for each pixel of the captured images captured by the camera, and fuses the extracted non-glossy pixels for each pixel of the captured images to generate the fused image.

6. The image processing method of claim 1, wherein the first processing includes a first lighting control that controls the lighting intensity that illuminates the object imaged by the camera, and a second lighting control that controls the lighting intensity of the lighting so that the total amount of light emitted during execution of the first processing becomes a predetermined amount of light, and the first lighting control and the second lighting control are executed in accordance with the frame rate of the camera.

7. The image processing method of claim 6, wherein the first processing is executed N times (N is an integer equal to or greater than 3) corresponding to the frame rate of the camera CM, the first lighting control is executed M times until M (M is an integer equal to or greater than 2, M<N) captured images used in the fusion image are captured, and at least one lighting intensity at which the total amount of light irradiated during the execution of the first processing becomes the predetermined amount of light is calculated based on the lighting intensity at which each of the M captured images was captured, and the second lighting control is executed based on the calculated lighting intensity.

8. An image processing device capable of communicating between a camera capable of capturing an image of an object and a light that illuminates the object, comprising: a first control unit that executes a first process multiple times to acquire an image of the object, detect glossy pixels from among the pixels included in the captured image, generate a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image, and determine the illumination intensity of the illumination for the next frame based on the captured image and the gloss image to control the illumination; and a second control unit that executes a second process while executing the first process multiple times to generate a fusion image by fusing a plurality of the captured images based on the plurality of gloss images generated in the first process.

9. An image acquisition program executed by a processor capable of communicating between a camera capable of capturing an image of an object and a light that illuminates the object, the image acquisition program causing the processor to perform the following steps: acquiring a captured image of the object; detecting glossy pixels from among the pixels included in the captured image, generating a gloss image indicating the presence or absence and degree of gloss of each pixel included in the captured image, and determining the illumination intensity of the illumination for the next frame based on the captured image and the gloss image, and controlling the illumination; and executing, while the first process is being performed multiple times, a second process that generates a fused image by fusing multiple captured images based on the multiple gloss images generated in the first process.

Images