Image processing device, image processing method and program

Abstract

To shorten time required for an inspection of an object.SOLUTION: An image processing device acquires a plurality of images to be obtained by imaging each of a plurality of objects under a plurality of imaging conditions and controls conversion processing to each of the plurality of images and inspection processing based on the converted images. The conversion processing and the inspection processing are performed in parallel by pipeline processing and pieces of conversion processing to each of the plurality of images are performed in parallel.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to image processing techniques for inspecting objects. [Background technology]

[0002] When inspecting the appearance of industrial products, it is common to use images obtained by capturing images of the product. Patent Document 1 discloses a method for extracting defect candidate positions in an inspection object based on multiple captured images of the inspection object illuminated with light from different directions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-191105 Summary of the Invention [Problem to be solved by the invention]

[0004] In the prior art, there was a problem that when multiple tests were performed, the tests took a long time.

[0005] Therefore, the object is to reduce the time required to inspect an object. [Means for solving the problem]

[0006] In order to solve the above problem, the image processing device of the present invention has an acquisition means for acquiring multiple images obtained by imaging each of multiple objects under multiple imaging conditions, and a control means for controlling a conversion process for each of the multiple images and an inspection process based on the converted images, wherein the control means performs the conversion process and the inspection process in parallel by pipeline processing, and performs the conversion process for each of the multiple images in parallel. [Effects of the Invention]

[0007] According to the present invention, the time required to inspect an object can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] A diagram showing the appearance and hardware configuration of the visual inspection system. [Figure 2] Diagram showing the functional configuration of the visual inspection system [Figure 3] Flowchart showing processing in the imaging control device [Figure 4] Flowchart showing processing in the imaging device [Figure 5] Flowchart showing processing in an image processing device [Figure 6] Flowchart showing processing in an image processing device [Figure 7] Timing chart of processing in the image processing device [Figure 8] Flowchart showing processing in an image processing device [Figure 9] Timing chart of processing in the image processing device [Figure 10] A diagram showing an example of a user interface [Figure 11] 1 is a flowchart showing processing in an image processing system; [Figure 12] Flowchart showing processing in an image processing device [Figure 13] Flowchart showing processing in an image processing device [Figure 14] Timing chart of processing in the image processing device DETAILED DESCRIPTION OF THE INVENTION

[0009] Each embodiment will be described below with reference to the drawings. Note that the following embodiments do not necessarily limit the present invention. Furthermore, not all of the combinations of features described in each embodiment are necessarily essential to the solution of the present invention.

[0010] [First embodiment] When inspecting multiple objects, such as mass-produced industrial products, it is necessary to complete everything from imaging to inspection processing within a short takt time (the manufacturing time per product). Pipeline processing, in which processing is divided and executed sequentially, is known as a method for shortening processing time. In other words, by dividing the process from imaging to inspection processing into steps that can each be completed within the takt time and connecting them as a pipeline, inspection within the takt time becomes possible. However, improving inspection accuracy requires a combination of high-resolution images and multifaceted defect detection methods, which can sometimes result in inspections not being able to be completed within the takt time.

[0011] Therefore, in this embodiment, conversion processes such as decoding and degamma processing for captured images and multiple types of inspection processes are performed in parallel using pipeline processing. Furthermore, conversion processes for multiple captured images are performed in parallel. This allows inspection to be performed within the takt time without increasing computational resources, and improves inspection accuracy.

[0012] <Appearance and hardware configuration of the visual inspection system> Fig. 1(a) is a diagram showing an example of the hardware configuration of a visual inspection system according to this embodiment. Fig. 1(b) is a front view of the overall appearance of the visual inspection system, and Fig. 1(c) is a top view of the overall appearance of the visual inspection system. The visual inspection system according to this embodiment includes an image processing system 1, a start signal output interface 101, a transport control device 111, and a transport device 112.

[0013] The image processing system 1 has an imaging control device 102, an imaging device 103, an image processing device 104, a display 105, a mouse 106, a keyboard 107, and a lighting device 108. The image processing system 1 is connected to a transport control device 111 that controls a transport device 112. The transport control device 111 transports an object 113 to be inspected to the image processing system 1 using the transport device 112, and sends an inspection start signal to the image processing system 1 via the start signal output interface 101.

[0014] The imaging control device 102 has a control unit 114 and controls the imaging device 103 and the lighting device 108 to capture an image of the object 113 in synchronization with the lighting of the light source. The imaging control device 102 is, for example, a one-board microcomputer equipped with a GPIO (General Purpose Input / Output). Specifically, upon receiving an inspection start signal via a start signal input interface 115, the imaging control device 102 sends an imaging instruction to the imaging device 103 via a release signal output interface 116. The imaging control device 102 also receives a synchronization signal via a synchronization signal input interface 117 from the imaging device 103 to notify an external strobe light source of the light emission timing in synchronization with the imaging. Furthermore, in response to the received synchronization signal, the imaging control device 102 turns on the light sources of the lighting device 108 in a predetermined order and combination via a lighting signal output interface 119. Through the above operations, an image of the object 113 illuminated by light from a predetermined light source can be captured. In this embodiment, the lighting of the light source and the imaging are synchronized based on a synchronization signal output from the imaging device 103, but the method of synchronizing the lighting of the light source and the imaging is not limited to the above example. For example, a pulse signal may be output from the control unit 114 to simultaneously output an imaging instruction to the imaging device 103 and an instruction to turn on the light source to the lighting device 108. The imaging control device 102 is connected to the image processing device 104 via a USB interface 118, and receives commands from the image processing device 104 and provides information indicating the state of the imaging control device 102 to the image processing device 104.

[0015] The imaging device 103 has a control unit 125 and an imaging optical system 121 including a lens, an imaging element, etc. The imaging device 103 generates a captured image by quantizing an optical image obtained by capturing an image based on an imaging command received via a release signal input interface 120 using an image processing engine 123. The imaging device 103 transfers the generated captured image to the image processing device 104 via a USB interface 124. Note that, in this embodiment, an example is described in which a still image captured using a digital camera is acquired and used, but a predetermined frame may be extracted as a still image from a moving image captured using a video camera and used. The imaging device 103 sends a synchronization signal to the imaging control device 102 via a synchronization signal output interface 122.

[0016] The image processing device 104 has a RAM 126, a ROM 127, a CPU 128, a GPU 129, and a USB interface 130. Each component is connected via an internal bus. The processing shown in the flowcharts described below is stored in the ROM 127 as program code. This program code is loaded into the RAM 126 and executed by the CPU 128 and GPU 129.

[0017] The illumination device 108 includes multiple light sources 109. In this embodiment, the light sources 109 are LEDs, but other light sources such as xenon lamps may also be used. The multiple light sources 109 are installed in a hemispherical arrangement above the object 113, and since at least one of the zenith angle and the azimuth angle differs from each other, it is possible to irradiate the object 113 with light from different directions. In inspections using captured images, it is necessary to change the light irradiation method depending on the appearance inspection item. For example, when inspecting the gloss of an object, it is necessary to irradiate light from a direction that allows light reflected specularly from the inspection surface to be captured. Furthermore, when inspecting the color or surface shape of an object, it is necessary to irradiate light from a direction that does not allow light reflected specularly from the inspection surface to be captured. Therefore, some of the light sources 109 are installed in a direction with a large incident angle with respect to the installation surface of the object 113, enabling imaging under imaging conditions that make it easy for the imaging device 103 to receive diffusely reflected light. Furthermore, some of the light sources 109 are installed in a direction where the angle of incidence with respect to the installation surface of the object 113 is small, enabling imaging under imaging conditions where the imaging device 103 is likely to receive specularly reflected light. The light sources 109 may each have a different light-emitting surface or spectral characteristics. For example, spot-illumination type light sources such as light sources 1091 to 1098 may be installed in a direction where the angle of incidence with respect to the installation surface of the object 113 is large, and a ring light source in which light-emitting elements are arranged in a circular ring shape such as light source 1099 may be installed in a direction where the angle of incidence is small. The illumination device 108 turns on a predetermined light source among the light sources 109 for a preset time in response to a command from the control unit 114.

[0018] In the image processing system 1 of this embodiment, the imaging control device 102, the imaging device 103, the image processing device 104, and the lighting device 108 are separate devices, but a plurality of devices may be integrated.

[0019] <Processing performed by the image processing system> FIG. 2 is a diagram showing an example of the functional configuration of the appearance inspection system according to this embodiment. The control unit 114 of the imaging control device 102 includes an imaging control unit 202. The imaging control unit 202 includes a release signal output unit 207, a synchronization signal input unit 208, a synchronization signal count unit 209, and an illumination signal output unit 210. The control unit 125 of the imaging device 103 includes an imaging unit 203. The imaging unit 203 includes a release signal input unit 211, a control unit 212, a synchronization signal output unit 213, and an image transmission unit 214. The image processing device 104 includes an image processing unit 204. The image processing unit 204 includes an inspection image acquisition unit 215, an image conversion unit 216, an image synthesis unit 217, a color and shape inspection unit 218, a gloss inspection unit 219, and an output unit 220. The transport control device 111 includes a start signal output unit 201. The transport device 112 includes a transport unit 206. The lighting device 108 includes a lighting unit 205 .

[0020] Fig. 11 is a flowchart of processing executed by the image processing system 1 in this embodiment. When the object 113 is transported to a predetermined position by the transport device 112, the start signal output unit 201 of the transport control device 111 sends an inspection start signal to the imaging control unit 202. The processing in Fig. 11 starts when the imaging control unit 202 receives the inspection start signal. Hereinafter, each step (process) will be represented by adding an S before the reference number.

[0021] In S1101, the imaging control unit 202 and the imaging unit 203 acquire multiple captured images obtained by imaging the object 113 with a predetermined light source turned on and transfer the captured images to the image processing unit 204. Specifically, the release signal output unit 207 first receives an inspection start signal and sends a release signal to the release signal input unit 211. When the control unit 212 detects that the release signal input unit 211 has received the release signal, it executes an imaging operation. In this embodiment, high-speed continuous imaging is performed using a known continuous imaging function. The continuous imaging function is a function that repeatedly captures images at a predetermined speed while a release signal is continuously sent. In this embodiment, imaging is performed at 30 frames per second. During imaging, the control unit 212 causes the synchronization signal output unit 213 to output a synchronization signal to synchronize the timing of the lighting of the external strobe light source with the timing of the shutter curtain opening. When the synchronization signal input unit 208 inputs a synchronization signal, the synchronization signal count unit 209 counts the number of synchronization signal inputs, i.e., the number of captured images. The lighting signal output unit 210 switches the light source to be turned on sequentially according to the number of captured images, and outputs a lighting signal to the illumination unit 205. The image transmission unit 214 repeatedly transfers captured images obtained by turning on a predetermined light source and capturing an image of the object 113 to the inspection image acquisition unit 215 as needed. Through the above processing, in S1101, the imaging control unit 202 and the imaging unit 203 can transfer multiple captured images to the image processing unit 204.

[0022] In S1102, the color and shape inspection unit 218 and the gloss inspection unit 219 perform inspection processing based on the multiple captured images acquired by the inspection image acquisition unit 215. The color and shape inspection unit 218 detects defects by performing spatial filtering on the inspection image, which includes normal information and color information obtained by combining captured images using the photometric stereo method. The response value to the spatial filter is integrated and quantified to determine the degree of abnormality, and the pass / fail of the inspection is determined by comparing the degree of abnormality with a judgment threshold. The calculated degree of abnormality is displayed in an abnormality display area 1009 of the inspection screen user interface (UI) shown in FIG. 10. The judgment threshold can be set in a threshold setting area 1003 of the inspection screen UI shown in FIG. 10. The gloss inspection unit 219 uses a captured image obtained by receiving specularly reflected light from the inspection surface of the object as an inspection image containing gloss information, and performs spatial filtering in the same manner as described above to detect defects. The photometric stereo method is a technique that can acquire normal information representing the surface shape of an object and color information representing the color of the object by combining captured images corresponding to multiple lighting directions. The above inspection process method is an example, and other inspection methods may be used.

[0023] In this embodiment, the appearance inspection items are three types: color, surface shape, and gloss. The appearance inspection items are not limited to the above examples, and may be anything that represents the appearance and can be identified by capturing an image. For example, material or pattern may also be used. The inspection image acquisition unit 215 displays the inspection screen UI shown in FIG. 10 on the display 105 and accepts instructions from the user. One or more appearance inspection items are set in the inspection screen UI shown in FIG. 10. The user can input information into the inspection screen UI displayed on the display 105 using the mouse 106 or keyboard 107. When a product to be inspected is selected from a plurality of pre-registered products in the product selection area 1001 using a drop-down menu, the appearance inspection item corresponding to the selected product to be inspected is set in the item setting area 1002. The user can also change the appearance inspection item by inputting information into the radio buttons in the item setting area 1002.

[0024] 10 has an examination start button 1004 for instructing the start of an examination, and an examination stop button 1005 for instructing the end of an examination. The examination screen UI also has an examination date area 1006 for displaying the examination date, and an examination time area 1007 for displaying the examination time.

[0025] The output unit 220 also displays the results of the inspection process on the display 105. For example, in the inspection screen UI shown in FIG. 10, the judgment result area 1008 displays "OK" if the inspection is passed, and "NG" if the inspection is failed. The abnormal event display area 1010 also displays that an abnormal event such as an imaging failure has occurred. The output unit 220 also outputs to the transport unit 206 instructions to transport the products that have passed the inspection and the products that have failed the inspection to a subsequent process.

[0026] <Processing by the imaging control unit 202 and the imaging unit 203 (S1101)> Details of the processing performed by the imaging control unit 202 and the imaging unit 203 in S1101 will be described. FIG. 3 is a flowchart of the processing executed by the imaging control device 102 in this embodiment. In S301, the imaging control unit 202 waits until a signal is input from the transport control device 111 or the imaging device 103, and accepts the input signal. In S302, the imaging control unit 202 determines the type of the input signal. If the input signal is an inspection start signal, the process proceeds to S303, and if the input signal is a synchronization signal, the process proceeds to S305. In S303, the synchronization signal count unit 209 sets the value of the image number to 0. In S304, the release signal output unit 207 sends a release signal indicating release ON to the release signal input unit 211.

[0027] In S305, the lighting signal output unit 210 outputs a lighting signal to the illumination unit 205 to light up the light source corresponding to the image number. In S306, the synchronization signal count unit 209 increments the image number by one. In S307, the synchronization signal count unit 209 determines whether the image number is less than a predetermined number of images, and if it is less than the predetermined number of pixels, the process returns to S301. Here, the predetermined number of images is the number of images used in the inspection process. If the image number is equal to or greater than the predetermined number of images, in S308, the release signal output unit 207 sends a signal indicating release OFF to the release signal input unit 211.

[0028] 4 is a flowchart of processing executed by the imaging device 103 in this embodiment. The processing executed by the imaging device 103 is made up of two flows, flow 400 and flow 410, which proceed in parallel. In S401, the release signal input unit 211 waits until a signal is input from the release signal output unit 207 of the imaging control unit 202, and accepts the input signal. In S402, the release signal input unit 211 sets the release flag to release ON or release OFF indicated by the input signal. In S403, the release signal input unit 211 notifies the control unit 212 that the release flag has been changed, and the process returns to S401 to wait for the next signal to be input.

[0029] In S411, the control unit 212 waits until a release flag change notification is input from the release signal input unit 211, and accepts the input notification. In S412, the control unit 212 determines whether the release flag indicates release ON or release OFF. If the release flag indicates ON, in S413 the synchronization signal output unit 213 outputs a synchronization signal. This turns on one of the light sources corresponding to the image number output by the lighting signal output unit 210 in S305 of FIG. 3. In S414, the control unit 212 captures an image of the object 113 with the light source turned on. In S415, the image transmission unit 214 transmits the captured image to the image processing unit 204, and the process returns to S412. This image capture is repeated until the release flag indicates OFF in S412. If the release flag indicates OFF in S412, the control unit 212 returns to S411 and waits for the next release flag change notification. Here, inter-thread synchronization provided by the OS (Operating System) is used as a mechanism for notifying the control unit 212 of a change in the release flag from the release signal input unit 211. For example, a condition variable can be used.

[0030] As described above, the processing by the imaging control unit 202 and the processing by the imaging unit 203 cooperate with each other to sequentially transmit a predetermined number of captured images to be used in the inspection processing to the image processing unit 204 .

[0031] <Processing by the image processing unit 204 (S1102)> The processing performed by the image processing unit 204 in S1102 will be described in detail below. Fig. 5 is a flowchart of the processing performed by the image processing device 104 in this embodiment. The processing performed by the image processing device 104 is made up of three flows, flow 500, flow 510, and flow 520, which proceed in parallel.

[0032] Flow 500 shows the flow of image acquisition processing executed by the inspection image acquisition unit 215. In S501, the inspection image acquisition unit 215 initializes the value of the acquired counter to −1. In S502, the inspection image acquisition unit 215 initializes the value of the image counter to 0. In S503, the inspection image acquisition unit 215 waits until a captured image is sent from the image transmission unit 214 and receives the sent captured image. In S504, the inspection image acquisition unit 215 determines whether the value of the image counter is 0, i.e., whether the sent captured image is the first captured image of the inspection object. If the value of the image counter is not 0, the process proceeds to S506. If the value of the image counter is 0, in S505, the inspection image acquisition unit 215 adds 1 to the value of the acquired counter and sends a change notification of the acquired counter to the image conversion unit 216. In S506, the inspection image acquisition unit 215 stores the acquired captured image in a conversion queue. In S507, the inspection image acquisition unit 215 adds 1 to the value of the image counter. In S508, the inspection image acquisition unit 215 determines whether the image counter has reached a predetermined number required for the inspection process. If the image counter has reached the predetermined number, the process returns to S502; if not, the process returns to S503.

[0033] Flow 510 shows the flow of image conversion processing executed by the image conversion unit 216. In S511, the image conversion unit 216 initializes the value of the converted counter to −1. In S512, the image conversion unit 216 receives the change notification of the acquired counter sent in S505, and if the value of the acquired counter is 1 or greater, it subtracts 1 and proceeds to S513, and if it is 0 or less, it waits for the next change notification. In S513, the image conversion unit 216 adds 1 to the value of the converted counter and sends a change notification of the converted counter to the image synthesis unit 217. In S514, the image conversion unit 216 performs image conversion processing, the details of which will be described later, and returns to S512.

[0034] Flow 520 shows the flow of the inspection process executed by the image composition unit 217, color and shape inspection unit 218, gloss inspection unit 219, and output unit 220. In S521, the image composition unit 217 receives the change notification of the converted counter sent in S513. If the value of the converted counter is 1 or greater, the image composition unit 217 subtracts 1 and proceeds to S522. If the value is 0 or less, the image composition unit 217 waits for the next change notification. In S522, the image composition unit 217, color and shape inspection unit 218, gloss inspection unit 219, and output unit 220 perform an inspection process, the details of which will be described later. In S523, the output unit 220 displays the results of the inspection process on the display 105. The output unit 220 also outputs a transport instruction based on the inspection result to the transport unit 206. For example, if the inspection result is a failure, the corresponding inspection object is transported to a transport path for failure. Here, the mechanism for notifying the image conversion unit 216 of the change in the acquired counter from the inspection image acquisition unit 215 and the image conversion unit 216 of the change in the converted counter from the image synthesis unit 217 uses inter-thread synchronization provided by the OS. For example, a condition variable can be used.

[0035] In this embodiment, if there are multiple inspection targets, the notification of the change in the converted counter in S513 is not sent for the second-to-last inspection target, and the processing of S522 is not performed. Furthermore, the notification of the change in the acquired counter in S505 is not sent for the last inspection target, and the processing of S513, S514, and S522 is not performed. To avoid this, a timer function that invokes processing after a certain period of time is used, and S505 and S513 are started again after the takt time period has elapsed. At this time, the previously set timer is canceled and a new timer is set. Here, the timer function uses a function provided by the OS.

[0036] <Processing by the image conversion unit 216 (S514)> The processing performed by the image conversion unit 216 in S514 will now be described in detail. FIG. 6A is a flowchart of the processing executed by the image processing device 104 in this embodiment. In S601 to S603, the image conversion unit 216 extracts a predetermined number of acquired images from the conversion queue and asynchronously invokes the conversion processing of S602 for each image. The conversion processing for each image is performed in parallel. A specific example of the conversion processing for each image will be described below. In this embodiment, it is assumed that the captured image is compressed in H.265 format and stored in HEIF format. The image conversion unit 216 decodes and expands the HEIF image using a known method. Note that the format of the captured image is not limited to HEIF format. For example, JPEG format or other compressed formats may also be used. Next, the image conversion unit 216 performs degamma processing on the decoded image data (RGB data) to convert it into RGB data that is linear with respect to luminance values. Note that color space conversion may also be performed to convert the converted RGB data into predetermined RGB data, such as sRGB data, using a conversion table based on the characteristics of the image capture device 103.

[0037] In S604 to S607, the image conversion unit 216 performs the processes of S605 and S606 on the converted image in the order in which the conversion processes were asynchronously called in S602. In S605, the image conversion unit 216 waits for the conversion process to be completed and acquires the RGB data that is the conversion result. In S606, the image conversion unit 216 stores the conversion result in an inspection queue. When the process of S606 has been completed for a predetermined number of images, the image conversion process of S514 ends.

[0038] <Processing by the image synthesis unit 217 to the output unit 220 (S522)> Details of the processing performed by the image synthesis unit 217, color and shape inspection unit 218, gloss inspection unit 219, and output unit 220 in S522 will be described below. FIG. 6B is a flowchart of the processing executed by the image processing device 104 in this embodiment. In S611, the image synthesis unit 217 extracts a predetermined number of converted images from the inspection queue. Here, the gloss images obtained by capturing images with the light source 1099 turned on were captured in the order specified in S302 in the imaging control processing of FIG. 3, and therefore are stored in positions corresponding to that order in the conversion queue and inspection queue. Note that the method for identifying gloss images from among the converted images is not limited to the method using the order in the inspection queue, and may be, for example, a method of attaching an identification flag when storing in the queue.

[0039] In S612, the image synthesis unit 217 synthesizes images obtained by capturing images with the light sources 1091 to 1098 turned on, using a known photometric stereo method. Specifically, two images are synthesized: a shape image representing normal information, i.e., unevenness, of the surface of the object 113, and a color image representing color information according to reflectance. In S613, the gloss inspection unit 219 performs an inspection process using the gloss image corresponding to the light source 1099. In S614, the color and shape inspection unit 218 performs an inspection process using the shape image obtained by synthesis. In S615, the color and shape inspection unit 218 performs an inspection process using the color image obtained by synthesis. The inspection processes in S613, S614, and S615 are performed by performing spatial filtering on the image and integrating the response values ​​to calculate the resulting value as the degree of abnormality. The processes in S613, S614, and S615 are performed in parallel. Furthermore, if multiple types of spatial filters are applied to the image to calculate the respective degrees of abnormality, the spatial filtering processes are also performed in parallel.

[0040] In S616, the output unit 220 integrates the calculated abnormality degrees and compares them with a predetermined threshold to determine whether the inspection has passed or failed. In the integration of the abnormality degrees in this embodiment, the maximum value of all the calculated abnormality degrees is used as the integrated value, but this is not limited to this and the minimum value, average value, median value, or the like may also be obtained using known statistical methods.

[0041] FIG. 7 is a timing chart of the processes executed by the image processing device 104 of this embodiment. Image acquisition 700 indicates the processing timing of flow 500, image conversion 701 indicates the processing timing of flow 510, and inspection 702 indicates the processing timing of flow 520. To distinguish between different inspection targets, processes performed on the same inspection target are depicted with the same background. Detecting minute defects requires the use of high-resolution, high-gradation images. To complete image acquisition 701 within the takt time, a compression method with a high compression ratio must be used to reduce data size. However, a compression method with a high compression ratio requires a long processing time for decompression. According to this embodiment, processing multiple images in parallel allows image conversion 701 to be completed within the takt time. Furthermore, to handle various defect types, it is necessary to increase the number of filters used in the inspection processes in S613, S614, and S615. According to this embodiment, performing these mutually independent filter processes in parallel allows inspection 702 to be completed within the takt time. Therefore, since each of the processes of image acquisition 700, image conversion 701, and inspection 702 is completed within the takt time, image acquisition 700 can be started at intervals of the takt time. In other words, images of one inspection object can be captured at intervals of the takt time, and image acquisition, image conversion, and inspection can be repeated. Furthermore, by synchronizing the start timing of each process, the processing volume within each takt time becomes uniform, allowing for stable operation.

[0042] [Second embodiment] In the first embodiment, the start timing of image acquisition, image conversion, and inspection was synchronized with the start of the takt time as shown in Figure 7, but in this embodiment, the start timing of image conversion and inspection is made different from the start of the takt time. Note that the configuration of the visual inspection system in this embodiment is the same as that of the first embodiment, so a description thereof will be omitted. Below, differences between this embodiment and the first embodiment will be mainly described. Note that the same components as those in the first embodiment will be described using the same reference numerals.

[0043] <Processing by the image processing unit 204 (S1102)> The processing performed by the image processing unit 204 in S1102 will be described in detail below. Fig. 8 is a flowchart of the processing performed by the image processing device 104 in this embodiment. The processing performed by the image processing device 104 is made up of three flows, flow 800, flow 810, and flow 820, which proceed in parallel.

[0044] Flow 800 shows the flow of image acquisition processing executed by the inspection image acquisition unit 215. The processing from S502 to S508 is the same as in the first embodiment. In S801, the inspection image acquisition unit 215 notifies the image conversion unit 216 that the image has been acquired, and the process returns to S502. Flow 810 shows the flow of image conversion processing executed by the image conversion unit 216. The processing of S514 is the same as in the first embodiment. In S811, the image conversion unit 216 receives a notification that the image has been acquired from the inspection image acquisition unit 215. In S812, the image conversion unit 216 notifies the image synthesis unit 217 and the gloss inspection unit 219 that the image has been converted, and the process returns to S811. Flow 820 shows the flow of inspection processing executed by the image synthesis unit 217, color and shape inspection unit 218, gloss inspection unit 219, and output unit 220. The processing of S522 and S523 is the same as in the first embodiment. In S821, the image synthesis unit 217 and the gloss inspection unit 219 receive a notification that the image has been converted from the image conversion unit 216. The image acquisition notification and the image conversion notification are transmitted using inter-thread synchronization provided by the OS. For example, a condition variable can be used.

[0045] FIG. 9 is a timing chart of the processes executed by the image processing device 104 of this embodiment. Image acquisition 900 indicates the processing timing of flow 800, image conversion 901 indicates the processing timing of flow 810, and inspection 902 indicates the processing timing of flow 820. To distinguish between different inspection targets, processes performed on the same inspection target are shown with the same background. In this embodiment, only the start of image acquisition is synchronized with the start of the takt time, and subsequent image conversion and inspection each start when the previous process is completed. This eliminates the need to use the timer function used in the first embodiment for the second-to-last and last inspection targets, thereby simplifying control.

[0046] [Third embodiment] In the above-described embodiment, image conversion is performed after a predetermined number of images are acquired, but in this embodiment, the image acquisition and image conversion flows are integrated, and image conversion is performed sequentially each time an image is acquired. Note that the configuration of the appearance inspection system in this embodiment is the same as that in the first embodiment, so a description thereof will be omitted. Below, differences between this embodiment and the first embodiment will be mainly described. Note that the same components as in the first embodiment will be described using the same reference numerals.

[0047] <Processing by the image processing unit 204 (S1102)> The processing performed by the image processing unit 204 in S1102 will be described in detail. FIG. 12 is a flowchart of processing executed by the image processing device 104 in this embodiment, which combines flows 500 and 510 in the first embodiment. In S1201, the inspection image acquisition unit 215 initializes the value of an image counter to 0. In S1202, the inspection image acquisition unit 215 waits until a captured image is sent from the image transmission unit 214, and receives the sent captured image. In S1203, the image conversion unit 216 asynchronously invokes conversion processing for the captured image acquired in S1202. Specifically, the processing of S601 to S603 in the first embodiment is performed. In S1204, the inspection image acquisition unit 215 adds 1 to the value of the image counter. In S1205, the inspection image acquisition unit 215 determines whether the image counter has reached a predetermined number required for inspection processing. If the image counter has reached a predetermined number, the process proceeds to S1206; if not, the process returns to S1202. In S1206, the image conversion unit 216 stores the conversion result in the inspection queue. Specifically, the process performs the processes from S604 to S607 in the first embodiment. In S1207, the image conversion unit 216 notifies the image synthesis unit 217 and the gloss inspection unit 219 that the image conversion has been completed, and the process returns to S1201. Note that the flow of the inspection process is the same as flow 820 in the second embodiment.

[0048] According to this embodiment, image conversion processing can be performed while waiting for the image to be transferred from the image transmission unit 214, and the time required from image acquisition to image conversion can be further shortened.

[0049] [Fourth embodiment] In the first embodiment, the image conversion unit 216 retrieved a predetermined number of images from the conversion queue in steps S601 to S603 of FIG. 6 and performed conversion processing on each image in parallel to shorten processing time. When decoding requires large computational resources, such as in HEVC (H.265) used in HEIF, processing multiple images in parallel can monopolize computational resources and affect other processes. In this embodiment, the image conversion processing is divided into three parts: data structure analysis, decoding, and color conversion, and each part is processed for each image. FIG. 13 is a flowchart of the image conversion processing in this embodiment, and flow 1300 replaces steps S601 to S603 of FIG. 6.

[0050] In S1301, the image conversion unit 216 asynchronously invokes the decoding process for each image. The decoding process is performed in parallel with the processes from S1302 onwards in a separate thread. Details of the decoding process will be described later in flow 1320. In S1302 to S1305, the image conversion unit 216 performs data structure analysis processing for a predetermined number of images. In S1303, the image conversion unit 216 analyzes the data structure of the HEIF format, extracting the metadata required for decoding and the data to be decoded. Here, the progress status of the data structure analysis, decoding, and color conversion processing is managed as a job list with the status of each image as an element. In S1303, images for which data analysis has been completed are added to the job list as having an analysis completed status. In S1304, the decoding process thread is notified that analysis of one image has been completed.

[0051] In steps S1306 to S1310, the image conversion unit 216 performs color space conversion processing on the image decoded by the decoding processing thread. S1307 is a step of waiting for the completion of processing by the decoding processing thread, and when completion of the decoding processing is notified in step S1324 (described later), the decoded image data is extracted from the job list. S1308 is a process of converting the extracted image data into the color space used for the inspection processing in step S522. Specifically, the decoded data in YUV space is converted into RGB space. Degamma processing may also be performed. S1309 is a step of storing the converted image data in the inspection queue.

[0052] Flow 1320 shows the flow of the decoding process. From S1321 to S1325, the image conversion unit 216 processes image decoding in a thread separate from flow 1300 in parallel. S1322 is a step of waiting for the analysis completion notification of S1304. When the completion of the data structure analysis process is notified in S1304, the decoding target data and metadata for the analyzed image are extracted from the job list. S1323 is a step of decoding the HEVC data based on the extracted data to generate image data in YUV space. Here, if an image is stored in HEIF format divided into multiple tile images, the process also combines these into a single image. S1324 marks the decoded image as being in a decoding process completion state, and notifies the thread of flow 1300 waiting in S1307 of the decoding completion.

[0053] FIG. 14 shows the timing of the data analysis, decoding, and color space conversion processes described in flows 1300 and 1320. Each rectangle represents the data structure analysis, decoding, and color space conversion process for each image, and rectangles with the same pattern represent processing for the same image. For convenience, the number of images used in the inspection process in S522 is five, but this is not limited to this number as long as there are multiple images. Threads 1400 and 1401 are threads that execute the processes of flows 1300 and 1320, respectively, and show that both threads operate in parallel. Analysis 1402 is the part that repeats data structure analysis for each image in a loop from S1302 to S1305. Decoding 1403 is the part that repeats decoding for each image in a loop from S1321 to S1325, and shows that it starts after the first image is completed in analysis 1402. Conversion 1404 is the part that performs color space conversion processing in a loop from S1306 to S1310. Conversion 1404 starts after data analysis for all images is completed in analysis 1402, and waits until decoding processing in decoding 1403 is completed for images that have not yet been decoded. As shown in FIG. 14, according to this embodiment, image conversion processing for multiple images can be performed efficiently with a limited number of threads. If data analysis or color space conversion processing takes time, each may be processed in parallel using separate threads. If decoding processing takes time, the number of processing threads for decoding processing may be increased to two or more.

[0054] The format of the captured image is not limited to the HEIF format. For example, it may be JPEG or another compressed format. Furthermore, the division of the image conversion process is not limited to division into data analysis, decoding, and color space conversion, but may be other divisions or division numbers.

[0055] [Other embodiments] In the above-described embodiment, the timing at which the output unit 220 outputs a transport instruction based on the inspection results to the transport unit 206 is delayed by several cycles from the image capture, as shown in the timing charts of FIGS. 7 and 9 . Therefore, the inspection time required from the time the transport device 112 transports the inspection object to a predetermined position and captures an image until the transport path is controlled based on the inspection results is set in advance. Specifically, the inspection time required is set to the time it takes for the transport device 112 to move N inspection objects and to be longer than the delay time shown in the timing charts. The inspection start signal from the start signal output unit 201 is also received by the image processing unit 204, and the time of reception is recorded as the inspection start time. Furthermore, in S523, the output unit 220 measures the elapsed time from the recorded reception time, delays the time by the difference between the elapsed time and the set inspection time, and then outputs the transport instruction to the transport unit 206. Instead of receiving the inspection start signal at the image processing unit 204, the time at which the image processing unit 204 acquires the first image of the inspection object may be used as the inspection start time. As described above, in this embodiment, the inspection time is fixed, so the transport device 112 can set up a transport path for rejected products at a position N inspection targets downstream from the predetermined inspection position.

[0056] In the above-described embodiment, the imaging unit 203 captures images of the inspection target objects one by one. However, multiple inspection target objects may be placed within the imaging range of the imaging device 103, allowing multiple objects to be simultaneously captured. In this case, a step of segmenting the converted image into regions is further provided before the image synthesis in S612, and the processes in S612 to S616 are performed for each image corresponding to the segmented regions. The segmentation of the converted image into regions is performed based on the placement information of the inspection target identified by the conveying device 112. Note that a known object detection method may be applied to one of the inspection images, for example, a glossy image captured with the light source 1099 turned on, to detect the inspection target regions, and the detected region information may be used for region segmentation. According to this embodiment, capturing multiple inspection targets in a single image allows the process from image acquisition to image conversion to be performed more efficiently than repeating the process for each inspection target. In other words, even when using high-resolution and high-gradation images, the processing time for image acquisition and image conversion per inspection target can be kept within the takt time, thereby improving inspection accuracy.

[0057] Furthermore, since the processes from S612 to S616 are independent for each divided area, by performing each process in parallel, the processing time can be further reduced, the number of supported defect types can be expanded, and inspection accuracy can be improved.

[0058] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0059] 104 Image Processing Equipment 204 Image Processing Unit 215 Inspection image acquisition unit 216 Image conversion unit 218 Color and Shape Inspection Department 219 Gloss Inspection Department

Claims

1. an acquisition means for acquiring a plurality of images obtained by capturing images of a plurality of objects under a plurality of imaging conditions; a control means for controlling a conversion process for each of the plurality of images and an inspection process based on the converted images; The control means The conversion process and the inspection process are performed in parallel by pipeline processing; performing conversion processing on each of the plurality of images in parallel; 1. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the control means includes an inspection means for performing the inspection process of inspecting at least one of the shape and color of the surface of one of the plurality of objects based on an image obtained by combining the plurality of images corresponding to the one of the plurality of objects.

3. 3. The image processing apparatus according to claim 2, wherein said inspection means synthesizes said plurality of images by a photometric stereo method.

4. 2. The image processing device according to claim 1, wherein the control means includes an inspection means for performing the inspection process of inspecting the gloss of the surface of one of the plurality of objects based on one of the plurality of images corresponding to the one of the plurality of objects.

5. 5. The image processing apparatus according to claim 4, wherein the one image is an image obtained by capturing an image under imaging conditions in which light specularly reflected from the one object is received.

6. 2. The image processing apparatus according to claim 1, wherein in addition to the conversion process and the inspection process, the acquisition of the plurality of images by the acquisition means is also performed in parallel by pipeline processing.

7. 2. The image processing device according to claim 1, wherein the conversion processing is one of decoding a compressed image, degamma processing, and color space conversion, or a combination thereof.

8. The image processing device according to claim 1 , wherein the plurality of imaging conditions differ from each other in the direction of light irradiated onto each of the plurality of objects.

9. 2. The image processing apparatus according to claim 1, further comprising an output unit for outputting the results of the inspection processing.

10. 10. The image processing apparatus according to claim 9, wherein the output means controls the transportation of the object for which the inspection process has been completed.

11. 11. The image processing device according to claim 10, wherein the output means calculates the difference between a predetermined required time and the time required for processing the inspection of one of the plurality of objects, and controls the transportation of the object based on the difference.

12. the control means further comprises means for dividing the plurality of images into regions corresponding to the plurality of objects after performing the conversion process on each of the plurality of images; 2. The image processing apparatus according to claim 1, wherein the control means performs the inspection process for each divided area in parallel.

13. The image processing device according to claim 1, characterized in that the control means divides the conversion processing for one image into multiple steps, executes the conversion processing for the multiple images collectively for all images for each of the divided steps, and performs some or all of the processing for each step in parallel.

14. 14. The image processing apparatus according to claim 13, wherein the divided steps are image data structure analysis, decoding processing, and color space conversion processing.

15. A program for causing a computer to function as the image processing device according to any one of claims 1 to 14.

16. an acquisition step of acquiring a plurality of images obtained by imaging each of a plurality of objects under a plurality of imaging conditions; a control step of controlling a conversion process for each of the plurality of images and an inspection process based on the converted images, In the control step, The conversion process and the inspection process are performed in parallel by pipeline processing; performing conversion processing on each of the plurality of images in parallel; An image processing method comprising:

Images