Image recognition device and article manufacturing device equipped with the same

Abstract

The present invention provides an image recognition device and an article manufacturing device equipped with the image recognition device, which can speed up the recognition process for multiple camera images. The image recognition device (4) includes: a storage unit (42) that stores multiple camera images obtained by photographing articles as reserve images; a recognition unit (43) that performs a recognition process for recognizing the type of the reserve images; and an input unit (44) that performs an input process for inputting the multiple reserve images stored in the storage unit (42) to the recognition unit (43) at the same time, wherein the recognition unit (43) is configured to perform multiple recognition processes in parallel, and the recognition processes are performed on each of the multiple reserve images input at the same time by the input unit (44). The article manufacturing device (1) includes the image recognition device (4).

Description

Technical Field

[0001] The present invention relates to an image recognition device for recognizing the type of an image and an article manufacturing device including the image recognition device. Background Art

[0002] In recent years, image recognition devices for identifying the type of images have been used when inspecting the appearance of articles. As one example of such image recognition devices, for example, an image classification device for inspecting the appearance of a semiconductor substrate (hereinafter referred to as a substrate) as the article is known, as disclosed in Patent Document 1.

[0003] The image classification device includes: an imaging device that captures an inspection target area on a substrate to generate image data; and an inspection / classification device that classifies image data in which defects are captured among the image data generated by the imaging device according to the type of the defect.

[0004] The inspection / classification device includes: a defect detection unit that detects defects from image data; and a classification control unit that performs classification processing to classify the image data in which defects are detected by the defect detection unit according to the type of the defect. Moreover, the classification control unit is configured to classify (identify) the image data each time the defect detection unit detects a defect from the image data.

[0005] Therefore, the image classification device can confirm whether the substrate has defects, and can be adjusted to the following state: by classifying and summarizing the image data that captured the defects according to the types of defects, the defects can be analyzed efficiently.

[0006] Patent Document 1: Japanese Patent Application Publication No. 2016-115115 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] In addition, depending on the state of the substrate being inspected, a large amount of image data that captures defects may sometimes be generated. However, in the above-mentioned previous image classification device, the classification control unit is configured to classify the image data that captures defects in the order in which the image data is generated (the order in which the defects are detected). Therefore, while the classification processing of one image data is being performed, the classification processing of other image data is in a waiting state.

[0009] Therefore, in the above-mentioned conventional image classification device, the classification process of image data is largely delayed, and it takes time to classify all the image data that have captured defects, which is a problem. In addition, when the above-mentioned conventional image classification device is assembled into an article manufacturing device, the manufacturing process flow of the article is stagnant, which is also a problem.

[0010] Furthermore, this problem is not limited to devices for detecting defects from image data, but is a problem common to all devices having a function of recognizing images.

[0011] Therefore, in view of the above-mentioned actual situation, the present invention aims to provide an image recognition device capable of speeding up the recognition process for a plurality of captured images and an article manufacturing device including the image recognition device.

[0012] Solutions for solving problems

[0013] The image recognition device of the present invention comprises:

[0014] A storage unit that stores a plurality of camera images obtained by photographing an object as reserve images;

[0015] a recognition unit that performs recognition processing for recognizing the kind of the stock image; and

[0016] an input unit for performing an input process of inputting the plurality of reserve images stored in the storage unit to the recognition unit at once,

[0017] The recognition unit is configured to execute a plurality of recognition processes in parallel, wherein the recognition processes are performed on each of the plurality of reserve images input collectively by the input unit.

[0018] In addition, the image recognition device of the present invention may further include:

[0019] an image acquisition unit that continuously and repeatedly executes an acquisition process of acquiring a plurality of the camera images; and

[0020] a defect candidate extraction unit that performs extraction processing, in which a defect candidate image that captures a defect candidate point that is a candidate for a defect is extracted from the plurality of camera images acquired by the image acquisition unit, and the defect candidate image is saved as the reserve image in the storage unit,

[0021] The recognition unit is configured to recognize whether the defect candidate point of the defect candidate image stored in the storage unit as the reserve image is a defect or a normal part in the recognition process.

[0022] The image recognition device is configured to complete the execution of the subsequent acquisition processing executed after the previously executed acquisition processing before the extraction processing, the input processing, and the recognition processing performed on the camera image acquired through the previously executed acquisition processing are completed.

[0023] In the image recognition device of the present invention, it may be possible that:

[0024] The recognition processing file for executing the recognition processing is a processing file other than the processing file for executing other processing.

[0025] The article manufacturing device of the present invention comprises:

[0026] Object transport device, which transports objects;

[0027] A camera device that photographs the article being transported by the article transport device to produce a plurality of camera images;

[0028] Any of the above image recognition devices; and

[0029] a marking device, which is arranged on a conveying path of the article conveyed by the article conveying device and at a position downstream of the article conveying device in a conveying direction, and performs a marking process of adding a mark to an article captured in the stock image having the recognition result according to the recognition result of the stock image by the recognition unit of the image recognition device,

[0030] The image recognition device is configured to complete the recognition processing on the camera image after the camera device photographs the object to produce the camera image and before the marking device performs the marking processing on the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of an article manufacturing device according to one embodiment of the present invention.

[0032] Figure 2 It is an explanatory diagram of the imaging device involved in this embodiment.

[0033] Figure 3 This is a block diagram of an image recognition device according to this embodiment.

[0034] Figure 4 It is an operation explanation diagram of the article manufacturing device involved in this embodiment.

[0035] Figure 5 It is a diagram for explaining the operation of the image recognition device according to this embodiment.

[0036] Description of Reference Numerals

[0037] 1: article manufacturing device; 2: article conveying device; 3: camera device; 4: image recognition device; 5: marking device; 30: camera unit; 40: image acquisition unit; 41: defect candidate extraction unit; 42: storage unit; 43: recognition unit; 44: input unit; 45: buffer unit; 430: receiving unit; 431: recognition execution unit. DETAILED DESCRIPTION

[0038] Hereinafter, an image recognition device according to an embodiment of the present invention will be described with reference to the drawings.

[0039] The image recognition device according to the present embodiment is configured to recognize the type of a captured image of an article, etc., and is used, for example, to inspect manufactured articles. In addition, in the present embodiment, an article manufacturing device in which the image recognition device is incorporated is described.

[0040] The article manufacturing apparatus according to the present embodiment is an apparatus for manufacturing a long sheet-like article (hereinafter referred to as a sheet) such as a polarizing film.

[0041] like Figure 1 As shown, the article manufacturing device 1 includes: an article conveying device 2, which conveys the article; and a camera device 3, which photographs the article being conveyed by the article conveying device 2 to produce a plurality of camera images (in Figure 2 an image marked with a mark "P1" in the image); an image recognition device 4, which recognizes the type of the camera image produced by the camera device 3; and a marking device 5, which is arranged on a conveying path where the article is conveyed by the article conveying device 2 and is located downstream of the article conveying device 2 in the conveying direction. The marking device 5 performs a marking process of adding a mark to the article captured in the reserve image having the recognition result according to the recognition result of the recognition unit of the image recognition device 4 on the reserve image.

[0042] The article transport device 2 is configured to perform roller transport for transporting a sheet while feeding the sheet from a sheet roll.

[0043] The article transport device 2 is configured to continue transporting the sheet even while the imaging device 3 captures the article and creates the captured image, the image recognition device 4 recognizes the captured image, and the marking device 5 performs marking processing.

[0044] Therefore, the article manufacturing device 1 involved in this embodiment is constructed as follows: a camera device 3 is used to photograph a sheet in a conveying state (a state of being conveyed from the upstream side to the downstream side in the conveying direction) to produce a camera image, and a marking device 5 is used to mark the sheet in the conveying state.

[0045] Then, for the captured image created by the imaging device 3 , recognition by the image recognition device 4 is completed after the captured image is created and before the marking process is performed on the portion of the sheet captured in the captured image.

[0046] like Figure 2 As shown, the imaging device 3 includes an imaging unit 30 that continuously performs an operation of imaging an object (sheet) to create an imaged image (hereinafter referred to as an imaging operation).

[0047] In the imaging device 3, a plurality of imaging units 30 are arranged in a row in the width direction of the sheet (a direction perpendicular to the sheet conveying direction). The plurality of imaging units 30 are configured to continuously perform imaging operations at the same or substantially the same timing.

[0048] Therefore, the imaging device 3 is configured to continuously output a plurality of captured images created by imaging the sheet at the same or substantially the same timing.

[0049] like Figure 3 As shown, the image recognition device 4 comprises: an image acquisition unit 40, which continuously and repeatedly executes the acquisition process of acquiring a camera image; a defect candidate extraction unit 41, which executes the extraction of a defect candidate image (in which a defect candidate is captured) from a plurality of camera images acquired by the image acquisition unit 40. Figure 2 The storage unit 42 performs a storage process of storing the multiple defect candidate images extracted by the defect candidate extraction unit 41 as reserve images; the recognition unit 43 performs a recognition process of recognizing the type of the reserve image; the input unit 44 performs an input process of inputting the multiple reserve images stored in the storage unit 42 to the recognition unit 43; and the buffer unit 45 performs a storage process of temporarily storing command information, which is used to perform each of the extraction process, the input process, and the recognition process. In addition, the recognition device can be any device as long as it has a CPU, a memory, a hard disk, etc.

[0050] The image acquisition unit 40 is configured to acquire a plurality of captured images output from the imaging device 3 (a plurality of captured images produced by the imaging unit 30 at the same or substantially the same timing). In addition, the image acquisition unit 40 may be configured to directly acquire captured images from the imaging device 3 or may be configured to indirectly acquire captured images from the imaging device 3 (that is, may be configured to acquire captured images output from the imaging device 3 via another device other than the imaging device 3).

[0051] The defect candidate extraction unit 41 is configured to create (extract) a defect candidate image in which defect candidate points that are likely to be defects are captured, from the captured image acquired by the image acquisition unit 40 .

[0052] Specifically, the defect candidate extraction unit 41 is configured to perform detection processing for detecting defect candidate points from a camera image and candidate image production processing for producing a defect candidate image, and save the produced defect candidate image as a reserve image to the storage unit 42. In the candidate image production processing, the defect candidate image is produced by extracting an area including pixels of the defect candidate points detected in the detection processing and a specified number of pixels around the pixel.

[0053] In addition, in order to detect defect candidate points from the captured image in the detection process, for example, binarization processing and particle analysis may be performed.

[0054] A plurality of defect candidate images extracted from the camera images captured by the defect candidate extraction unit 41 at the same time point are saved (stored) together as reserve images in the storage unit 42. In addition, in the case where only one defect candidate image is extracted from the defect candidate extraction unit 41, this only one defect candidate image can also be stored in the storage unit 42.

[0055] In the input processing of the input unit 44, the defect candidate images stored in the storage unit 42 as reserve images, more specifically, a plurality of defect candidate images produced by the camera unit 30 at the same or substantially the same timing, are input to the recognition unit 43 at once.

[0056] The recognition unit 43 has: a receiving unit 430, which receives multiple defect candidate images input by the input unit 44; and a recognition execution unit 431, which executes recognition processing in parallel, and the recognition processing is performed on each defect candidate image of the multiple defect candidate images received by the receiving unit 430.

[0057] The recognition execution unit 431 is configured to recognize the type of the defect candidate image by performing recognition processing on the defect candidate image. The recognition execution unit 431 involved in this embodiment is configured to recognize the correctness of the defect candidate point captured in the defect candidate image, that is, whether the defect candidate point is a defect or a normal part. In this way, the recognition execution unit 431 is configured to recognize whether the defect candidate image is an image in which a defect is captured or an image of a normal part without a defect.

[0058] The recognition execution unit 431 is a structure equivalent to a so-called recognizer. In addition, the recognition execution unit 431 involved in the present embodiment is composed of a recognition processing file, which is obtained by converting a recognizer produced (learned) in a device other than the image recognition device 4 into a processing file (a dynamic link library in the present embodiment). In addition, it is preferred that the recognition processing file used to execute the recognition execution unit 431 is other data other than the processing file used to execute other processing (for example, processing performed in the image acquisition unit 40, the defect candidate extraction unit 41, the storage unit 42, the input unit 44, etc.). In this way, the recognition processing file is independent of the processing file used to execute other processing, so that the processing content of the recognition unit 43 can be changed by simply rewriting the existing recognition processing file into a new recognition processing file.

[0059] In addition, the recognition unit 43 may be configured to execute the recognition processing of each of the plurality of defect candidate images received by the receiving unit 430 in parallel, or may be configured to execute the recognition processing of each of the plurality of defect candidate images in parallel by executing a predetermined number of images at a time.

[0060] In addition, the recognition unit 43 can be configured to execute multiple recognition processes in parallel within the recognition execution unit 431, where the recognition processes are performed on each of multiple defect candidate images, or it can be configured to execute multiple tasks in parallel, where the task is to perform recognition processing on one defect candidate image in the recognition execution unit 431.

[0061] In this way, the recognition unit 43 is configured to perform recognition of a plurality of (two or more) defect candidate images in parallel.

[0062] The command information written to the buffer unit 45 includes a plurality of captured images produced by the imaging device 3 at the same or substantially the same timing, and commands for executing extraction processing, input processing, and recognition processing related to the plurality of captured images.

[0063] The image recognition device 4 of this embodiment includes two buffer units 45 , and while executing extraction processing, input processing, and recognition processing based on command information written in one buffer unit 45 , other command information other than the command information written in one buffer unit 45 is written to the other buffer unit 45 .

[0064] Moreover, at the time point when the extraction processing, input processing, and recognition processing based on the command information written in one buffer unit 45 are completed, the extraction processing, input processing, and recognition processing are sequentially executed based on the command information written in another buffer unit 45.

[0065] Furthermore, when the extraction process, the input process, and the recognition process based on the command information written in the other buffer unit 45 are completed, other command information other than the above-mentioned command information is newly written into the other buffer unit 45 .

[0066] In this way, in each buffer unit 45 , reading of written command information (that is, execution of each process based on the command information) and writing of the command information are performed alternately at different timings.

[0067] In addition, every time the imaging device 3 performs an imaging operation, command information including a captured image created by the imaging operation and a command for processing related to the captured image is alternately written to one buffer unit 45 and the other buffer unit 45 .

[0068] As a recognition result of the defect candidate image, information indicating whether the defect candidate image is an image that has captured a defect (defective image) is output from the recognition unit 43 of the image recognition device 4 as the recognition result, so the marking device 5 is configured to add a mark indicating the presence of a defect to the defective items among the items being conveyed by the item conveying device 2 during the marking process.

[0069] The article involved in this embodiment is a long thin sheet, and the marking device 5 may be configured as follows: when the recognition unit 43 outputs information indicating that the defect candidate image is a defective image as a recognition result, the marking device 5 adds a mark to the part captured in the camera image that is the extraction source of the defect candidate image. In addition, when the recognition unit 43 outputs information indicating that the defect candidate image is a normal image as a recognition result, the article may be passed without adding a mark.

[0070] In addition, as a mark added to the article by the marking device 5, for example, labeling, printing, engraving, etc. can be used.

[0071] The structure of the article manufacturing apparatus 1 according to the present embodiment is as described above. Next, the operation of the article manufacturing apparatus 1 according to the present embodiment will be described.

[0072] In the article manufacturing device 1, if Figure 4 As shown, the article conveying device 2 continuously conveys sheet-like articles (S1), and the imaging device 3 continuously performs imaging operations (S3, S8, S12) on the conveyed articles.

[0073] In the article manufacturing device 1 involved in the present embodiment, as described above, while processing is being executed based on the command information written in the one buffer unit 45 (S2, S10), command information is written to the other buffer unit 45 (S5, S13), and while processing is being executed based on the command information written in the other buffer unit 45 (S6), command information is written to the one buffer unit 45 (S9).

[0074] Thus, the image recognition device 4 completes preparation for processing the camera image produced by the camera device 3 through the next camera action while processing the camera image produced by the camera device 3 through the camera action. Therefore, the image recognition device 4 can start the next new process immediately after the process being executed is completed, thereby also being able to start the next recognition process (S6, S10) during the execution of the step after the recognition process (in this embodiment, the marking process (S3, S7) performed by the marking device 5).

[0075] The processing of the image recognition device 4 is further described in detail. Figure 5 As shown, the imaging device 3 outputs a plurality of captured images created by the imaging operation ( S14 ) to the image recognition device 4 ( S15 ).

[0076] The plurality of captured images output to the image recognition device 4 are acquired by the image acquisition unit 40 ( S16 ), and written to the one buffer unit 45 in a manner included in the command information ( S17 ).

[0077] Then, various processes related to the camera image are started based on the command information written in the buffer unit 45. Specifically, first, the defect candidate extraction unit 41 extracts a defect candidate image of a candidate defect captured from a plurality of camera images (S18), and saves the defect candidate image as a reserve image in the storage unit 42 (S19).

[0078] The plurality of defect candidate images stored in the storage unit 42 as reserve images are input to the recognition unit 43 at once (S20). Next, the recognition unit 43 performs recognition processing in parallel for each of the plurality of defect candidate images (S21).

[0079] Then, when the recognition result is output from the recognition unit 43, the marking device 5 performs marking processing according to the recognition result (refer to Figure 4 ). Thus, a series of processes on the article in the article manufacturing device 1 are completed.

[0080] While processing is being performed based on the command information written in the one buffer unit 45, the camera device 3 performs the next camera action (S22), and the multiple camera images produced by the next camera action are output to the image recognition device 4 (S23) before the recognition processing (S21) performed on the camera image produced by the previous camera action (S14) is completed, and are acquired by the image acquisition unit 40 (S24), and then written to the other buffer unit 45 in a state included in the command information (S25).

[0081] Therefore, before the marking process for the objects captured in the video image produced by the previous video action is completed, various processes related to the video image produced by the next (subsequent) video action are performed in sequence. Specifically, the following processes are performed in sequence: the defect candidate image is extracted by the defect candidate extraction unit 41 (S26); the defect candidate image is saved to the saving unit 42 (S27); a plurality of defect candidate images are input to the recognition unit 43 at a time through input processing (S28); and recognition processing is performed in parallel by the recognition unit 43, and the recognition processing is performed on each of the plurality of defect candidate images (S29).

[0082] Furthermore, in the process of performing processing based on the command information written in the other buffer unit 45, new command information is written to the one buffer unit 45 (S30), so that processing based on the new command information is started immediately after the processing based on the command information written in the other buffer unit 45 is completed.

[0083] As described above, according to the image recognition device 4 involved in this embodiment, multiple recognition processes are performed in parallel in the recognition unit 43, and the recognition processes are performed on each of the multiple reserve images input by the input unit 44. Therefore, it is possible to suppress the recognition process for each reserve image from becoming a waiting state (i.e., waiting for the recognition process for other reserve images to be completed).

[0084] Therefore, in the image recognition device 4, the time for waiting to execute the recognition processing for each reserve image can be eliminated, thereby speeding up the recognition processing for a large number of camera images. In addition, the same effect can also be obtained in the article manufacturing device 1 equipped with the image recognition device 4.

[0085] In addition, in the image recognition device 4, it is configured to complete the execution of the (subsequent) acquisition processing and storage processing subsequent to the previously executed acquisition processing before the extraction processing, the input processing, and the recognition processing performed on the multiple camera images acquired by the previously executed acquisition processing are completed. Therefore, there is sufficient time from the time point when each of the extraction processing, the storage processing, the recognition processing, and the input processing performed on the multiple camera images acquired by the previously executed acquisition processing is completed to the start of the sequential execution of each of the extraction processing, the storage processing, the recognition processing, and the input processing performed on the multiple camera images acquired by the subsequently executed acquisition processing.

[0086] In addition, after a camera image is produced by the camera action of the camera device 3 and before the marking device 5 performs marking processing on the object photographed by the camera device 3 (the portion of the sheet captured in the camera image), the image recognition device 4 performs various processing (extraction processing, storage processing, recognition processing, input processing) on ​​the camera image, so that the image recognition device 4 can recognize the camera image in real time, that is, in the flow of the manufacturing process of the object (in this embodiment, to confirm whether there are defects).

[0087] In addition, in the article manufacturing device 1 involved in the present embodiment, the recognition processing of the camera image obtained by the camera device 3 is completed before the marking processing as a post-process is performed on the article photographed by the camera device 3. Therefore, the recognition of the camera image (article) can be completed without delaying the manufacturing process of the article, and the recognition of the camera image can also be performed and completed on the production line (in the flow of the manufacturing process of the article).

[0088] In addition, the image recognition device and the article manufacturing device of the present invention are not limited to the above-mentioned embodiments, and it is a matter of course that various changes can be added without departing from the scope of the present invention.

[0089] The article manufacturing apparatus 1 of the above-described embodiment is an apparatus for manufacturing a sheet, but is not limited to this configuration. For example, the article manufacturing apparatus 1 may also manufacture a single sheet-shaped article or the like.

[0090] In the article manufacturing apparatus 1 of the above embodiment, the plurality of imaging units 30 are arranged in a row in the width direction of the sheet. However, the number and arrangement of the imaging units 30 may be appropriately changed depending on the article to be manufactured (article to be inspected).

[0091] In the above embodiment, the image acquisition unit 40 can acquire multiple camera images (multiple camera images produced at the same time point) output from the camera device 3 as independent images, or acquire an image obtained by synthesizing the multiple camera images, but it is not specifically mentioned.

[0092] In the above-mentioned embodiment, the defect candidate extraction unit 41 is configured to use an area including a pixel of a defect candidate point detected by the detection process and a predetermined number of pixels around the pixel as a defect candidate image, but is not limited to this structure. For example, the defect candidate extraction unit 41 may also use the image itself of the defect candidate point as a defect candidate image.

[0093] In the above embodiment, the image recognition device 4 includes two buffer units 45 , but the present invention is not limited to this configuration. For example, the image recognition device 4 may include one buffer unit 45 , or may include three or more buffer units 45 .

Claims

1. An image recognition device, comprising: an image acquisition unit that continuously and repeatedly executes an acquisition process of acquiring a plurality of camera images captured by the camera device at the same or substantially the same timing; a defect candidate extraction unit, which performs an extraction process, in which a defect candidate image is extracted from the plurality of camera images acquired by the image acquisition unit, and the defect candidate image is saved as a reserve image in a saving unit, wherein the defect candidate image is an image in which a defect candidate point is captured; a recognition unit that performs recognition processing for recognizing whether the defect candidate point of the reserve image is a defect or a normal portion; an input unit that performs an input process of reading out the plurality of reserve images captured at the same or substantially the same timing and acquired by the image acquisition unit from the storage unit and inputting the plurality of reserve images together to the recognition unit; and a plurality of buffer units that perform a storage process of temporarily storing the reserve image in association with command information for performing the extraction process, the input process, and the recognition process, Among them, the image recognition device is configured to enable other buffer units among the multiple buffer units to execute and complete the storage processing before executing and completing the extraction processing, the input processing and the recognition processing based on the reserve image and the command information stored in any buffer unit among the multiple buffer units.

2. The image recognition device according to claim 1, characterized in that: Also available: The image recognition device is configured to cause the image acquisition unit to execute and complete the acquisition process for acquiring the new plurality of camera images written into other buffer units among the plurality of buffer units before the input process and the recognition process based on the reserve image and the command information stored in any buffer unit among the plurality of buffer units are completed.

3. The image recognition device according to claim 1 or 2, characterized in that: The recognition processing file for executing the recognition processing is a processing file other than the processing file for executing other processing.

4. An article manufacturing device, comprising: Object transport device, which transports objects; A camera device that photographs the article being transported by the article transport device to produce a plurality of camera images; The image recognition device according to claim 1 or 2; and a marking device, which is arranged on a conveying path of the article conveyed by the article conveying device and at a position downstream of the article conveying device in a conveying direction, and performs a marking process of adding a mark to an article captured in the stock image having the recognition result according to the recognition result of the stock image by the recognition unit of the image recognition device, in, The image recognition device is configured to complete the recognition process on the captured image after the camera device captures the image of the object to create the captured image and before the marking device performs the marking process on the object.

Images