Intelligent production line monitoring system and monitoring method

Abstract

An intelligent production line monitoring system and monitoring method can monitor an image of an object to be inspected taken by an imaging device, mainly enabling an operating station host to provide a marking module to re-judge the classification decision of a classifier subsystem, and can achieve the purpose of verifying the classification decision or checking whether there is any missed judgment based on human-machine collaboration; in addition, the classifier subsystem can automatically screen out classification decisions with lower credibility to effectively reduce the number of re-judgments; in addition, a group image of an object to be inspected composed of multiple images of the object to be inspected can be first analyzed to compare the difference image features therefrom, so as to be suitable for situations where certain defect training samples are insufficient; in addition, the marking module can simultaneously re-judge highly relevant historical classification decisions; and a second imaging device can also be included so that the system can automatically mark the defect position based on the images of the object to be inspected before and after repair, and then learn the value judgment of whether there is a defect.

Description

Technical Field

[0001] The present invention relates to machine learning, image recognition, image processing and defect detection technologies, and in particular to an intelligent production line monitoring system and monitoring method that can improve detection efficiency and accuracy, can re-judge the judgment results of classifiers, can adjust historical judgment data, can be applicable to objects to be inspected with a small number of training samples, and can avoid missing defects. Background Art

[0002] With the advancement of technology, the precision requirements of electronic parts are becoming increasingly higher. In order to ensure the product yield, the production process of electronic parts usually goes through two stages: optical inspection and re-judgment. The optical inspection stage mainly uses an automatic appearance inspection device to obtain the image of the object to be inspected and determine whether the object to be inspected has defects. The re-judgment stage is to further check and mark the defects determined in the optical inspection stage manually.

[0003] However, although the detection efficiency of the automatic appearance inspection device is better than that of the traditional manual inspection method and can reduce the large amount of manpower consumed in the production process, if the judgment results of the automatic appearance inspection device are misjudged or missed, defective products may flow to the customer end. In addition, in order to reduce the workload in the re-judgment stage, operators can usually only check and mark the images of the defective products judged by the automatic appearance inspection device, and cannot check and mark the complete or other partial images of the objects to be inspected.

[0004] Furthermore, at present, in the re-judgment stage, since the operators still need to check and mark each defective image judged by the automatic appearance inspection device, there is still a disadvantage of high manpower consumption, and there may still be a certain probability of missed detection due to reasons such as human visual fatigue. Moreover, if the number of defect training samples of some electronic parts is insufficient, the operators in the re-judgment stage may find it difficult for the automatic appearance inspection device to determine whether there are defects, and they may need to judge whether each image of the object to be inspected has defects and mark them by themselves. Therefore, in the case of very few defects, there is still a disadvantage of high manpower consumption in the re-judgment stage.

[0005] Therefore, since the existing defect monitoring system still has the above-mentioned shortcomings, how to propose a defect monitoring technology based on the human-machine collaboration mode that can improve the industrial process, improve the efficiency and accuracy of defect detection, be applicable to objects to be inspected with a small number of training samples, and avoid missing defect features is still a problem to be solved. Summary of the invention

[0006] In order to solve the above-mentioned deficiencies in the prior art, one of the purposes of the present invention is to provide an intelligent production line monitoring system, which can monitor an image of an object to be inspected taken by an imaging device. The system mainly includes a training subsystem, an operation station host and a classifier subsystem; the training subsystem stores a training module corresponding to the type of the object to be inspected; the operation station host is connected to the training subsystem and runs a marking module; the classifier subsystem is respectively connected to the imaging device, the training subsystem and the operation station host, and the classifier subsystem can read the training module to analyze the image features of the image of the object to be inspected, and make a classification decision to send to the operation station The host computer comprises a computer program product, wherein if the classification decision is deemed abnormal, the classification decision may include a complete image and an abnormal image with at least one re-judgment mark; the marking module may input a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host computer sends the updated classification decision to the training subsystem; the marking module is also used to mark a missed judgment mark associated with the complete image, so as to input a second re-judgment operation to generate a second classification decision that replaces the classification decision, and the operation station host computer sends the updated classification decision to the training subsystem; then, the training subsystem may update the training module and a marking database connected to the training module in real time based on the updated classification decision and the second classification decision. After the present invention is implemented according to this embodiment, since the operating station host can not only monitor the abnormal images with re-judgment marks, but also check and mark the complete image of the inspected object, compared with the existing practice that the operator can only check and mark the defective image emitted by the automatic appearance inspection device, the present invention can enable the operating station host to monitor the complete image and the abnormal image judged as abnormal by the classifier subsystem during the re-judgment stage, thereby avoiding the abnormal inspected objects missed by the classifier subsystem from being transported to the next stage of the production line, and at least achieving the beneficial effect of reducing the probability of defective products flowing to the customer end.

[0007] In another embodiment, an image correlation analysis module may be included, which can analyze a group image of an object to be inspected consisting of multiple images of the object to be inspected. If there are differences between the multiple images of the object to be inspected, at least one difference image feature can be compared from the group image of the object to be inspected, and accordingly, the classifier subsystem can parse the difference image features generated by the image correlation analysis module to make a classification decision. After the present invention is implemented according to this embodiment, the image correlation analysis module can treat multiple images of the object to be inspected (i.e., original images) obtained by the imaging device, or multiple abnormal images of the object to be inspected filtered out by the imaging device based on automatic optical detection technology, as group images of the object to be inspected, and compare the difference image features, and it can be applied to the situation where there are generally fewer abnormal images of certain objects to be inspected (i.e., training samples of defective images), and the aforementioned abnormal images of the object to be inspected can also include defect location information corresponding to the re-judgment mark.

[0008] In another embodiment, a second imaging device may be included, which is connected to the operating station host and the training subsystem respectively, and can image the inspected object that has been repaired based on the abnormal image to obtain at least one image of the inspected object after repair, and can mark at least one image difference feature between the inspected object image and the repaired inspected object image, and send the marked image difference feature and the repaired inspected object image to the training subsystem in real time. After the present invention is implemented according to this embodiment, it is particularly suitable for inspected objects that are repaired in real time, for which the scrapping cost is higher than the repair cost and the manual repair cost is generally lower than the return cost of the client. In addition, the operating station host of this embodiment does not need to run the marking module, so that the training subsystem can learn the location of the defect and the value judgment of whether there is a defect based on the inspected object images before and after the repair.

[0009] The classifier subsystem may have a tag screening unit, which can screen out classification decisions with lower credibility, and then send them to the operation station host by the classifier subsystem.

[0010] Among them, the operation station host can run a historical mark review module, which can display a historical classification decision that is highly correlated with the classification decision. The historical classification decision includes a historical image with at least one historical mark. The historical mark review module can input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

[0011] The at least one image of the object to be inspected obtained by the classifier subsystem from the imaging device is at least one abnormal image of the object to be inspected filtered out by the imaging device based on the AOI technology.

[0012] Among them, the second imaging device is an augmented reality display, which is used to present the classification decision on the augmented reality display, and the second imaging device is also used to record a repair point information of the inspected object during repair, so as to specify the repair point information as at least one image difference feature between the inspected object image and the repaired inspected object image, and send it to the training subsystem by the second imaging device, so that the training subsystem can update the training module and a marking database connected to the training module in real time based on the marked image difference feature and the repaired inspected object image.

[0013] Another object of the present invention is to provide an intelligent production line monitoring method, wherein an imaging device captures an image of an object to be inspected, and after obtaining at least one image of the object to be inspected, the following steps can be performed: (A) an image feature analysis and classification decision making step: a classifier subsystem reads a training module to analyze the image features of the image of the object to be inspected, and makes at least one classification decision to send to an operating station host. If the classification decision is considered abnormal, the classification decision includes a complete image and an abnormal image with a re-judgment mark; (B) an automatic screening mark step: If there are multiple re-judgment marks, the classifier subsystem screens out the classification decisions with lower credibility, and then the classifier subsystem sends them to the operation station host; (C) an input re-judgment operation step: a marking module of the operation station host inputs a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host sends it to the training subsystem; and (D) a real-time update training module step: the training subsystem real-time updates the training module and a marking database connected to the training module based on the updated classification decision.

[0014] Among them, when step (C) is executed, the marking module also marks a missed mark associated with the complete image to input a second re-judgment operation to generate a second classification decision that replaces the classification decision, and the second classification decision is sent to the training subsystem by the operation station host. Thereafter, when step (D) is executed, the training subsystem also updates the training module and the marking database in real time based on the second classification decision.

[0015] Wherein, when step (B) is executed, the operation station host displays a historical classification decision that is highly correlated with the classification decision, and the historical classification decision includes a historical image with at least one historical tag, and enables a historical tag review module to input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

[0016] Wherein, before executing step (A), the at least one image of the object to be inspected obtained by the classifier subsystem from the imaging device is at least one abnormal image of the object to be inspected filtered out by the imaging device based on AOI technology.

[0017] Another object of the present invention is to provide an intelligent production line monitoring method, wherein an imaging device images an object to be inspected, and after obtaining at least one image of the object to be inspected, the following steps can be performed: (A) a group image analysis step: an image correlation analysis module analyzes a group image of the object to be inspected consisting of multiple images of the object to be inspected. If there are differences between the multiple images of the object to be inspected, the image correlation analysis module compares at least one difference image feature from the group image; if there are no differences between the multiple images of the object to be inspected, the image correlation analysis module transfers the group image of the object to the classifier subsystem; (B) an image feature analysis and classification decision making step: if there are differences between the multiple images of the object to be inspected, a classifier subsystem reads a training module to analyze the difference image feature and make a classification decision; if there are no differences between the multiple images of the object to be inspected, the classifier subsystem The training module is read to analyze the image features of the multiple images of the inspected object and make the classification decision, which is sent by the classifier subsystem to an operation station host, wherein if the classification decision is considered abnormal, the classification decision includes a complete image and an abnormal image with at least one re-judgment mark; (C) an automatic screening mark step: if the re-judgment mark is multiple, the classifier subsystem screens out the classification decision with lower credibility, and then the classifier subsystem sends it to the operation station host; (D) an input re-judgment operation step: a marking module of the operation station host inputs a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host sends it to the training subsystem; and (E) a real-time update training module step: the training subsystem updates the training module and a marking database connected to the training module in real time based on the updated classification decision.

[0018] Among them, when step (D) is executed, the marking module also marks a missed mark associated with the complete image to input a second re-judgment operation to generate a second classification decision that replaces the classification decision, and the second classification decision is sent to the training subsystem by the operation station host. Thereafter, when step (E) is executed, the training subsystem also updates the training module and the marking database in real time based on the second classification decision.

[0019] Wherein, when step (C) is executed, the operation station host displays a historical classification decision that is highly correlated with the classification decision, and the historical classification decision includes a historical image with at least one historical tag, and enables a historical tag review module to input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

[0020] Wherein, before executing step (A), the at least one image of the object to be inspected obtained by the classifier subsystem from the imaging device is at least one abnormal image of the object to be inspected filtered out by the imaging device based on AOI technology.

[0021] Another object of the present invention is to provide an intelligent production line monitoring method, wherein an imaging device captures an image of an object to be inspected, and after obtaining at least one image of the object to be inspected, the following steps can be performed: (A) a step of analyzing image features and making classification decisions: a classifier subsystem reads a training module to analyze the image features of the image of the object to be inspected, and makes at least one classification decision to send to an operating station host. If the classification decision is considered abnormal, the classification decision includes at least one abnormal image; (B) a step of capturing a repaired image and marking: a second imaging device captures the base image of the object to be inspected, and the base image of the object to be inspected is captured by the second imaging device. The object to be inspected that has been repaired based on the abnormal image is imaged to obtain at least one repaired object image, and at least one image difference feature between the object image and the repaired object image is marked, and the repaired object image is regarded as a non-abnormal image and sent to the training subsystem by the second imaging device; and (C) a step of instantly updating the training module: the training subsystem instantly updates the training module and a marking database connected to the training module based on the marked image difference feature and the repaired object image.

[0022] Wherein, before executing step (A), the at least one image of the object to be inspected obtained by the classifier subsystem from the imaging device is at least one abnormal image of the object to be inspected filtered out by the imaging device based on AOI technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0024] Figure 1 : A block diagram of a first embodiment of the present invention (I);

[0025] Figure 2 : Block diagram of the first embodiment of the present invention (II);

[0026] Figure 3 : A schematic diagram of the re-judgment operation of the first embodiment of the present invention;

[0027] Figure 4 : A block diagram of a variation of the first embodiment of the present invention;

[0028] Figure 5 : A method flow chart of a variation of the first embodiment of the present invention;

[0029] Figure 6 : A block diagram of another variation of the first embodiment of the present invention;

[0030] Figure 7 : A block diagram of a second embodiment of the present invention;

[0031] Figure 8: A flow chart of a method according to a second embodiment of the present invention;

[0032] Fig. 9 : A block diagram of a variation of the second embodiment of the present invention;

[0033] Fig.10 : A block diagram of another variation of the second embodiment of the present invention;

[0034] Fig.11 : A block diagram of a third embodiment of the present invention;

[0035] Fig.12 : A method flow chart of the third embodiment of the present invention;

[0036] Fig.13 : A block diagram of a variation of the third embodiment of the present invention.

[0037] Description of Reference Numerals

[0038] 10Intelligent production line monitoring system

[0039] 101 Imaging Device

[0040] 102 Training Subsystem

[0041] 1021 Training Module

[0042] 1022 Tag Database

[0043] 103 Operation Station Host

[0044] 1031 Marking Module

[0045] R Reference Image

[0046] E Abnormal images

[0047] OP1 Review Operation

[0048] C1' Updated classification decision

[0049] OP2 Second review operation

[0050] C2 Second Classification Decision

[0051] 1032 History Mark Review Module

[0052] HC Historical Classification Decision

[0053] OP3 Third review operation

[0054] HC' Updated historical classification decisions

[0055] 104 Classifier Subsystem

[0056] 1041 Marker Screening Unit

[0057] C1 Classification decision

[0058] C1_LOW Classification decision

[0059] T Review mark

[0060] 105 Image Correlation Analysis Module

[0061] 106 second imaging device

[0062] ARD Augmented Reality Display

[0063] Fix patch information

[0064] I Image of the object being inspected

[0065] I_G Group image of inspected objects

[0066] I' Image of the inspected object after repair

[0067] S1 Intelligent production line monitoring method

[0068] S105 Analyze group images

[0069] S110 Analyzes image features and makes classification decisions

[0070] S120 Automatic Screening Mark

[0071] S130 Input re-judgment operation

[0072] S140 Live Update Training Module

[0073] S2 Intelligent production line monitoring method

[0074] S210 Analyze image features and make classification decisions

[0075] S220 captures repaired images and marks

[0076] S230 real-time update training module. DETAILED DESCRIPTION

[0077] The technical scheme in the embodiment of the present invention is described clearly and completely below in conjunction with the accompanying drawings in the embodiment of the present invention. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0078] See also Figure 1~Figure 2, which are all block diagrams of the first embodiment of the present invention, and please refer to Figure 3 The intelligent production line monitoring system 10 of this embodiment can monitor an image I of an object to be inspected taken by an imaging device 101, and mainly includes a training subsystem 102, an operation station host 103 and a classifier subsystem 104. The classifier subsystem 104 can be connected to the imaging device 101, the training subsystem 102 and the operation station host 103 through a network, wherein:

[0079] The imaging device 101 may include a plurality of image sensing units (not shown in the figure), and these image sensing units may be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) element. The imaging device 101 may also include at least one lens (not shown in the figure) for focusing the image I of the object to be inspected on the image sensing unit;

[0080] The training subsystem 102 can store at least one training module 1021 corresponding to the type of the object to be inspected (e.g., corresponding to a specific material number). The training module 1021 can be, for example, LeNet, AlexNet, VGGnet, NIN, GoogLeNet, MobileNet, SqueezeNet, ResNet, SiameseNet, NASNet, RNN or other training models based on convolutional neural networks, but not limited thereto, and some training models can correspond to training models for tasks such as object detection, object cutting, object classification, etc., and the training module 1021 can be connected to a label database 1022, the label database 1022 itself can be a database host, or a collection of multiple data tables (Table) stored in the training subsystem 102, the label database 1022 can store multiple pre-training data sets, for example, including pre-labeled classification decisions and corresponding training sample images, the label database 1022 can also store multiple images of objects I and multiple corresponding classification decisions updated at any time (updated from the return data of the operation station host 103, the classification decision can be the updated classification decision C1' and / or the second classification decision C2, the details are described below);

[0081] The operation station host 103 is connected to the training subsystem, and the operation station host 103 runs a marking module 1031. The operation station host 103 can also read the training subsystem 102 to adjust the weight parameters of the training models of each training module 1021 in real time through the network, or adjust the pre-training data set used by the training module 1021 in real time through the network.

[0082] The classifier subsystem 104 can read the training module 1021 to analyze the image features of the inspected object image I, and make a classification decision C1 to send to the operation station host 103. If the classification decision C1 is considered abnormal, the classification decision C1 may include a complete image and an abnormal image with at least one re-judgment mark T; preferably, the classification decision C1 of the present embodiment may also include a reference image; preferably, before the classifier subsystem 104 makes the classification decision C1, the classifier subsystem 104 or the imaging device 101 may first perform an image processing program on the inspected object image I. The aforementioned image processing program may be defined as one or a combination of an image preprocessing program, an image segmentation program (Region Segmentation) and a feature extraction program; preferably, the inspected object image I received by the classifier subsystem 104 from the imaging device 101 may also be an image of the inspected object image I received by the imaging device 101 based on an automated optical detection (Automated Optical Multiple images of abnormal objects to be inspected are filtered out by using the AOI (Automatic Inspection) technology, and the aforementioned abnormal objects to be inspected images may also include defect location information corresponding to the re-judgment mark T;

[0083] The marking module 1031 may input a re-judgment operation OP1 associated with the abnormal image to generate an updated classification decision C1′, which is then sent to the training subsystem 102 by the operation station host 103. The marking module 1031 may also mark a missed judgment mark (not shown) associated with the complete image to input a second re-judgment operation OP2, thereby generating a second classification decision C2 that replaces the classification decision C1, which is then sent to the training subsystem 102 by the operation station host 103.

[0084] The marking module 1031 can be presented on the display screen of the operation station host 103 in the form of a graphical user interface (GUI), such as Figure 3 As shown, the figure illustrates that the classification decision C1 may include a complete image, a re-judgment mark T, a reference image R and an abnormal image E, and the re-judgment operation OP1 may be generated by, for example, triggering a "GOOD" or "NG" button element (Button) shown in the figure, and the complete image may be presented by, for example, switching of a display mode, but is not limited thereto;

[0085] Accordingly, the training subsystem 102 can update the training module 1021 and the labeled database 1022 connected to the training module 1021 in real time according to the updated classification decision C1' and the second classification decision C2, that is, the updated classification decision C1' and the second classification decision C2 can be input into the pre-training data set of the labeled database 1022 in real time, so that the training module 1021 can train the training model of the training module 1021 in real time according to the updated pre-training data set;

[0086] As an example, the object to be inspected may be a printed circuit board (PCBA), a fastener, a flexible printed circuit board (Flexible Print Circuit), a rubber product (Rubber), a medical image (such as X-ray, ultrasound, CT, MRI, etc.), a digital pathological image or an image sensor (Image Sensor), but are not limited thereto; if the printed circuit board is inspected, it can be monitored whether there are short circuits, empty solder joints, too much tin, too little tin, tin holes, foreign matter, etc.; if the fastener is inspected, it can be monitored whether there are scratches, foreign matter, missing corners, etc.; if the flexible printed circuit board or rubber product is inspected, it can be monitored whether there are scratches, burrs, missing corners, foreign matter, etc.; if the image sensor is inspected, it can be monitored whether there are defects, foreign matter, etc.; if the medical image is inspected, for example, the lesion can be monitored.

[0087] As mentioned above, after the present invention is implemented according to the first embodiment, since the operation station host 103 can not only monitor the abnormal image with the re-judgment mark T, but also check and mark the complete image of the inspected object I, compared with the existing practice that the operator can only check and mark the defective image emitted by the automatic appearance inspection device, the present invention can enable the operation station host 103 to monitor the complete image and the abnormal image judged as abnormal by the classifier subsystem 104 during the re-judgment stage, and can avoid the abnormal inspected objects missed by the classifier subsystem 104 from being transported to the next stage of the production line, and at least achieve the beneficial effect of reducing the probability of defective products flowing to the client. In addition, assuming that the inspected object is a general screw, since its repair cost (internal cost) may generally be higher than the client's return cost (external cost), this embodiment is particularly suitable for such inspected objects that do not require immediate repair.

[0088] Please continue to read Figure 4 and Figure 5, which are respectively a block diagram and a method flow chart of a variation of the first embodiment of the present invention. As a further improvement, the classifier subsystem 104 of the intelligent production line monitoring system 10 of this embodiment also has a mark filtering unit 1041 that can be adjusted to an on or off state, and the present invention proposes an intelligent production line monitoring method S1 in this embodiment. After the classifier subsystem 104 analyzes the image features of the inspected object image I and makes a classification decision (step S110: analyzing the image features and making a classification decision), the mark filtering unit 1041 can, for example, automatically filter out a classification decision C1_LOW with a lower credibility based on a confidence score (confidence) corresponding to each re-judgment mark T, wherein the classification decision C1_LOW may include the re-judgment mark T and the abnormal image. Preferably, the classification decision C1_LOW of this embodiment may also include a reference image. Thereafter, the classification The training subsystem 104 may send the classification decision C1_LOW with a lower credibility to the operation station host 103 (step S120: automatic screening mark), so that the marking module 1031 can generate the updated classification decision C1' and / or the second classification decision C2 according to the classification decision C1_LOW with a lower credibility (step S130: input re-judgment operation), and update the marking database 1022 and the training module 1021 of the training subsystem 102 (step S140: real-time update of the training module). Thereby, the re-judgment mark T with a higher credibility and its corresponding abnormal image will no longer need to be re-judged by the operation station host 103, so the marking module 1031 of the operation station host 103 does not need to verify (or re-judgment) the re-judgment mark T for each abnormal image one by one, and can reduce the number of times the operation station host 103 inputs the re-judgment operation, thereby reducing the manpower consumption in the re-judgment stage.

[0089] Please continue to read Figure 6 , which is a block diagram of another variation of the first embodiment of the present invention, and please refer to Figure 2In order to enable the marking module 1031 of the operation station host 103 to have the opportunity to correct the previously misjudged classification decision when currently inputting the re-judgment operation OP1 and / or the second re-judgment operation OP2, so as to avoid the next time when the operation station host 103 receives an incorrect classification decision C1 from the classifier subsystem 104, as a further improvement, the operation station host 103 of the intelligent production line monitoring system 10 of this embodiment may run a historical marking re-judgment module 1032, which may display a historical classification decision HC that is highly correlated with the classification decision C1, wherein the historical classification decision HC may include a historical image with at least one historical marking. More specifically, for example, when the classification decision C1 received by the operation station host 103 includes an abnormal image with a "short circuit" feature, the historical marking re-judgment module Group 1032 can read out the historical classification decision HC which also includes the "short circuit" feature from the marking database 1022 of the training subsystem 102, but the historical classification decision HC has not been classified as an "abnormal" (i.e., previously re-judged as a good product) historical image, and the historical marking re-judgment module 1032 of this embodiment can be used to input a third re-judgment operation OP3 associated with the historical classification decision HC to modify the past historical classification decision HC, and then update the historical classification decision HC originally stored in the training subsystem 102 to an updated historical classification decision HC', which means that the updated historical classification decision HC' is immediately input into the pre-training data set of the marking database 1022, so that the training module 1021 can immediately train the training model stored in the training module 1021 based on the updated pre-training data set.

[0090] See also Figure 7 and Figure 8 , which are respectively a block diagram and a method flow chart of the second embodiment of the present invention. Figure 1~Figure 3The technology of the embodiments shown is similar, and the main difference is that the intelligent production line monitoring system 10 of the present embodiment may also include an image correlation analysis module 105 connected to the imaging device 101. Therefore, when the imaging device 101 in the intelligent production line monitoring method S1 completes imaging of the object to be inspected, the image correlation analysis module 105 may first analyze an object group image I_G composed of multiple object images I (step S105: analyzing group images). If there are differences between the multiple object images I, the image correlation analysis module 105 may compare at least one difference image feature from the object group image I_G, for example, capture the difference in a specific image position in the object images I, and make a classification decision C1; conversely, if there are no differences between the multiple object images I, the image correlation analysis module 105 105 can directly transfer multiple images of the object I to the classifier subsystem 104, so that the classifier subsystem 104 can directly analyze the image features of the multiple images of the object I and make a classification decision C1. The classifier subsystem 104 of this embodiment can be connected to the training subsystem 102, the image correlation analysis module 105 and the operation station host 103 respectively, wherein the classifier subsystem 104 of this embodiment can read the training module 1021 to analyze the difference image features sent by the object group image I_G, and make a classification decision C1 to send to the operation station host 103. In addition, the classification decision C1 may include information that can be input by the marking module 1031 and information sent by the operation station host 103 to the training subsystem 102, which are similar to the first embodiment and will not be repeated here.

[0091] Continuing from the above, after the present invention is implemented according to the second embodiment, the image correlation analysis module 105 can regard the multiple images of the inspected object I (i.e., the original images) acquired by the imaging device 101, or the multiple abnormal inspected object images filtered out by the imaging device 101 based on the automatic optical inspection (AOI) technology, as the inspected object group image I_G, and perform a comparison of difference image features. This can be applicable to the situation where the number of abnormal images of certain inspected objects (i.e., training samples of defective images) is generally small, and the aforementioned abnormal inspected object images can also include defect location information corresponding to the re-judgment mark T.

[0092] Please continue to read Fig. 9 , which is a block diagram of a variation of the second embodiment of the present invention, and please refer to Figure 1As a further improvement, the classifier subsystem 104 of the intelligent production line monitoring system 10 of the present embodiment may also have a label screening unit 1041, which may, for example, screen out a classification decision C1_LOW with a lower credibility based on a confidence score corresponding to each re-judgment mark (T, ...), wherein the classification decision C1_LOW may include the re-judgment mark T and the abnormal image. Preferably, the classification decision C1_LOW of the present embodiment may also include a reference image. Thereafter, the classifier subsystem 104 may send the classification decision C1_LOW with a lower credibility to the operation station host 103, and then the labeling module 1031 may continue to generate an updated classification decision C1' and / or a second classification decision C2. Thus, the labeling module 1031 of the present embodiment allows the user to avoid verifying (or re-judgmenting) the re-judgment mark T for each abnormal image one by one, thereby reducing the number of times the operation station host 103 inputs the re-judgment operation, thereby reducing manpower consumption.

[0093] Please continue to read Fig.10 , which is a block diagram of another variation of the second embodiment of the present invention. As a further improvement, the operating station host 103 of the intelligent production line monitoring system 10 of this embodiment can run a historical tag review module 1032, which can display a historical classification decision HC that is highly correlated with the classification decision C1, wherein the historical classification decision HC may include a historical image with at least one historical tag. The historical tag review module 1032 can be used to input a third review operation OP3 associated with the historical classification decision HC to modify the past classification decision, and update the historical classification decision HC stored in the training subsystem 102 to an updated historical classification decision HC', which means that the updated historical classification decision HC' is immediately input into the pre-training data set of the tag database 1022, so that the training module 1021 can immediately train the training model stored in the training module 1021 based on the updated pre-training data set.

[0094] See also Fig.11 and Fig.12 , which are respectively a block diagram and a method flow chart of the third embodiment of the present invention, the intelligent production line monitoring system 10 of this embodiment and Figure 1The technology of the embodiments shown is similar, and the main difference is that the intelligent production line monitoring system 10 of this embodiment may also include a second imaging device 106 connected to the operation station host 103 and the training subsystem 102 respectively, and the present invention proposes an intelligent production line monitoring method S2 in this embodiment. After the classifier subsystem 104 analyzes the image features of the inspected object image I and makes a classification decision C1 (step S210: analyzing the image features and making a classification decision), the second imaging device 106 can capture the inspected object that has been repaired based on the abnormal image sent by the classifier subsystem 104 (the classification decision C1 of this embodiment may not have a re-judgment mark) to obtain at least one repaired inspected object image I' (step S220: capturing the repaired image and marking), and the second imaging device 106 can mark (for example, based on automatic optical inspection (AOI) technology) at least one image difference feature between the inspected object image I and the repaired inspected object image I', and the repaired The image of the object under test I' is then regarded as a non-abnormal image and is sent by the second imaging device 106 to the training subsystem 102, so that the training subsystem 102 can update the training module 1021 and the marking database 1022 connected to the training module 1021 in real time according to the marked image difference features and the repaired image of the object under test I' (step S230: real-time update of the training module); wherein, as an example, the second imaging device 106 mentioned in this embodiment can complete the information connection by being set in the operation station host 103, and the operation station host 103 itself can also have a repair function, and the operation station host 103 can be an industrial or consumer robot; in addition, the component composition of the second imaging device 106 can be similar to the imaging device 101 of the first or second embodiment, and will not be repeated here; in addition, the operation station host 103 or the classifier subsystem 104 can also mark the image difference features between the object under test image I and the repaired image of the object under test I'.

[0095] As mentioned above, after the present invention is implemented according to the third embodiment, for inspected objects whose scrapping cost is higher than the repair cost and whose manual repair cost is generally lower than the client's return cost, such as PCBA plug-ins (Dual in-line Package, DIP) that need to be shipped immediately after production, the third embodiment is particularly suitable for such inspected objects that need to be repaired immediately. In addition, the operating station host 103 of this embodiment does not need to run the marking module 1031, so that the training subsystem 102 can learn the location of the defect and the value judgment of whether there is a defect based on the images of the inspected object before and after the repair.

[0096] Please continue to read Fig.13, which is a block diagram of a variation of the third embodiment of the present invention. As a further improvement, the second imaging device of the intelligent production line monitoring system 10 of this embodiment can be an augmented reality display ARD, which is used to make the abnormal image included in the classification decision C1 present on the augmented reality display ARD, and the re-judgment mark and reference image in the classification decision C1 of this embodiment can also be synchronously presented on the augmented reality display ARD, and the second imaging device of this embodiment (i.e., the augmented reality display ARD) can also record a repair point information Fix of the inspected object when it is repaired, so as to specify the repair point information Fix as an image difference feature, and send it to the training subsystem 102 by the second imaging device, so that the training subsystem 102 can update the training module 1021 and the label database 1022 connected to the training module 1021 in real time. Accordingly, this embodiment can achieve the purpose of replacing the inspection object because it allows the user to repair the inspected object through the augmented reality display ARD while automatically recording the repair point information of the inspected object. Figure 1 The function of the marking module 1031 is shown without requiring the user to perform a re-judgment operation or input a second re-judgment operation.

[0097] Among them, as an example, if the imaging device, training subsystem, operation station host, classifier subsystem, image correlation analysis module and second imaging device referred to in the present invention are all physical devices, they can all include a processor, which can have functions such as logical operations, temporary storage of operation results, and saving execution instruction locations. The processor can be, for example, a central processing unit (CPU), a virtual processor (vCPU), a microprocessor (MPU), a microcontroller (MCU), an application processor (AP), an embedded processor, a special application integrated circuit (ASIC), a tensor processing unit (TPU) or a graphics processing unit (GPU), etc., but are not limited to these.

[0098] As an example, the training subsystem, classifier subsystem and image correlation analysis module referred to in the present invention may be a server or a software module, and the aforementioned server and operation station host may be a physical server, or a server running in the form of a virtual machine (VM), or a server running in the form of a virtual private server (Virtual Private Server), or a public cloud, or a private cloud, or an edge terminal device (edge ​​device), or an embedded system (Embedded System) or a mobile device (such as a mobile phone), but are not limited to these.

[0099] As an example, the network referred to in the present invention may be a public or private network, such as a wireless network (e.g., 3G, 4G LTE, Wi-Fi, Bluetooth), a wired network, a local area network (LAN), a wide area network (WA), etc., but is not limited thereto.

[0100] In summary, the present invention, after being implemented, can at least achieve the beneficial effects of improving industrial processes, increasing defect detection efficiency and accuracy, being suitable for objects to be inspected with a small number of training samples, avoiding missed defects, reducing the number of defect re-determinations, correcting historical misjudgment decisions, automatically recording repair points without the need to re-determine classifier results, and being able to train models online in real time.

[0101] The above description is only a preferred embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Claims

1. An intelligent production line monitoring system, used for monitoring at least one image of an object to be inspected captured by an imaging device, characterized in that: include: a training subsystem storing a training module corresponding to the type of the object to be inspected; An operation station host connected to the training subsystem, the operation station host running a marking module; a classifier subsystem, connected to the imaging device, the training subsystem and the operating station host respectively; The classifier subsystem is used to read the training module to analyze the image features of the inspected object image and make a classification decision to send to the operation station host. If the classification decision is considered abnormal, the classification decision includes a complete image and an abnormal image with at least one re-judgment mark; The marking module is used to input a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host sends it to the training subsystem. The marking module is also used to mark a missed judgment mark associated with the complete image to input a second re-judgment operation to generate a second classification decision that replaces the classification decision, and the operation station host sends it to the training subsystem; as well as The training subsystem updates the training module and a label database connected to the training module in real time according to the updated classification decision and the second classification decision.

2. The intelligent production line monitoring system according to claim 1, characterized in that: The classifier subsystem has a tag screening unit, which is used to screen out the classification decisions with lower credibility, and then send them to the operation station host by the classifier subsystem.

3. The intelligent production line monitoring system according to claim 1, characterized in that: The operation station host runs a historical tag review module, which displays a historical classification decision that is highly correlated with the classification decision. The historical classification decision includes a historical image with at least one historical tag. The historical tag review module is used to input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

4. The intelligent production line monitoring system according to claim 1, 2 or 3, characterized in that: The at least one detected object image obtained by the classifier subsystem from the imaging device is at least one abnormal detected object image filtered out by the imaging device based on the AOI technology.

5. The intelligent production line monitoring system according to claim 1, characterized in that: Also includes: A second imaging device is connected to the operating station host and the training subsystem respectively, and is used to capture the image of the object to be inspected that has been repaired based on the abnormal image to obtain at least one repaired object image, and the repaired object image is regarded as a non-abnormal image, which is then sent by the second imaging device to the training subsystem so that the training subsystem can update the training module and the marking database connected to the training module in real time based on the marked object image, at least one image difference feature between the repaired object image, and the repaired object image.

6. The intelligent production line monitoring system according to claim 5, characterized in that: The second imaging device is an augmented reality display, which is used to present the classification decision on the augmented reality display, and the second imaging device is also used to record a repair point information of the inspected object when it is repaired, so as to specify the repair point information as the image difference feature between the inspected object image and the repaired inspected object image, and send it to the training subsystem by the second imaging device, so that the training subsystem can update the training module and the marking database connected to the training module in real time based on the marked image difference feature and the repaired inspected object image.

7. The intelligent production line monitoring system according to claim 5 or 6, characterized in that: The at least one detected object image obtained by the classifier subsystem from the imaging device is at least one abnormal detected object image filtered out by the imaging device based on the AOI technology.

8. An intelligent production line monitoring system for monitoring at least one image of an object to be inspected taken by an imaging device, characterized in that: include: a training subsystem storing a training module corresponding to the type of the object to be inspected; An operation station host connected to the training subsystem, the operation station host running a marking module; An image correlation analysis module, connected to the imaging device, for analyzing a group image of the object to be inspected consisting of a plurality of images of the object to be inspected, so as to compare at least one difference image feature from the group image; a classifier subsystem, connected to the training subsystem, the image correlation analysis module and the operation station host respectively; The classifier subsystem is used to read the training module to analyze the difference image features and make a classification decision to send to the operation station host, if the classification decision is considered abnormal, the classification decision includes a complete image and an abnormal image with at least one re-judgment mark; The marking module is used to input a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host sends it to the training subsystem. The marking module is also used to mark a missed judgment mark associated with the complete image to input a second re-judgment operation to generate a second classification decision that replaces the classification decision, and the operation station host sends it to the training subsystem; as well as The training subsystem updates the training module and a label database connected to the training module in real time according to the updated classification decision and the second classification decision.

9. The intelligent production line monitoring system according to claim 8, characterized in that: The classifier subsystem has a tag screening unit, which is used to screen out the classification decisions with lower credibility, and then send them to the operation station host by the classifier subsystem.

10. The intelligent production line monitoring system according to claim 8, characterized in that: The operation station host runs a historical tag review module, which displays a historical classification decision that is highly correlated with the classification decision. The historical classification decision includes a historical image with at least one historical tag. The historical tag review module is used to input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

11. The intelligent production line monitoring system according to claim 8, 9 or 10, characterized in that: The at least one detected object image obtained by the classifier subsystem from the imaging device is at least one abnormal detected object image filtered out by the imaging device based on the AOI technology.

12. An intelligent production line monitoring method, characterized in that: An imaging device captures an object to be inspected, and after obtaining at least one image of the object to be inspected, the following steps may be performed: A step of analyzing image features and making classification decisions: a classifier subsystem reads a training module to analyze the image features of the inspected object image, and makes at least one classification decision to send to an operation station host. If the classification decision is considered abnormal, the classification decision includes a complete image and an abnormal image with a re-judgment mark; A step of inputting a re-judgment operation: a marking module of the operation station host inputs a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host sends it to the training subsystem, and the marking module also marks a missed judgment mark associated with the complete image to input a second re-judgment operation, and generates a second classification decision that replaces the classification decision, and the operation station host sends it to the training subsystem; as well as A first real-time updating training module step: the training subsystem real-time updates the training module and a label database connected to the training module according to the updated classification decision and the second classification decision.

13. The intelligent production line monitoring method according to claim 12, characterized in that: The following steps are also included: An automatic screening and marking step: If there are multiple re-judgment marks, the classifier subsystem will screen out the classification decisions with lower credibility, and then the classifier subsystem will send them to the operation station host for the input re-judgment operation step.

14. The intelligent production line monitoring method according to claim 13, characterized in that: in, When the automatic screening and marking step is executed, the operation station host displays a historical classification decision that is highly correlated with the classification decision, and the historical classification decision includes a historical image with at least one historical mark, and enables a historical mark review module to input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

15. The intelligent production line monitoring method according to claim 12, 13 or 14, characterized in that: in, Before the steps of analyzing image features and making classification decisions are performed, the at least one image of the object to be inspected obtained by the classifier subsystem from the imaging device is at least one abnormal image of the object to be inspected filtered out by the imaging device based on AOI technology.

16. The intelligent production line monitoring method according to claim 12, characterized in that: The following steps are also included: a step of capturing and marking a repaired image: a second imaging device captures the inspected object that has been repaired based on the abnormal image to obtain at least one repaired inspected object image, and marks at least one image difference feature between the inspected object image and the repaired inspected object image, and regards the repaired inspected object image as a non-abnormal image, and sends the repaired inspected object image to the training subsystem by the second imaging device; and A second step of updating the training module in real time: the training subsystem updates the training module and the marking database connected to the training module in real time according to the marked image difference features and the repaired image of the inspected object.

17. The intelligent production line monitoring method according to claim 16, characterized in that: in, Before the steps of analyzing image features and making classification decisions are performed, the at least one image of the object to be inspected obtained by the classifier subsystem from the imaging device is at least one abnormal image of the object to be inspected filtered out by the imaging device based on AOI technology.

18. An intelligent production line monitoring method, characterized in that: An imaging device captures an object to be inspected, and after obtaining at least one image of the object to be inspected, the following steps may be performed: a step of analyzing group images: an image correlation analysis module analyzes a group image of the object to be inspected consisting of a plurality of images of the object to be inspected; if there are differences between the plurality of images of the object to be inspected, the image correlation analysis module compares at least one difference image feature from the group image; if there are no differences between the plurality of images of the object to be inspected, the image correlation analysis module transfers the group image of the object to the classifier subsystem; A step of analyzing image features and making a classification decision: if there are differences between the multiple images of the object to be inspected, a classifier subsystem reads a training module to analyze the difference image features and make a classification decision; if there are no differences between the multiple images of the object to be inspected, the classifier subsystem reads the training module to analyze the image features of the multiple images of the object to be inspected, and makes the classification decision, and the classification decision is sent by the classifier subsystem to an operating station host, wherein if the classification decision is considered abnormal, the classification decision includes a complete image and an abnormal image with at least one re-judgment mark; a step of inputting a re-judgment operation: a marking module of the operation station host inputs a re-judgment operation associated with the abnormal image to update the classification decision, and the operation station host sends it to the training subsystem; the marking module also marks a missed judgment mark associated with the complete image to input a second re-judgment operation, and generates a second classification decision that replaces the classification decision, and the operation station host sends it to the training subsystem; and A step of updating the training module in real time: the training subsystem updates the training module and a label database connected to the training module in real time according to the updated classification decision and the second classification decision.

19. The intelligent production line monitoring method according to claim 18, characterized in that: The following steps are also included: An automatic screening and marking step: If there are multiple re-judgment marks, the classifier subsystem will screen out the classification decisions with lower credibility, and then the classifier subsystem will send them to the operation station host for the input re-judgment operation step.

20. The intelligent production line monitoring method according to claim 19, characterized in that: in, When the automatic screening and marking step is executed, the operation station host displays a historical classification decision that is highly correlated with the classification decision, and the historical classification decision includes a historical image with at least one historical mark, and enables a historical mark review module to input a third review operation associated with the historical classification decision to update the historical classification decision stored in the training subsystem.

21. The intelligent production line monitoring method according to claim 18, 19 or 20, characterized in that: in, Before the group image analysis step is performed, the at least one detected object image obtained by the classifier subsystem from the imaging device is at least one abnormal detected object image filtered out by the imaging device based on the AOI technology.

Images