Article inspection device, creation method of learned model, and article inspection method

Abstract

To achieve high detection sensitivity while restricting nondefective erroneous detection, even when there are many types of foreign objects, in an inspection of articles.SOLUTION: An article inspection device (1) images conveyed articles to inspect the articles by using the obtained inspection image, and comprises: an image storage section (11) for storing the inspection image; an inspection processing section (13) for outputting a plurality of inference values, by subjecting the inspection image to a learned model (12) which is created according to machine learning by dividing one or more types of foreign objects that may be contained in the inspection image into a respective class, and which outputs information capable of deriving an inference value for each of the classes; and a determination section (14) for setting a threshold for each of the classes, and determining a foreign object by comparing the inference value for each of the classes with the threshold for each of the classes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an article inspection device, a method for creating a trained model, and an article inspection method. [Background technology]

[0002] One method for inspecting products and other objects is to use inspection images based on photography. A camera captures images of the objects as they move along a conveyor, and the images are inspected to see if any foreign objects are visible.

[0003] Patent Document 1 describes an inspection device that includes an image storage unit that captures multiple images of an object W to be inspected using different input systems under predetermined imaging conditions corresponding to each input system, and stores multiple inspection images each consisting of a set of the multiple images of the object obtained by the imaging, and a determination unit that processes the multiple inspection images stored in the image storage unit pixel by pixel based on a learning model that has been previously created by learning using images captured under the same imaging conditions as the multiple inspection images to determine the degree of quality defect, and compares the degree of quality defect with a preset threshold value to determine the quality state of the object to be inspected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-114828 Summary of the Invention [Problem to be solved by the invention]

[0005] Product inspection is carried out using a trained AI model that can identify defects in the shape, color, chipping, and other appearance of products traveling on a conveyor as foreign objects that are different from non-defective products, and can also identify foreign objects that are different from the products traveling on the conveyor as foreign objects.When there are many types of possible foreign objects, the sensitivity of foreign object detection can decrease due to the impact of dealing with other factors that may result in false positives.

[0006] False positive detection refers to the false positive detection of an object that is actually a good product. To avoid false positive detection, it is possible to adjust the threshold value of the trained model. However, changing the threshold value may result in the inability to detect a different type of foreign object. In other words, the detection sensitivity of foreign objects decreases.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide an article inspection device, a method for creating a trained model, and an article inspection method that increase detection sensitivity while suppressing false positive detections even when there are many types of foreign matter. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the present invention is characterized by the following [1] to [6]. [1] An article inspection device (1) that inspects an article using an inspection image obtained by capturing an image of the article being conveyed, an image storage unit (11) for storing the inspection image; an inspection processing unit (13) that applies a trained model (12) to the inspection image, the trained model (12) being created by performing machine learning on the inspection image by dividing the inspection image into classes for one or more types of foreign matter that may be included in the inspection image, and that outputs information from which an inference value can be derived for each of the classes, and outputs a plurality of inference values; a determination unit (14) that determines whether a foreign object is present by setting a threshold value for each of the classes and comparing the inferred value for each of the classes with the threshold value for each of the classes, Article inspection equipment. [2] The article inspection device according to [1], further comprising a storage means for readably storing the class and the type of foreign matter in association with each other. [3] the determination unit applies a mask to each of the classes so as not to detect foreign matter; [1] The article inspection device according to the present invention. [4] Output statistics including the number of detected foreign objects for each type of foreign object. [1] The article inspection device according to the present invention. [5] a step of subjecting an inspection image obtained by capturing an image of a conveyed article to machine learning a model that outputs information from which an inference value can be derived for each class, using learning data created by classifying the inspection image into classes based on the type of foreign matter that may be included in the inspection image; How to create a trained model. [6] 1. A method for inspecting an article using an article inspection device having a processor and a memory, comprising: an inference value output step of applying a trained model, which is created by performing machine learning on an inspection image obtained by capturing an image of the transported item, by dividing the image into classes based on one or more types of foreign matter that may be included in the inspection image, and which outputs information from which an inference value can be derived for each class, to the inspection image, and outputting a plurality of inference values; a determination step of determining whether a foreign object is present by setting a threshold value for each of the classes and comparing the inferred value for each of the classes with the threshold value for each of the classes; A method for inspecting goods. [Effects of the Invention]

[0009] According to the present invention, even when there are many types of foreign matter, it is possible to increase the detection sensitivity while suppressing erroneous detection of non-defective products. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating an example of an article inspection system including an article inspection apparatus according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a conceptual diagram illustrating an example of an operating screen of an article inspection device according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a conceptual diagram illustrating a screen during stoppage of an article inspection device according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a conceptual diagram illustrating a screen during stoppage of an article inspection device according to an embodiment of the present disclosure. [Figure 5]FIG. 10 is a conceptual diagram illustrating a foreign object. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a statistical display by type of foreign matter. [Figure 7] FIG. 10 is a conceptual diagram illustrating an example of a statistical display by type of foreign matter. [Figure 8] FIG. 10 is a conceptual diagram illustrating an example of a statistical display by type of foreign matter. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0012] FIG. 1 is a block diagram illustrating an example of an article inspection system including an article inspection device according to an embodiment of the present disclosure.

[0013] The article inspection system 100 includes an article inspection device 1, a conveyor 2, an imaging means 3, and a display operation unit 4.

[0014] The conveyor 2 transports the object to be inspected W. The object to be inspected W is, for example, a product. In addition to the product, foreign matter may be mixed in the conveyor 2. The object to be inspected W may be a foreign matter, and it is necessary to detect the foreign matter as a foreign matter rather than as a non-defective product. There are a wide variety of types of foreign matter.

[0015] The imaging means 3 captures an image of an object being transported. The object is transported, for example, by a conveyor 2. An example of an object is an object to be inspected W. The imaging means 3 is, for example, a camera, an X-ray camera, or a near-infrared camera. The camera captures an external image of the object to be inspected W. The X-ray camera captures a transmitted image of the object to be inspected W. The near-infrared camera captures an external image or a transmitted image of the object to be inspected W.

[0016] The article inspection device 1 includes a processor and a memory. The processor is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array).

[0017] The memory included in the article inspection device 1 stores the programs executed by the processor and various data used during execution. The memory may include a HDD, ROM, RAM, etc., and stores various programs (OS, application software, etc.) executed by the processor and various data. The memory comprises an image storage unit 11, a trained model 12, and a class information storage unit 15.

[0018] The image storage unit 11 stores the inspection image captured by the imaging means 3 .

[0019] The trained model 12 is a model obtained by performing machine learning in advance so as to input information based on an inspection image and output information from which multiple inferred values ​​can be derived. The trained model 12 may input the inspection image itself, or may input data obtained by performing preprocessing on the inspection image. The trained model 12 may output multiple inferred values ​​themselves, or may output information from which inferred values ​​can be derived by calculation. In the latter case, the processor appropriately performs calculations on the output information from the trained model 12 to derive multiple inferred values.

[0020] The trained model 12 is a trained model that is created by performing machine learning on classes that are divided into classes according to the type of foreign matter that may be contained in the inspection image, and outputs information that can be used to derive an inference value for each class.

[0021] In addition, the trained model 12 is generated by machine learning the training data created by dividing the inspection image obtained by photographing the transported item into classes based on the type of foreign matter that may be contained in the inspection image into classes, and then using this data to generate a model that outputs information from which an inference value can be derived for each class.

[0022] The class information storage unit 15 is a storage means for readably storing classes and types of foreign matter in association with each other.

[0023] The processor of the article inspection device 1 realizes the functions of the inspection processing unit 13 and the determination unit 14 by reading and executing a program stored in the memory.

[0024] The inspection processing unit 13 includes one or more foreign matter processing units 131 to 13n and a determination unit 14. Each of the foreign matter processing units 131 to 13n performs processing to detect foreign matter from the inspection image stored in the image storage unit 11. When multiple foreign matter processing units are provided, each foreign matter processing unit may be implemented with a different algorithm.

[0025] The foreign substance processing units 131 to 13n perform foreign substance processing using a trained model. More specifically, the foreign substance processing units 131 to 13n inspect the inspection image for foreign substances by applying a trained model 12 that outputs information from which an inference value can be derived for each class, which is created by performing machine learning on the inspection image by dividing the inspection image into classes according to the type of foreign substance that may be contained in the inspection image.

[0026] The determination unit 14 determines whether or not a foreign substance is detected based on the values ​​output from the foreign substance processing units 131 to 13n.

[0027] Foreign object processing units 131 to 13n that use trained model 12 output a plurality of inferred values. Determination unit 14 compares the plurality of inferred values ​​with a plurality of thresholds, respectively, to determine whether a foreign object is detected.

[0028] The display operation unit 4 includes an input device and an output device. The input device may be, for example, a general user input device such as a mouse, keyboard, or touch panel. A user inputs operation information for the article inspection device 1 from the input device. The output device may be, for example, a display device such as a monitor, and outputs information acquired from the article inspection device 1 to the user.

[0029] FIG. 2 is a conceptual diagram illustrating an example of a screen displayed during operation of an article inspection device according to one embodiment of the present disclosure.

[0030] The in-operation screen 5 is displayed, for example, on the output device of the display operation unit 4. The inspection status display area 51 displays information indicating the inspection status, such as operating, stopped, etc. The common information display area 52 displays general information, such as the current date and time.

[0031] The captured image display area 53 displays the inspection image captured by the imaging means 3. The inspection image reflects the product and foreign matter. The inspection information display area 54 displays inspection information. The inspection information includes, for example, the number of inspected objects that passed the inspection, the number of inspected objects that failed the inspection, the number of foreign matter detected, and statistical data related to the foreign matter. The statistical data related to the foreign matter will be described later.

[0032] The operation button area 55 has operation buttons arranged therein for operating the article inspection device 1. The user operates the article inspection device 1 by pressing the operation buttons.

[0033] 3 and 4 are conceptual diagrams illustrating screens displayed during stoppage of an article inspection device according to an embodiment of the present disclosure.

[0034] The inspection status display area 61, common information display area 62, inspection information display area 64, and operation button area 65 on the stopped screen 6 are similar to the inspection status display area 51, common information display area 52, inspection information display area 54, and operation button area 55 on the operating screen 5, so detailed explanations will be omitted.

[0035] In the stopped screen 6 shown in FIG. 3 , a setting display area 63 is displayed in a portion corresponding to the captured image display area 53 of the driving screen 5. The setting display area 63 displays a user interface such as check boxes for setting a threshold value for foreign object processing for each type of foreign object. In this example, the user can set a threshold value for a checked foreign object among foreign objects A to M. For example, when the user selects foreign object G in the setting display area 63 by checking the check box for foreign object G, a pop-up screen for setting a threshold value for the selected foreign object type, foreign object G, is displayed. In the pop-up screen, the threshold value can be changed by pressing the left or right arrow key or the like, and the change to the threshold value is confirmed by pressing the update button.

[0036] In the stopped screen 6 shown in FIG. 4, a setting display area 63 is displayed in a portion corresponding to the captured image display area 53 of the driving screen 5. In the setting display area 63, a user interface such as check boxes for masking foreign objects is displayed. In this example, it is possible to mask checked foreign objects out of foreign objects A to M. The masking process will be described later.

[0037] FIG. 5 is a conceptual diagram illustrating an example of a foreign object that is different from the product. In addition to the product, foreign objects may be mixed into the conveyor 2. There are various types of foreign objects, and the following are shown as examples: foreign object A, foreign object B, foreign object D, foreign object G, and foreign object I. The trained model 12 performs machine learning on these foreign objects. Note that defective products with poor appearance (shape, color, chipping) may also be added to the machine learning as foreign objects.

[0038] Various foreign objects appear in the inspection image. The inferred value for foreign object A derived based on the output from the trained model 12 is 0.9. The inferred value for foreign object B derived based on the output from the trained model 12 is 0.5. The inferred value for foreign object C derived based on the output from the trained model 12 is 0.7. In such a case, the problem is what value should be set for the threshold value X used by the determination unit 14. Note that the determination unit 14, for example, determines that an object is a foreign object when the inferred value exceeds the threshold, and determines that an object is a non-defective object when the inferred value is equal to or less than the threshold.

[0039] Suppose threshold value X is set to 0.8 to avoid false positive detection of foreign object A. In this case, the inferred value for foreign object A is 0.9, so the inferred value is greater than the threshold, and the judgment unit 14 can judge foreign object A to be a foreign object. The inferred value for foreign object B is 0.5, so the inferred value is less than the threshold, and the judgment unit 14 will judge foreign object B to be a non-defective object. In other words, foreign object B cannot be detected as a foreign object. The inferred value for foreign object C is 0.7, so the inferred value is less than the threshold, and the judgment unit 14 will judge foreign object C to be a non-defective object. In other words, foreign object C cannot be detected as a foreign object.

[0040] As can be seen from the above example, when there is a possibility that many different types of foreign matter may be present in the inspection image, it is difficult to determine what value to set the threshold to.Furthermore, there may be cases where a non-defective product is mistakenly determined to be a foreign matter, or conversely, a foreign matter detection error may occur where a foreign matter is mistakenly determined to be a non-defective product.

[0041] Therefore, the product inspection device 1 of the present disclosure uses a trained model 12 that is created by performing machine learning on classes that are divided into classes for each type of foreign matter that may be contained in an inspection image, and outputs information from which an inferred value can be derived for each class. Note that a class may contain one type of foreign matter, or may contain multiple types of foreign matter. In the former case, the trained model 12 outputs information from which an inferred value can be derived for each type of foreign matter. Furthermore, the correspondence between the classes of the trained model 12 that outputs information from which an inferred value can be derived for each class and the type of foreign matter is determined based on the correspondence between classes and types of foreign matter stored in the class information storage unit 15.

[0042] The following describes an example in which a class contains one type of foreign object and the trained model 12 outputs an inferred value itself. For example, the trained model 12 outputs an inferred value for foreign object A, an inferred value for foreign object B, and an inferred value for foreign object C.

[0043] The determination unit 14 uses a plurality of thresholds. For example, the determination unit 14 uses a threshold XA for foreign matter A, a threshold XB for foreign matter B, and a threshold XC for foreign matter C.

[0044] Three inferred values ​​derived from the output of the trained model 12, that is, an inferred value EA for foreign object A, an inferred value EB for foreign object B, and an inferred value EC for foreign object C, are grouped together and represented as inferred values ​​(EA, EB, EC). The three inferred values ​​obtained by inputting an inspection image containing foreign object A into the trained model 12 were (0.9, 0.01, 0.01). The three inferred values ​​obtained by inputting an inspection image containing foreign object B into the trained model 12 were (0.01, 0.5, 0.01). The three inferred values ​​obtained by inputting an inspection image containing foreign object C into the trained model 12 were (0.01, 0.01, 0.7).

[0045] In this case, if the judgment unit 14 sets three thresholds, namely, a threshold XA=0.9 for foreign matter A, a threshold XB=0.5 for foreign matter B, and a threshold XC=0.7 for foreign matter C, the judgment unit 14 can accurately detect foreign matter A, foreign matter B, and foreign matter C while avoiding false positive detection.

[0046] [Masking process] If the threshold value used by the determination unit 14 is set to the maximum inferred value, for example, 1.0, it is possible to mask foreign matter associated with that threshold value in the determination. This is because, no matter what the inferred value is, the inferred value will never exceed the threshold, and therefore the object W to be inspected will not be detected as a foreign matter. For example, if foreign matter B is to be masked, the determination unit 14 simply sets the threshold XB for foreign matter B to 1.0. The user can specify which type of foreign matter to mask via the setting display area 63 shown in FIG. 4. Note that the masking process may be performed such that the processing in the foreign matter processing units 131-13n and the determination in the determination unit 14 are not performed for the selected foreign matter, thereby shortening the processing time.

[0047] Fig. 6 is a conceptual diagram illustrating an example of a statistical display for each type of foreign matter. Fig. 7 is a conceptual diagram illustrating an example of a statistical display for each type of foreign matter. Fig. 8 is a conceptual diagram illustrating an example of a statistical display for each type of foreign matter.

[0048] FIG. 6 illustrates an example in which one inferred value is derived from the output of the trained model 12, and one threshold value is compared by the judgment unit 14 with this inferred value. The product inspection device 1 outputs a statistical value including the number of detected foreign objects for each type of foreign object. In this case, the number of detected foreign objects was 15 for foreign object G, 9 for foreign object B, and 1 for foreign object D. Note that the meaning of each item in this example is as follows: Foreign object refers to the type of foreign object. Count refers to the number of times that type of foreign object was detected. Count (%) refers to the frequency at which that type of foreign object was detected, expressed as a percentage of the total. Status refers to a classification of HIGH, MODERATE, or LOW according to the number of foreign objects of that type detected.

[0049] FIG. 7 illustrates a case in which an inferred value is derived for each type of foreign matter from the output of the trained model 12, and the threshold used by the judgment unit 14 is also set for each type of foreign matter. In this case, the number of foreign matter detections was one for foreign matter G, one for foreign matter B, and one for foreign matter D. Compared to the example in FIG. 6, the number of times that actually good products were mistakenly detected as foreign matters has decreased, and appropriate foreign matter detection has become possible. This is because the threshold can be set separately for each type of foreign matter, making it possible to make detailed foreign matter judgments depending on the type of foreign matter. Note that the meaning of each item in this example is the same as in FIG. 6.

[0050] FIG. 8 illustrates a case in which an inference value is derived for each type of foreign object from the output of the trained model 12, and the threshold used by the judgment unit 14 is also set for each type of foreign object. Furthermore, in FIG. 8, foreign objects B and G are masked. That is, the threshold XB for foreign object B is set to 1.0, and the threshold XG for foreign object G is set to 1.0. As a result, the number of detected foreign objects is 0 for foreign object G, 0 for foreign object B, and 1 for foreign object D. Since foreign objects G and B are masked, they are no longer detected as foreign objects. Note that the meaning of each item in this example is the same as in FIG. 6.

[0051] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications and alterations can be made within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. For example, the steps in the methods disclosed herein may be performed in any order as long as no contradictions arise. Furthermore, the components in the above embodiments may be combined in any order as long as they do not deviate from the spirit of the disclosure.

[0052] For example, in the embodiment, an example was described in which foreign objects different from the products flowing on the conveyor are inspected as foreign objects, but the present invention can also be applied to inspecting whether or not foreign objects are contained within products (items) flowing on the conveyor. [Explanation of symbols]

[0053] 1. Item inspection equipment 11 Image storage unit 12 Pre-trained models 13 Inspection processing section 14 Judgment section 2 Conveyor 3. Imaging Method 4 Display operation section 5 Driving screen 51 Inspection status display area 52 Common information display area 53 Captured image display area 54 Examination information display area 55 Operation button area 6 Stopped screen 61 Inspection status display area 62 Common information display area 63 Setting display area 64 Examination information display area 65 Operation button area 100 Item Inspection System

Claims

1. An article inspection device that inspects an article using an inspection image obtained by capturing an image of the article being conveyed, an image storage unit that stores the inspection image; an inspection processing unit that applies a trained model to the inspection image, the trained model being created by performing machine learning on the inspection image by dividing the inspection image into classes for one or more types of foreign matter that may be included in the inspection image, and that outputs information from which an inference value can be derived for each of the classes, to output a plurality of inference values; a determination unit that sets a threshold for each of the classes and determines whether or not a foreign object is present by comparing the inferred value for each of the classes with the threshold for each of the classes, Article inspection equipment.

2. 2. An object inspection apparatus according to claim 1, further comprising storage means for readably storing the class and the type of foreign matter in association with each other.

3. the determination unit applies a mask to each of the classes so as not to detect foreign matter; The article inspection device according to claim 1 .

4. Output statistics including the number of detected foreign objects for each type of foreign object. The article inspection device according to claim 1 .

5. a step of subjecting an inspection image obtained by capturing an image of a conveyed article to machine learning a model that outputs information from which an inference value can be derived for each class, using learning data created by classifying the inspection image into classes based on the type of foreign matter that may be included in the inspection image; How to create a trained model.

6. 1. A method for inspecting an article using an article inspection device having a processor and a memory, comprising: an inference value output step of applying a trained model, which is created by performing machine learning on an inspection image obtained by capturing an image of the transported item and classifying the inspection image into classes based on one or more types of foreign matter that may be included in the inspection image, and which outputs information from which an inference value can be derived for each of the classes, to the inspection image, and outputting a plurality of inference values; a determination step of determining whether a foreign object is present by setting a threshold value for each of the classes and comparing the inferred value for each of the classes with the threshold value for each of the classes; A method for inspecting goods.

Images