METHOD FOR DISCLOSURE OF METAL SUBSTANCES, DEVICE FOR DISCLOSURE OF METAL SUBSTANCES, AND METHOD FOR GENERATING TRAINED MODEL

Abstract

To provide a metal substance determination method, a metal substance determination device and a generation method of a learned model which can determine whether or not a foreign matter existing in an inspection object is a metal substance.SOLUTION: A metal substance determination method comprises: an image acquisition step of acquiring a photographed image obtained by imaging an inspection object WK; a channel image generation step of separating and extracting a plurality of channel components from the photographed image and generating a plurality of channel images BC, GC, RC obtained by converting the photographed image into an image for each channel component; and a determination step of determining whether or not a foreign matter FS existing in the inspection object WK is a metal substance from the channel images BC, GC, RC of the inspection object WK by using a learned model that has deep-learned the channel image of the metal substance for learning and the channel image of the non-metal substance for learning as teacher data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a metallic substance discriminating method, a metallic substance discriminating device, and a method for generating a trained model. [Background technology]

[0002] 2. Description of the Related Art There is known a metallic substance discrimination method for discriminating whether or not a substance such as a foreign matter present in an inspection object is a metallic substance (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-62237 A Summary of the Invention [Problem to be solved by the invention]

[0004] The inspection method (metallic substance discrimination method) described in Patent Document 1 has the disadvantage of making the process complicated by observing the fluorescent X-rays generated when a part (inspection object) is irradiated with X-rays and performing qualitative or quantitative analysis of the elements contained in the inspection object to discriminate whether the substance present in the inspection object is metallic or non-metallic.

[0005] An object of the present invention is to provide a metallic substance discrimination method, a metallic substance discrimination device, and a method for generating a trained model that can determine in a simple process whether a foreign substance present in an inspection object is a metallic substance or not. [Means for solving the problem]

[0006] A metallic substance discrimination method according to one embodiment of the present invention is a metallic substance discrimination method for discriminating whether or not a foreign substance present in an inspection object is a metallic substance, and includes the following steps: an image acquisition step for acquiring an image of the inspection object; a channel image generation step for generating a plurality of channel images by separating and extracting a plurality of channel components from the image and converting the image into images for each channel component; and a discrimination step for discriminating whether or not a foreign substance present in the inspection object is a metallic substance from the channel images of the inspection object using a trained model that has been deep-learned using the channel images of a metallic substance for training and the channel images of a non-metallic substance for training as teacher data.

[0007] In one embodiment of the metal substance discrimination method of the present invention, an imaging process is carried out to image the object to be inspected under each of a plurality of lights having different illumination methods, and in the channel image generation process, it is preferable to generate the plurality of channel images for each image captured under each of the lights.

[0008] In the metal substance discrimination method according to one aspect of the present invention, it is preferable that one of the plurality of illuminators is a coaxial epi-illuminator.

[0009] A metallic substance discrimination device according to one embodiment of the present invention is a metallic substance discrimination device that discriminates whether or not a foreign substance present in an inspection object is a metallic substance, and includes an image acquisition means for acquiring an image of the inspection object, a channel image generation means for separating and extracting a plurality of channel components from the image, and generating a plurality of channel images by converting the image into images for each channel component, and a discrimination means for discriminating whether or not a foreign substance present in the inspection object is a metallic substance from the channel image of the inspection object using a trained model that has been deep-learned using the channel images of a metallic substance for training and the channel images of a non-metallic substance for training as teacher data.

[0010] A trained model generation method according to one embodiment of the present invention is a trained model generation method for determining whether a foreign object present in an inspection object is a metallic substance, and includes the following steps: a training imaging step of imaging a training metallic substance and a training non-metallic substance; a training channel image generation step of separating and extracting multiple channel components from the image captured in the training imaging step and generating multiple channel images by converting the image captured in the training imaging step into images for each channel component; and a model generation step of performing deep learning on the channel images of the training metallic substance and the channel images of the training non-metallic substance as teacher data to generate the trained model.

[0011] In a method for generating a trained model according to one embodiment of the present invention, it is preferable that in the training imaging step, training metallic materials and training non-metallic materials are imaged under each of a plurality of lights having different lighting methods, and that in the training channel image generation step, the plurality of channel images are generated for each image captured under each light.

[0012] In the method for generating a trained model according to one aspect of the present invention, it is preferable that one of the multiple illuminations is a coaxial epi-illumination. Effect of the Invention

[0013] According to one aspect of the present invention, by simply imaging the object to be inspected, it is possible to determine from the captured image whether or not a foreign matter is a metallic substance, thereby eliminating the need for qualitative or quantitative analysis of elements and enabling a simple process for determining whether or not a foreign matter present in the object to be inspected is a metallic substance. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is an explanatory diagram of a metal substance discrimination system according to an embodiment. [Diagram 2] 1 is an explanatory diagram of a metal substance discrimination method according to an embodiment; [Diagram 3] An explanatory diagram of a method for generating a trained model according to one embodiment. [Figure 4]1 is an explanatory diagram of a metal substance discrimination method according to an embodiment; [Diagram 5] FIG. 11 is an explanatory diagram of a metallic substance discrimination system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Device configuration] In Figure 1, the metal substance discrimination device EA is a device that discriminates whether a foreign matter FS (see Figure 2) present in an inspection object WK (see Figure 2) is a metal substance or not, and is composed of a computer such as a personal computer or a server. The metallic substance determining apparatus EA comprises a storage means 10, a processing means 20, and an operation means 30, and together with an imaging means 40 and an output means 50 constitute a metallic substance determining system EA1.

[0016] The storage means 10 is composed of a memory, a hard disk, etc., and stores various programs for controlling the metallic substance discrimination device EA and the metallic substance discrimination system EA1. The storage means 10 also stores a trained model for discriminating whether or not a metallic substance is contained in the inspection object WK.

[0017] The processing means 20 is configured with a processor such as a CPU (Central Processing Unit) or a GPU (Central Graphics Processing Unit), and includes an image acquisition means 21, a foreign matter extraction means 22, a channel image generation means 23, and a discrimination means 24. The image acquiring means 21 acquires an image of the inspection object WK from the imaging means 40. The foreign substance extraction means 22 detects foreign substances FS that are not actually present in the inspection object WK from the captured image acquired by the image acquisition means 21, and extracts captured images of the foreign substance FS portions from the captured images. The channel image generating means 23 separates and extracts a plurality of channel components from the captured image, and generates a plurality of channel images RC, GC, BC (see FIG. 2) by converting the captured image into images for each channel component. For example, if the color space used in the captured image is RGB (red, green, blue), three channel images RC, GC, BC each consisting of R (red), G (green), and B (blue) color channel images are generated from one captured image, and if the color space used in the captured image is CMY (cyan, magenta, yellow), three channel images RC, GC, BC each consisting of C (cyan), M (magenta), and Y (yellow) color channel images are generated from one captured image. Note that the channel components of the captured image are coordinate components of the color space used in the captured image. The discrimination means 24 uses the trained model stored in the storage means 10 to discriminate, from the channel images RC, GC, and BC, whether or not a foreign matter FS present in the inspection object WK is a metallic substance.

[0018] The operation means 30 is made up of a keyboard, an operation panel, a touch panel, a mouse, various switches, a microphone for voice input operation, etc., and is capable of inputting operation signals by various operations to the metallic material discrimination apparatus EA.

[0019] The imaging means 40 is composed of a camera, a camera, a microscope with an imaging function, an imaging sensor, etc., and is capable of transmitting the captured image to the processing means 20.

[0020] The output means 50 is composed of display devices such as a display or panel, and alarm devices such as indicator lights or speakers, and is configured to output the determination results by the metal substance discrimination device EA on a screen or by lighting up or sound, etc.

[0021] [Trained model] The trained model is generated by deep learning using the training channel images RC, GC, and BC of metallic substances and the training channel images RC, GC, and BC of non-metallic substances as training data, and is generated by the following procedure shown in Figure 3.

[0022] First, metallic and non-metallic substances of various types, compositions, and shapes are prepared for learning, and images of these metallic and non-metallic substances are taken (step ST11). Images may be taken under a single selected illumination method, or under each of a plurality of selected illumination methods. Examples of illumination methods include transmitted illumination, coaxial epi-illumination, lateral illumination, and variable illumination. In this embodiment, images of one metallic and non-metallic substance are taken under lateral illumination using a ring illumination and under coaxial epi-illumination.

[0023] Next, the multiple channel components are separated and extracted from the captured image captured in step ST11, and multiple channel images RC, GC, and BC are generated from the captured image (step ST12). The reason for generating multiple channel images RC, GC, and BC from the captured image is that, since the way metal substances are captured, for example, the way the luster peculiar to metal is captured, differs for each of the channel images RC, GC, and BC, using the channel images RC, GC, and BC as teacher images rather than the captured image itself provides more information available for discrimination and improves discrimination accuracy. In this embodiment, R, G, and B channel components are separated and extracted from an image captured under side illumination, and the captured image is converted into color channel images for each of the R, G, and B channel components to generate three channel images RC, GC, and BC. Similarly, three channel images RC, GC, and BC are generated from an image captured under coaxial epi-illumination. That is, a total of six channel images RC, GC, and BC are generated, consisting of the three channel images RC, GC, and BC under side illumination and the three channel images RC, GC, and BC under coaxial epi-illumination. Each of the generated channel images RC, GC, and BC is labeled as metal or non-metal.

[0024] Thereafter, deep learning is performed using the channel images RC, GC, and BC of the learning metallic substances and the channel images RC, GC, and BC of the learning non-metallic substances as teacher data to generate a learned model in which the captured image is input and the determination result of whether the foreign matter FS is a metallic substance or not is output (step ST13). In the case of this embodiment, a convolutional neural network is used as the deep learning algorithm.

[0025] [Metallic substance identification method] A metallic material determining method carried out in the following procedure shown in FIG. 4 will be described by taking as an example the metallic material determining system EA1 equipped with the metallic material determining apparatus EA described above. First, a user of the metal material discrimination system EA1 (hereinafter simply referred to as "user") inputs a signal to start automatic operation to the metal material discrimination system EA1 via the operation means 30. Next, when the user or a conveying means (not shown), such as an articulated robot or a belt conveyor, places the inspection object WK at a predetermined position, the processing means 20 drives the imaging means 40 to image the inspection object WK as shown in FIG. 2 (step ST21). The imaging is performed under illumination using the same illumination method as that used when the trained model was created or an illumination method with similar characteristics. In the case of this embodiment, imaging is performed under side illumination 60A using a ring illumination and under coaxial epi-illumination 60B (see FIG. 2).

[0026] Thereafter, the image acquisition means 21 acquires a captured image of the inspection object WK from the imaging means 40 (step ST22). Next, the foreign matter extraction means 22 determines whether or not a foreign matter FS is present in the inspection object WK in the captured image acquired in step ST22 (step ST23). As a method for detecting foreign matter at this time, various techniques can be used, such as performing blob analysis after performing binarization processing on the captured image, performing differential filtering processing based on a captured image of a non-defective product of the inspection object WK, or using a judgment model by machine learning, and is not particularly limited. Then, if it is determined in step ST23 that there is no foreign matter FS, the process returns to step ST21 and proceeds to inspection of the next inspection object.

[0027] On the other hand, if it is determined in step ST23 that a foreign object FS is present, the foreign object extraction means 22 extracts an image of the foreign object FS portion from the captured image acquired in step ST22 (step ST24). Next, the channel image generation means 23 generates a plurality of channel images RC, GC, and BC from the captured image of the foreign object FS portion (step ST25). In the case of this embodiment, the channel image generation means 23 generates three channel images RC, GC, and BC of R, G, and B from the captured image under the side illumination 60A, and generates three channel images RC, GC, and BC of R, G, and B from the captured image under the coaxial epi-illumination 60B (see FIG. 2). Thereafter, the discrimination means 24 discriminates whether the foreign object FS is a metal substance from the channel images RC, GC, and BC using the trained model, and outputs the discrimination result to the output means 50 (step ST26). Next, the output means 50 notifies the user of the determination result by displaying it on a display device, turning on an alarm device, or emitting a sound from the alarm device (step ST27).

[0028] [Discrimination example] Metallic materials for training were aluminum, copper, lead, tin, zinc, SUS316L (stainless steel), SUS410L (stainless steel), and brass. Non-metallic materials for training were polyimide film, zirconia beads, pieces of phenolic resin, and pieces of paper. Images of each material were taken with a digital microscope under side illumination by a ring light. Images of these metallic and non-metallic materials were also taken with a digital microscope under coaxial epi-illumination. Then, six channel images RC, GC, and BC were generated from three images taken with side illumination and three images taken with coaxial epi-illumination for each material. Each channel image RC, GC, and BC was labeled as metallic or non-metallic, and a trained model was created using a convolutional neural network. When this trained model was used to discriminate the inspection object WK for verification, the prediction accuracy was over 95%, and good results were obtained.

[0029] In addition, the lighting method during imaging was set to side illumination using a ring light only, and the above metallic and non-metallic substances were captured with a digital microscope to generate three channel images (R, G, B) RC, GC, and BC, and a trained model was created from these channel images RC, GC, and BC. When this trained model was used to discriminate the inspection target WK for verification, the prediction accuracy decreased slightly, but it was still above 95%, and good results were obtained.

[0030] According to the above-described embodiment, by simply imaging the object to be inspected WK, it is possible to determine from the captured image whether or not the foreign matter FS is a metallic substance, so that there is no need to perform qualitative or quantitative analysis of the elements, and it is possible to determine whether or not the foreign matter FS present in the object to be inspected WK is a metallic substance in a simple process.

[0031] In addition, multiple channel images RC, GC, and BC are generated for each captured image taken under multiple lighting conditions with different lighting methods, which increases the amount of information available for discrimination and improves discrimination accuracy.

[0032] In addition, because one of the multiple lights used for shooting is a coaxial epi-illumination light, the characteristics of the metal are more easily reflected in the captured image, further improving the accuracy of discrimination.

[0033] As described above, the best configuration, method, etc. for carrying out the present invention are disclosed in the above description, but the present invention is not limited thereto. That is, the present invention is mainly illustrated and described with respect to a specific embodiment, but a person skilled in the art can make various modifications to the above-described embodiment in terms of shape, material, quantity, and other detailed configurations without departing from the scope of the technical idea and purpose of the present invention. In addition, the description of the shape, material, etc. disclosed above, which is limited, is illustratively described to facilitate understanding of the present invention, and does not limit the present invention, so that the description of the names of the members from which some or all of the limitations of the shape, material, etc. are removed is included in the present invention.

[0034] For example, the storage means 10 may be built in the metallic material determining apparatus EA, or may be of a type that is externally attached to the metallic material determining apparatus EA.

[0035] 5, the processing means 20 may include a model generating means 25 that generates a trained model by deep learning from a captured image acquired by the image acquiring means 21, and the discrimination means 24 may determine whether or not the foreign object FS is a metallic substance using the trained model generated by the model generating means 25. That is, the image acquiring means 21 may acquire a captured image of a metallic substance and a non-metallic substance for learning captured by the imaging means 40, the channel image generating means 23 may generate a plurality of channel images RC, GC, and BC from the captured image, and the model generating means 25 may generate a trained model using the plurality of channel images RC, GC, and BC. In this case, the operation of labeling each of the channel images RC, GC, and BC generated from the captured image of the foreign object FS portion as a metal or non-metal may be performed by the operation means 30. The image acquisition means 21 may acquire the image of the inspection object WK directly from the imaging means 40, or it may not acquire the image directly from the imaging means 40, and may acquire the image, for example, from an image storage means such as a server or database in which images acquired by the imaging means 40 are stored. The channel image generating means 23 may generate the multiple channel images RC, GC, BC not from an image from which the foreign matter FS portion has been extracted, but from the image itself captured by the imaging means 40. In this case, the foreign matter extracting means 22 may extract the foreign matter FS portion from the channel images RC, GC, BC, and the discriminating means 24 may discriminate whether or not the foreign matter FS is a metallic substance from the channel images RC, GC, BC from which the foreign matter FS portion has been extracted. The discrimination means 24 may discriminate whether or not a foreign matter FS present in the inspection object WK is a metallic substance, and may also discriminate the type and name of the metallic substance or non-metallic substance determined to be the foreign matter FS, and output these discrimination results to the output means 50. In this case, a label of metal or non-metal and a label of the type and name of the metallic substance or non-metallic substance may be attached to each of the channel images RC, GC, and BC of the metallic substance and non-metallic substance for learning, to create a trained model.

[0036] The operation means 30 may be configured so as to be separable from the metallic substance determination apparatus EA, or may be configured so as not to be separable from the metallic substance determination apparatus EA. The operation means 30 may or may not be included in the metallic substance discrimination device EA. If it is not included, the metallic substance discrimination system EA1 may be formed by the metallic substance discrimination device EA, the imaging means 40 and the output means 50, or an operation signal may be input to the metallic substance discrimination device EA from an external input device capable of communicating with the metallic substance discrimination device EA.

[0037] The imaging means 40 may be provided in the metallic substance discrimination device EA, or in the metallic substance discrimination system EA1, or it does not have to be provided in the metallic substance discrimination device EA or the metallic substance discrimination system EA1.If it is not provided, the image acquisition means 21 may acquire an image captured by an external imaging device capable of communicating with the metallic substance discrimination device EA or the metallic substance discrimination system EA1.

[0038] The output means 50 may be provided in the metallic substance discrimination device EA, or in the metallic substance discrimination system EA1, or it does not have to be provided in the metallic substance discrimination device EA or the metallic substance discrimination system EA1.If it is not provided, the determination result may be output to an external output device capable of communicating with the metallic substance discrimination device EA or the metallic substance discrimination system EA1.

[0039] As a deep learning algorithm for creating a trained model, in addition to a convolutional neural network, a deep neural network, a recurrent neural network, etc. can be used, and there is no particular limitation.

[0040] The type, material, shape, size, etc. of the inspection object WK, the foreign matter FS, the metal substance, and the nonmetallic substance are not particularly limited. For example, the inspection object WK may be an electronic component such as a capacitor or resistor, a battery material such as an anode material or a cathode material, a precision instrument part such as a personal computer or a printer, etc., or may be an assembly of multiple components such as a circuit board or an MLCC (Multi-Layer Ceramic Capacitor), or may be a single component such as a conductive tape or an insulating tape. The foreign matter FS may be mixed into the inspection object WK or may be attached to the inspection object WK. The metal substance may be any material such as iron, aluminum, stainless steel, lead, tin, zinc, gold, silver, copper, nickel, etc., and the nonmetallic substance may be any material other than a metal such as resin, plastic, wood, glass, ceramics, pottery, cloth, paper, fiber, etc.

[0041] The means and steps in the present invention are not limited in any way as long as they can perform the operations, functions, or steps described for those means and steps, and are not limited at all to the components and steps of a mere embodiment shown in the above embodiment. For example, the image acquisition step may be any step that acquires an image of an object to be inspected, and is not limited in any way as long as it is within the technical scope in light of the common general technical knowledge at the time of filing (the same applies to other means and steps). [Explanation of symbols]

[0042] EA…Metal substance discrimination device 10...Memory means 20...Processing means 21...Image acquisition means 22...Foreign matter extraction means 23...Channel image generating means 24...Discrimination means 30...Operation means 40...Imaging means 50...Output method BC…Channel Image FS…Foreign object GC...Channel image RC…Channel Image WK: Inspection object

Claims

1. An imaging step of imaging the inspection object under each of a plurality of illuminations (excluding transmitted illumination) having different illumination methods; A method for determining whether a foreign object present in an inspection object is a metallic substance, comprising the steps of: an image acquiring step of acquiring an image of the inspection object; a channel image generating step of generating a plurality of channel images by separating and extracting the plurality of channel components from the captured image and converting the captured image into images for the respective channel components; a discrimination process is carried out using a trained model that has been deep-learned using the channel images of a metallic substance for training and the channel images of a non-metallic substance for training as training data, to discriminate, from the channel images of the test object, whether or not a foreign object present in the test object is a metallic substance, and to discriminate the type and name of the metallic substance or non-metallic substance determined to be a foreign object; A metal substance discriminating method, wherein in the channel image generating step, the plurality of channel images are generated for each captured image taken under each illumination.

2. 2. The method according to claim 1, wherein one of said plurality of illuminations is a coaxial epi-illumination.

3. A metallic substance discrimination device for discriminating whether or not a foreign substance present in an inspection object is a metallic substance, comprising: an imaging means for imaging the inspection object under each of a plurality of illuminations (excluding transmitted illumination) having different illumination methods; an image acquisition means for acquiring an image of the inspection object; a channel image generating means for generating a plurality of channel images by separating and extracting a plurality of channel components from the captured image and converting the captured image into images for each of the channel components; Using a trained model that has been deep-learned using the channel images of a metallic substance for learning and the channel images of a non-metallic substance for learning as training data, a method is provided for determining whether a foreign object present in an object to be inspected is a metallic substance from the channel images of the object to be inspected. and a discrimination means for discriminating the type or name of the metallic substance or non-metallic substance determined to be a foreign object, The metal substance discriminating device is characterized in that the channel image generating means generates the plurality of channel images for each image captured under each illumination.

4. A method for generating a trained model for determining whether a foreign object present in an inspection object is a metal substance, comprising: a learning imaging step of imaging a learning metal material and a learning non-metal material; a learning channel image generating step of generating a plurality of channel images by separating and extracting a plurality of channel components from the captured image captured in the learning imaging step and converting the captured image into images for each of the channel components; A model generation process is carried out in which the channel images of the metallic substances for learning and the channel images of the non-metallic substances for learning are labeled with a metal or non-metal label and a label with the type or name of the metallic substance or the non-metallic substance, and the channel images are used as training data for deep learning to generate the trained model; In the learning imaging step, images of a learning metal material and a learning non-metal material are captured under a plurality of different illumination methods (excluding transmitted illumination), A method for generating a trained model, characterized in that in the training channel image generation process, the multiple channel images are generated for each captured image taken under each lighting.

5. The method for generating a trained model according to claim 4 , wherein one of the plurality of illuminations is a coaxial epi-illumination.

Images