Determination device, determination method, and determination program product

Abstract

The present invention provides a determination device, a determination method, and a determination program product for determining the presence or absence of a defective portion on the basis of a monochrome image obtained by capturing an image of the surface of a determination object using a learned model on the premise of inputting a color image. Provided is a determination device provided with: an image conversion unit that performs conversion in which a monochrome image obtained by capturing an image of a determination object is multiplied in accordance with the number of channels input to a learned model used for determining the presence or absence of a defective portion of the determination object; and a determination unit that inputs the monochrome image converted by the image conversion unit into the learning-completed model, and determines whether or not a defect portion is present in the object to be determined on the basis of an output from the learning-completed model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a determination device, a determination method, and a determination program. BACKGROUND

[0002] Patent Literature 1 discloses a technology of performing detection of surface defects of an inspection object using a convolutional neural network from an image obtained by photographing a surface of the inspection object.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: International Publication No. 2023 / 282043 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] When a surface of a determination object, which is a determination object of determination of presence or absence of defects, is photographed using a color image, in addition to generation of false colors near an edge having a sharp contrast change, a focal point becomes blurred due to an influence of supplemental processing of colors. The color image is not suitable for capturing outlines or features of fine defects of about 0.5 mm to 1 mm, defects of a pattern, and the like. Therefore, when a surface defect of a determination object is detected using an image, a monochrome image is used.

[0008] However, it is not possible to input a monochrome image to a learned model that needs to input an image having a plurality of channel directions like a color image.

[0009] The present disclosure is an invention accomplished in view of the above-described problems, and an object thereof is to provide a determination device, a determination method, and a determination program that perform determination of presence or absence of a defective portion from a monochrome image obtained by photographing a surface of a determination object using a learned model that extracts a feature amount of a color image.

[0010] MEANS FOR SOLVING THE PROBLEMS

[0011] The determination device according to the first aspect of the present disclosure includes an image conversion section that performs conversion of multiplying a monochrome image obtained by photographing a determination object by a number of input channels of a learned model used in determination of presence or absence of a defective portion of the determination object; and a determination section that inputs the monochrome image converted by the image conversion section to the learned model, and performs determination of presence or absence of a defective portion of the determination object based on an output from the learned model.

[0012] The determination device according to the second aspect of the present disclosure is the determination device according to the first aspect, in which the image conversion section doubles the monochrome image after masking regions other than the region to be determined.

[0013] The determination device according to the third aspect of the present disclosure is the determination device according to the first aspect or the second aspect, in which the object to be determined is a cast or an object manufactured using an integrated die casting method.

[0014] In the determination method according to the fourth aspect of the present disclosure, a processor performs processing of doubling a monochrome image obtained by photographing an object to be determined in accordance with the number of channels of input to a learned model used in determination of the presence or absence of a defective portion of the object to be determined, inputting the doubled monochrome image to the learned model, and determining the presence or absence of a defective portion of the object to be determined based on output from the learned model.

[0015] The determination program according to the fifth aspect of the present disclosure causes a computer to perform processing of doubling a monochrome image obtained by photographing an object to be determined in accordance with the number of channels of input to a learned model used in determination of the presence or absence of a defective portion of the object to be determined, inputting the doubled monochrome image to the learned model, and determining the presence or absence of a defective portion of the object to be determined based on output from the learned model.

[0016] Effects of Invention

[0017] According to the present disclosure, it is possible to provide a determination device, determination method, and determination program that determine the presence or absence of a defective portion of an object to be determined using a learned model for extracting a feature amount from a color image by doubling the number of channels of the learned model in accordance with a monochrome image obtained by photographing a surface of the object to be determined. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A diagram showing an outline of a determination device according to an embodiment of the disclosed technology.

[0019] Figure 2 A block diagram showing a hardware structure of a determination device.

[0020] Figure 3 A block diagram showing an example of a functional structure of a determination device.

[0021] Figure 4 A diagram showing an outline of a conversion of a monochrome image performed by an image conversion section.

[0022] Figure 5A graph for expressing an evaluation of a result of determination by the determination section.

[0023] Figure 6 A flowchart for expressing a flow of a determination process performed by the determination device. DETAILED DESCRIPTION

[0024] Hereinafter, one example of an embodiment of the present disclosure will be described with reference to the drawings. In addition, in each drawing, the same reference signs are attached to the same or equivalent constituent elements and portions. Further, the dimensional ratios of the drawings are exaggerated for the purpose of explanation, and thus there are cases where they differ from actual ratios.

[0025] Figure 1 A graph for expressing an outline of the determination device according to the present embodiment. The determination device 10 according to the present embodiment is a device that determines the presence or absence of a defective portion of a determination target object using a monochrome image 1 obtained by photographing the determination target object. The photographing of the determination target object is performed in an inspection device (not illustrated) that performs an inspection of the determination target object. The reason for using the monochrome image 1 for the image obtained by photographing the determination target object is that false colors do not occur compared to a color image, and the focus is clear.

[0026] The determination device 10 according to the present embodiment uses a learned model that has learned image data in determination of the presence or absence of a defective portion of a determination target object. The learned model is, for example, a model that has learned in an image database attached with teacher labels such as ImageNet. Since such an image database is constituted of color images, the learned model that has learned in the image database also assumes that the input is a color image. The color image has, for example, a plurality of channel directions such as R (red), G (green), and B (blue). Therefore, the learned model that has learned in the image database assumes the input of an image having a plurality of channel directions, and thus it is not possible to input the monochrome image 1 having only one channel direction to the learned model that has learned in the image database. That is, it is not possible to use the learned model that has learned in the image database as a feature quantity extractor for the monochrome image 1.

[0027] Further, in a case where the monochrome image is learned to generate the learned model, a large amount of data sets need to be prepared, and in order to learn the large amount of data sets, a large computational cost, detailed knowledge of hyperparameters, and a large amount of time for learning are required. Therefore, in order to determine the presence or absence of a defective portion of a determination target object from the monochrome image 1, it is desirable to use an existing learned model that has learned using an existing data set.

[0028] Accordingly, the determination device 10 according to the present embodiment performs a transformation process that is multiplied by the number of input channels of the learned model, with respect to the monochrome image 1 obtained by photographing the determination target object, and inputs the transformed monochrome image 1 to the learned model, and determines the presence or absence of a defective portion of the determination target object using the output from the learned model. By performing the transformation process in this way in cooperation with the number of input channels of the learned model, the determination device 10 according to the present embodiment can determine the presence or absence of a defective portion of the determination target object from the monochrome image 1, using the existing learned model that has been learned using the existing data set.

[0029] Figure 2 A block diagram showing the hardware structure of the determination device 10 is shown.

[0030] As shown in Figure 2 , the determination device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input section 15, a display section 16, and a communication interface (I / F) 17. Each of the structures is connected together in a manner that enables communication with each other via a bus 19.

[0031] The CPU 11 is a central arithmetic processing unit that executes various programs or controls each section. That is, the CPU 11 reads out a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 implements the control of each of the structures and various arithmetic processing according to the program recorded in the ROM 12 or the storage 14. In the present embodiment, a determination program that determines the presence or absence of a defective portion of a determination target object from a monochrome image 1 is stored in the ROM 12 or the storage 14.

[0032] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The storage 14 is constituted by a storage device such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory, and stores various programs including an operating system and various data.

[0033] The input section 15 includes a pointing device such as a mouse and a keyboard, and is used for various inputs.

[0034] The display section 16, which is, for example, a liquid crystal display, displays various information. The display section 16 can also be of a touch panel type and also functions as the input section 15.

[0035] Communication interface 17 is an interface for communicating with other devices, such as using standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0036] When executing the above-described determination procedure, the determination device 10 uses the aforementioned hardware resources to implement various functions. The functional structure implemented by the determination device 10 will be described below.

[0037] Figure 3 A block diagram illustrating an example of the functional structure of the determination device 10.

[0038] like Figure 3 As shown, the determination device 10 has an image transformation unit 101 and a determination unit 102 as functional structures. Each functional structure is implemented by the CPU 11 reading and executing the determination program stored in the ROM 12 or the memory 14. Furthermore, as... Figure 3 As shown, the determination device 10 has a learned model 110. The learned model 110 is, for example, a pre-learned convolutional neural network. The learned model 110 is, for example, stored in the memory 14. Furthermore, although in Figure 3 The diagram shows a case where the learned model 110 is included inside the determination device 10, but the learned model that performs the determination may also be included in a device different from the determination device 10.

[0039] The image transformation unit 101 performs a transformation that multiplies the number of input channels of the learned model 110 used by the determination unit 102 in determining whether the object has defects, based on the monochrome image 1 obtained by photographing the object to be determined. Furthermore, the object to be determined may be, for example, a casting or an object manufactured using integrated die-casting technology. If the number of input channels of the learned model 110 is R (red), G (green), and B (blue), then the image transformation unit 101 performs a transformation that triples the number of monochrome image 1. Additionally, the learned model 110 used by the determination unit 102 for determination is a model that has been learned in advance using a large amount of color image data in a manner capable of determining whether the object has defects.

[0040] Figure 4Fig. 1 is a diagram for showing an outline of the conversion of the monochrome image by the image conversion section 101. The learned model 110 is premised on the input of an image of 224 pixels x 224 pixels in three channels, for example. Therefore, the image conversion section 101 converts the monochrome image 1 of one channel to the monochrome image 1' of three channels by making it three times as large, so as to be able to be input to the learned model 110. That is, the image conversion section 101 performs processing of converting the monochrome image 1 of one channel to the monochrome image 1' of three channels. By converting the monochrome image 1 of one channel to the monochrome image 1' of three channels, it is possible to perform the input of the monochrome image 1' to the learned model 110.

[0041] In addition, in the present disclosure, not limited to the learned model that has learned the RGB image, it is also possible to use a learned model of a CMYK image of four channels. In this case, the image conversion section 101 makes the monochrome image four times as large. Further, the image conversion section 101 can also make the monochrome image N times as large according to the number of channels N of the image at the time of learning the model.

[0042] The determination section 102 performs determination of the presence or absence of a defective portion of the object of determination using the monochrome image 1' that has been converted by the image conversion section 101. Specifically, the determination section 102 inputs the monochrome image 1' that has been converted from the monochrome image 1 by the image conversion section 101 to the learned model 110, and performs determination of the presence or absence of a defective portion of the object of determination based on the output obtained from the learned model 110. The defective portion that the determination section 102 determines is, for example, a defective portion that has occurred at the time of manufacture, such as a notch, a burr, a casting wrinkle, a water residue, a dent, a chipped edge, a discoloration, a short shot, a burn, a scratch, a foreign matter mixed in, and the like.

[0043] Figure 5 Fig. 4 is a diagram for showing the evaluation of the determination result of the determination section 102. Figure 5 The vertical axis of the diagram shown is the ROC (Receiver Operating Characteristic) curve of the TPR (True Positive Rate), and the horizontal axis is the ROC curve of the FPR (False Positive Rate). Figure 5 Fig. 3 shows what kind of determination the determination section 102 obtains from the learned model 110 in the case where the threshold value that is set to NG is changed, in the case of a binary value of OK (no defect) and NG (defect).

[0044] As shown in Fig. 4, the determination section 102 obtains the determination of the learned model 110 in the case where the threshold value that is set to NG is changed, in the case of a binary value of OK (no defect) and NG (defect). Figure 5As shown, when the threshold value is less than the predetermined value A, a tendency of false detection in which OK is determined as NG even though it is actually OK is shown, and when the threshold value is A or more, a tendency of omission in which NG is determined as OK even though it is actually NG is shown. Therefore, in this case, by setting the threshold value to A, even when the monochrome image 1' obtained by converting the monochrome image 1 is input to the learned model 110, the determination unit 102 is able to perform determination of the presence or absence of a defective portion using the learned model 110 which is premised on input of a color image. Note that the threshold value A can of course vary depending on the data set used for learning of the learned model 110, and can be appropriately set depending on the result of learning.

[0045] The determination device 10 according to the present embodiment performs determination using the learned model 110 which is a model learned using a large amount of data of color images, by having the image conversion unit 101 perform conversion in which the monochrome image 1 is multiplied by the number of input channels of the learned model 110.

[0046] The monochrome image 1 obtained by photographing the determination target object sometimes includes a non-determination target region which is not a determination target region. For such an image, it is useless to input even the non-determination target region to the learned model 110.

[0047] Therefore, the image conversion unit 101 can also perform multiplication processing after masking a region other than the determination target region with respect to the monochrome image 1 obtained by photographing the determination target object. The image conversion unit 101 can either mask the non-determination target region with a predetermined pattern or detect an edge portion of the monochrome image 1 to distinguish the determination target region and the non-determination target region and mask the non-determination target region when masking a region other than the determination target region with respect to the monochrome image 1.

[0048] The pattern to be masked can also be generated from a master image prepared in advance. Although the image conversion unit 101 can also simply synthesize the pattern in the monochrome image 1, there can be cases where the photographing angle, the photographing range, and the like differ between the master image and the monochrome image 1 obtained by photographing the determination target object. Therefore, the image conversion unit 101 can also scan a scan range set in the monochrome image 1 in an arbitrary number and an arbitrary size to search for a position matching the master image, and perform affine transformation of the pattern in a manner matching the position, synthesize the deformed pattern realized by the affine transformation with the monochrome image 1, thereby masking the non-determination target region. Further, the image conversion unit 101 can also determine a masking object using semantic segmentation in which a label is attached to each pixel of an image. By semantic segmentation, processing of generating a pattern to be masked in advance is not necessary, and thus masking can be performed.

[0049] Next, the operation of the determination device 10 will be described.

[0050] Figure 6 A flowchart showing the flow of the determination processing performed by the determination device 10 is shown in FIG. 10. The determination processing is performed by the CPU 11 reading out a determination program from the ROM 12 or the storage 14, and expanding and executing it in the RAM 13.

[0051] The CPU 11 acquires the monochrome image 1 obtained by photographing the determination target in step S101 from the inspection device.

[0052] Following step S101, the CPU 11 performs a transformation that multiplies the monochrome image 1 obtained by photographing the determination target by the number of input channels of the learned model 110 used in the determination of the presence or absence of a defective portion of the determination target in step S102. If the number of input channels of the learned model 110 is R (red), G (green), and B (blue), the CPU 11 performs a transformation that makes the monochrome image 1 three times larger.

[0053] Following step S102, the CPU 11 inputs the transformed monochrome image 1 to the learned model 110, and performs the determination of the presence or absence of a defective portion of the determination target based on the output obtained from the learned model 110 in step S103.

[0054] The determination device 10 according to the present embodiment performs the transformation that multiplies the monochrome image 1 by the number of input channels of the learned model 110 using the image conversion section 101 by performing the above processing, and thereby can perform the determination using the learned model 110, which is a model that has been learned using a large amount of data of color images in advance.

[0055] The embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to the above examples. The embodiments described above are illustrative embodiments, and not restrictive ones. As is apparent from the technical idea of the technical solutions described above, various modifications and changes can be suggested by those skilled in the art. In addition, it will be obvious that these modifications and changes also belong to the technical scope of the present disclosure.

[0056] In addition, the effects described in the above embodiments are illustrative or exemplary effects, and are not limited to the effects described in the above embodiments. That is, according to the above embodiments, it is apparent that one skilled in the art can implement other effects according to the technical idea of the present disclosure instead of or in addition to the effects described in the above embodiments.

[0057] In addition, in the above-described embodiments, the determination processing executed by the CPU 21 reading software (programs) can also be executed by various processors other than the CPU, such as a GPU (Graphics Processing Unit). As the processor in this case, a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) that can change the circuit structure after manufacture, and an ASIC (Application Spec Integrated Circuit) and the like that are special-purpose circuits as processors having a circuit structure specially designed to execute a specific process can be exemplified. Furthermore, the determination processing can be executed by one of these various processors, or can be executed using a combination of two or more processors of the same kind or different kinds (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, and the like). Furthermore, the hardware structure of these various processors is more specifically a circuit composed of circuit elements such as semiconductor elements.

[0058] Furthermore, although in the above-described embodiments, the manner in which the program of the determination processing is stored (installed) in the ROM in advance is described, it is not limited thereto. For example, the program can also be provided in a form recorded in a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory. Furthermore, the program can also be provided in a form downloaded from an external device via a network.

[0059] Symbol Explanation

[0060] 10 determination device;

[0061] 101 image conversion section;

[0062] 102 determination section.

Claims

1. A determination device comprising: an image transformation section that performs transformation that doubles the number of channels of input to a learned model used in determination of the presence or absence of a defective portion of a determination target object in conjunction with a monochrome image obtained by photographing the determination target object; and a determination section that inputs the monochrome image after transformation by the image transformation section to the learned model, and performs determination of the presence or absence of a defective portion of the determination target object based on output from the learned model.

2. The determination device according to claim 1, wherein the image transformation section doubles after masking a region other than a determination target region for the monochrome image.

3. The determination device according to claim 1, wherein the determination target object is a cast object or an object manufactured using an integrated die casting method.

4. A determination method in which a processor performs processing that: performs transformation that doubles the number of channels of input to a learned model used in determination of the presence or absence of a defective portion of a determination target object in conjunction with a monochrome image obtained by photographing the determination target object; and inputs the monochrome image after transformation to the learned model, and performs determination of the presence or absence of a defective portion of the determination target object based on output from the learned model.

5. A determination program product including a determination program in which the determination program causes a computer to perform processing that: performs transformation that doubles the number of channels of input to a learned model used in determination of the presence or absence of a defective portion of a determination target object in conjunction with a monochrome image obtained by photographing the determination target object; and inputs the monochrome image after transformation to the learned model, and performs determination of the presence or absence of a defective portion of the determination target object based on output from the learned model. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images