Appearance inspection system and computer program

Abstract

Provided is a technique capable of suppressing both the missed detection of defects and over-detection. Provided is an appearance inspection system or the like that determines whether an object to be inspected is acceptable using an image of the object to be inspected. The appearance inspection system includes: a storage device that stores an acceptable product image, a reference table describing a plurality of thresholds, and an offset table describing a plurality of offset values; and an arithmetic circuit. The plurality of thresholds are respectively references for determining that an image of the object to be inspected included in the corresponding plurality of partial regions indicates a defect. The arithmetic circuit performs a process of determining whether the image feature amount of the partial image is classified into an over-detection category. When the image feature amount of the partial image is classified into the over-detection category, the arithmetic circuit further uses the offset value to change the threshold of the reference table at the position of the partial region, and uses the changed reference table to determine whether an image of a defect is included in the image of the object to be inspected.

Description

Technical Field

[0001] The present disclosure relates to an appearance inspection system and a computer program. Background Art

[0002] Conventionally, the following appearance inspection has been performed: by comparing an image obtained by photographing an object to be inspected with an image of a reference article (reference image), it is determined whether there is a defect in the object to be inspected, that is, whether the object to be inspected is a non-defective product.

[0003] In existing appearance inspection devices, after sufficient verification of the determination criteria for pass / fail determination, they are set manually in the device. When inspecting an object whose shape and surface pattern often change, it is necessary to frequently change the determination criteria, so it is difficult to use effectively.

[0004] For example, Japanese Patent Laid-Open Publication No. 2013-224833 discloses a technique for defining the determination criteria for pass / fail determination for each specific pixel and automatically setting the determination criteria using the average value / standard deviation of luminance. According to the technique of Japanese Patent Laid-Open Publication No. 2013-224833, it is possible to omit the frequent change of the determination criteria by manual work, and thus the appearance inspection can be used more effectively.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Patent Laid-Open Publication: Japanese Patent Laid-Open Publication No. 2013-224833 Summary of the Invention

[0008] Problems to be Solved by the Invention

[0009] However, in Japanese Patent Laid-Open Publication No. 2013-224833, if over-detection is suppressed, the determination criteria for the portion with a large deviation in pixel values become loose, so there is a possibility of overlooking a defect.

[0010] The present disclosure provides an appearance inspection system and a computer program capable of suppressing both the overlooking of defects and over-detection.

[0011] Means for Solving the Problems

[0012] The exemplary appearance inspection system of the present disclosure uses an image of an object to be inspected to determine whether the object to be inspected is qualified. Among them, the appearance inspection system has: a storage device that stores a plurality of images obtained by photographing a plurality of articles determined to be qualified products, a reference table describing a plurality of thresholds, and an offset table describing a plurality of offset values; an interface device that receives image data of the object to be inspected and attribute data indicating attributes related to manufacturing conditions; and an arithmetic circuit. The image of the object to be inspected and the plurality of images each include a plurality of partial regions. Each threshold of the plurality of thresholds in the reference table and each offset value of the plurality of offset values in the offset table are set corresponding to the plurality of partial regions respectively. The plurality of thresholds are respectively benchmarks for determining that the image of the object to be inspected included in each of the corresponding plurality of partial regions represents a defect. The plurality of offset values include a first type of offset value that changes a part of the plurality of thresholds and a second type of offset value that does not change the remaining part. The arithmetic circuit performs the following processing: Processing (a), performing a predetermined defect detection process on the image data, and when the image of the object to be inspected includes at least one defective image, extracting the object to be inspected as a defect candidate. In addition, the arithmetic circuit performs the following processing: Processing (b), cutting out a partial image from the image of the object to be inspected extracted as the defect candidate, wherein the position of the cut partial image is the position of the partial region corresponding to the first type of offset value in the offset table. In addition, the arithmetic circuit performs the following processing: Processing (c), determining whether the image feature amount of the cut partial image is classified into an over-detection category, wherein the over-detection category is a category in which the image feature amounts of the partial images of the plurality of images at the position of the partial region corresponding to the first type of offset value are classified at a ratio of a predetermined value or more. In addition, the arithmetic circuit performs the following processing: Processing (d), when the image feature amount of the cut partial image is classified into the over-detection category, using the first type of offset value to change the threshold of the reference table at the position of the partial region, and further improving the benchmark for determining that the image of the object to be inspected represents a defect. In addition, the arithmetic circuit performs the following processing: Processing (e), using the reference table with the changed threshold to determine whether the image of the object to be inspected includes at least one defective image.

[0013] An exemplary computer program of the present disclosure is executed by an arithmetic circuit of an appearance inspection system that determines whether an object to be inspected is qualified by using an image of the object to be inspected. The appearance inspection system includes: a storage device that stores a plurality of images obtained by respectively photographing a plurality of articles determined to be qualified products, a reference table describing a plurality of thresholds, and an offset table describing a plurality of offset values; an interface device that receives image data of the object to be inspected; and the arithmetic circuit. The image of the object to be inspected and the plurality of images each include a plurality of partial regions, and each threshold of the plurality of thresholds in the reference table and each offset value of the plurality of offset values in the offset table are set corresponding to the plurality of partial regions, respectively. The plurality of thresholds are respectively references for determining that an image of the object to be inspected included in each of the corresponding plurality of partial regions represents a defect. The plurality of offset values include a first type of offset value that changes a part of the plurality of thresholds and a second type of offset value that does not change the remaining part. The computer program causes the arithmetic circuit to execute the following processing (a): performing a defect detection process determined in advance on the image data, and extracting the object to be inspected as a defect candidate when an image of at least one defect is included in the image of the object to be inspected. In addition, the computer program causes the arithmetic circuit to execute the following processing (b): cutting out a partial image from the image of the object to be inspected extracted as the defect candidate, where the position of the cut partial image is the position of the partial region corresponding to the first type of offset value in the offset table. In addition, the computer program causes the arithmetic circuit to execute the following processing (c): determining whether an image feature amount of the cut partial image is classified into an over-detection category, where the over-detection category is a category in which image feature amounts of partial images of the plurality of images at the position of the partial region corresponding to the first type of offset value are classified at a ratio equal to or higher than a specified value. In addition, the computer program causes the arithmetic circuit to execute the following processing (d): when the image feature amount of the cut partial image is classified into the over-detection category, using the first type of offset value to change the threshold in the reference table at the position of the partial region, and further increasing the reference for determining that an image of the object to be inspected represents a defect. In addition, the computer program causes the arithmetic circuit to execute the following processing (e): determining whether an image of at least one defect is included in the image of the object to be inspected by using the reference table with the changed threshold.

[0014] Advantages of the Invention

[0015] According to an exemplary embodiment of the present disclosure, it is possible to suppress over-detection while suppressing the occurrence of undetected cases to a minimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1This is a diagram showing a structural example of the appearance inspection system 1000 of the present disclosure having the appearance inspection device 100.

[0017] Figure 2 This is a diagram mainly schematically showing a structural example of the appearance inspection device 100.

[0018] Figure 3 This is a diagram showing an example of the database 14Z in the storage device 14.

[0019] Figure 4 This is a diagram schematically showing an example of the image data obtained by photographing the workpiece 70.

[0020] Figure 5 This is a diagram showing an example of the image of the workpiece 70 divided into a plurality of partial regions 72.

[0021] Figure 6 This is a flowchart showing the process of the appearance inspection process.

[0022] Figure 7 This is a diagram showing an example of the result of the defect detection process when there is damage in a certain partial region 72b.

[0023] Figure 8 This is a diagram schematically showing the offset table 14c (upper part) and the position of the partial image cut out from the image 72 of the workpiece 70 (lower part).

[0024] Figure 9 This is for Figure 6 explaining the process of step S8.

[0025] Figure 10 This is for Figure 6 explaining step S10.

[0026] Figure 11 This is a flowchart showing the specific process of the appearance inspection process.

[0027] Figure 12 This is a flowchart showing the process of the database registration process.

[0028] Figure 13 This is a flowchart showing the process of the similar tendency confirmation process.

[0029] Figure 14 This is for Figure 13 specifically explaining steps S56 and S58.

[0030] Figure 15 This is a flowchart showing the process of the generation / update process of the offset table.

[0031] Figure 16It is a flowchart showing the process of updating the database 14Z.

[0032] Figure 17 It is a flowchart showing the process of applying the offset table.

[0033] Figure 18 It is a diagram showing the structure of the appearance inspection system 1100 of the modification example. Detailed implementation mode

[0034] Hereinafter, embodiments of the appearance inspection system of the present disclosure will be described with reference to the drawings. In this specification, detailed descriptions that are not necessary may sometimes be omitted. For example, detailed descriptions of well-known matters and repeated descriptions of substantially the same structures may sometimes be omitted. This is to avoid making the following description unnecessarily lengthy and to make it easy for those skilled in the art to understand. In addition, the inventor provides the drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and does not intend to limit the subject matter described in the claims by these. In the following description, the same reference numerals are given to the same or similar components.

[0035] 1. Structure of the appearance inspection system

[0036] Figure 1 It is a diagram showing a structural example of the appearance inspection system 1000 of the present disclosure having the appearance inspection device 100. Figure 2 It is a diagram mainly schematically showing a structural example of the appearance inspection device 100.

[0037] In the illustrated example, the appearance inspection system 1000 includes a camera device 30, an appearance inspection device 100, and a monitor 130. The appearance inspection device 100, the input device 120, and the monitor 130 can be implemented by a general digital computer system, such as a PC. In addition, the camera device 30 can be a digital camera. The camera device 30 is connected to the appearance inspection device 100 in a communicable manner through a wired communication cable or a wireless communication line (not shown). The monitor 130 is connected to the appearance inspection device 100 in a communicable manner through a wired communication cable (not shown).

[0038] The camera device 30 captures an object to be inspected and sends the image data obtained through the capture to the appearance inspection device 100. The appearance inspection device 100 performs an appearance inspection of the object to be inspected based on the image data received from the camera device 30 according to the process described later.

[0039] Refer to Figure 1 The process of appearance inspection of the object to be inspected performed in the appearance inspection system will be described. The object to be inspected is various products or components and other various items that are the objects of appearance inspection. Hereinafter, the object to be inspected may sometimes be referred to as a "workpiece".

[0040] The workpiece 70 is placed on the transfer table 62 and fixed to the transfer table 62 by the gripping mechanism. The workpiece 70 is, for example, a finished product of a hard disk drive, a housing of a hard disk drive assembled with a motor, or the like.

[0041] The transfer table 62 can move horizontally through the transfer platform 64 with the workpiece 70 placed thereon. The imaging device 30 is supported by the support member 60 above the transfer platform 64 so as to include the transfer platform 64 within its field of view. The imaging device 30 captures the workpiece 70 within the field of view. The workpiece 70 may also be gripped by a robotic arm and placed at the imaging position. Additionally, it may be that a light source (not shown) irradiates the workpiece 70 during imaging.

[0042] The image data obtained by imaging is sent from the imaging device 30 to the appearance inspection device 100. An example of the size of the image obtained by one imaging is 1600 pixels in width and 1200 pixels in height, and another example is 800 pixels in width and 600 pixels in height.

[0043] Next, refer to Figure 2 .

[0044] The appearance inspection device 100 includes an arithmetic circuit 10, a memory 12, a storage device 14, a communication circuit 16, and an image processing circuit 18. The components are connected to each other via a bus 20 so as to be able to communicate. Additionally, the appearance inspection device 100 has an interface device 22a for communicating with the imaging device 30, an interface device 22b for communicating with the input device 120, and an interface device 22c for outputting video data to the monitor 130. An example of the interface device 22a is a video input terminal or an Ethernet terminal. An example of the interface device 22b is a USB terminal. An example of the interface device 22c is an HDMI (registered trademark) terminal. Instead of these examples, other terminals may be used as the interface devices 22a to 22c. Additionally, the interface devices 22a to 22c may be wireless communication circuits for performing wireless communication. As such a wireless communication circuit, for example, a wireless communication circuit compliant with the Wi-Fi (registered trademark) standard that performs wireless communication using frequencies such as 2.4 GHz / 5.2 GHz / 5.3 GHz / 5.6 GHz is known.

[0045] The arithmetic circuit 10 can be, for example, an integrated circuit (IC) chip such as a central processing unit (CPU) or a digital signal processor. The memory 12 is a recording medium that stores a computer program 12p for controlling the operation of the arithmetic circuit 10 and constructs a learned convolutional neural network 12n described later. In addition, the entity of the learned convolutional neural network 12n is a plurality of parameters assigned to each artificial neuron constituting the input layer, the intermediate layer, and the output layer. The plurality of parameters include thresholds for comparing with the weights applied to the plurality of inputs to each artificial neuron and the weighted sum of the inputs (the sum of the products of the inputs and the weights). It should be noted that, in order for the neural network 12n to perform calculations and output results, the processing of the arithmetic circuit 10 is actually required. In addition, it is also possible to adopt hardware (application-specific integrated circuit (ASIC), GPGPU, programmable logic device) that integrates the plurality of parameters of the neural network 12n on the memory 12 and the arithmetic function of the arithmetic circuit 10 related to the neural network 12n.

[0046] The memory 12 does not need to be a single recording medium, and may also be a collection of multiple recording mediums. The memory 12 can include, for example, a semiconductor volatile memory such as a RAM and a semiconductor non-volatile memory such as a flash ROM. At least a part of the memory 12 may also be a removable recording medium.

[0047] The storage device 14 stores a database as a collection of various data. Figure 3 An example of the database 14Z in the storage device 14 is shown. Specifically, the storage device 14 stores a plurality of images 14a obtained by respectively photographing a plurality of articles determined to be qualified products, a reference table 14b describing a plurality of thresholds, and an offset table 14c describing a plurality of offset values. In addition, the storage device 14 may also store attribute data 14d indicating attributes related to the manufacturing conditions of the object to be inspected. The content and the like of each stored data will be described in detail later.

[0048] Refer again to Figure 2 。

[0049] The imaging device 30 is a device that outputs an image signal of data for generating an image of the object to be inspected. The image signal is sent to the arithmetic circuit 10 by wire or wirelessly. A typical example of the imaging device 30 is a camera having an area sensor such as a CMOS image sensor or a CCD image sensor in which a plurality of photodiodes are arranged in a matrix. The imaging device 30 generates data of a color image or a monochrome image of the object to be inspected. The imaging device 30 can use various cameras for appearance inspection.

[0050] Figure 4FIG. 0 schematically shows an example of image data obtained by photographing a workpiece 70. The image data is a frame image including an image of the workpiece 70. An exemplary image of the present embodiment is an arrangement of pixel values (“brightness” or “gray scale” values) of 256 gray levels that reflect the unevenness and patterns on the surface of the workpiece 70. Sometimes the pixel value is referred to as the luminance value or the density. In the case where the frame image is a color image, for example, the pixel value can be defined by three colors, red, green, and blue, each expressed in 256 levels of gray. In the present specification, sometimes the gray value, the luminance value, etc. are collectively referred to as “image feature amounts”. However, the “image feature amount” not only refers to the features related to the pixel value, but can also include, for example, the luminance histogram statistic for evaluating the basic properties of the image, and the Grey Level Co-occurrence Matrix (GLCM) feature amount of local features such as the strength of the contrast between pixels or the regularity of the pixels.

[0051] In one frame image, in addition to the workpiece 70, a part of the transfer table 62 can also be included as the background 70B. In Figure 4 the example of, the area 52 surrounded by the line 52L is the “inspection area”. One frame image can also include a plurality of inspection areas 52.

[0052] The position of the workpiece 70 in the frame image is aligned with a position predetermined in the frame image. In this alignment, a plurality of reference points of the workpiece 70 are made to coincide with a plurality of reference points in the field of view of the imaging device 30. In the first stage of such alignment, the physical configuration relationship of the workpiece 70 with respect to the lens optical axis of the imaging device 30 is adjusted. In the second stage of the alignment, the pixel positions (coordinates) of the captured image are adjusted. The alignment in the second stage includes translation, rotation, magnification, and / or reduction of the image using image processing techniques. As a result of such alignment, the inspection area 52 of each workpiece 70 is always aligned with the area surrounded by the line 52L. In this way, the images of the inspection areas 52 of each workpiece 70 can be processed in units of partial areas. In addition, the above alignment can be mainly executed by the arithmetic circuit 10 or the image processing circuit 18, but can also be executed by a mechanism (not shown) that changes the posture of the imaging device 30 to rotate or translate the imaging device 30 in a plane parallel to the transfer table 64 according to instructions from the arithmetic circuit 10 or the image processing circuit 18.

[0053] The monitor 130 is a device that displays the results of discrimination and the like executed by the image processing device 100. The monitor 130 can also display the image acquired by the imaging device 30.

[0054] The input device 120 is an input device that receives a designation including a selection area from a user and provides it to the arithmetic circuit 10. An example of the input device 120 is a touch panel, a mouse, and / or a keyboard. The monitor 130 and the input device 120 do not need to be always connected to the arithmetic circuit 10 by wire, and can be connected wirelessly or by wire via a communication interface only when needed. The monitor 130 and the input device 120 can also be a terminal device or a smart phone carried by a user.

[0055] 2. Outline of Appearance Inspection

[0056] The appearance inspection system 1000 of the present disclosure determines whether there is a defect in an object to be inspected using an image. The appearance inspection system 1000 of an exemplary embodiment uses a neural network to determine whether there is an image of a defect in the image of the object to be inspected. Whether there is a defect is determined for each partial area when the object image is divided into a plurality of partial areas.

[0057] Figure 5 An example of an image of the workpiece 70 divided into a plurality of partial areas 72 is shown. The plurality of partial areas 72 are in a grid pattern. Figure 5 An example of one partial area 72a that constitutes the plurality of partial areas 72 is illustrated. The plurality of partial areas 72 are set to be slightly larger than the image of the workpiece 70 so as to completely include the inspection area 52 of the workpiece 70 as a whole.

[0058] The appearance inspection system 1000 uses, for example, the pixel values of all the pixels that constitute the partial area 72a as an input to the neural network and obtains an output from the neural network. In the present embodiment, a convolutional neural network 12n that is pre-learned using images of various defects such as "dirt", "foreign matter", "damage", etc. and the names of the respective defects as teacher signals is used. In this specification, the type of defect is referred to as "category". The pixel values of all the pixels of the partial area 72a are input to the neural network. As a result, when it is determined that there is a defect, the neural network outputs what is considered to be the category name of the corresponding defect.

[0059] Further, in the present embodiment, while outputting the category name of the defect, a magnitude value indicating the degree of inconsistency between the image input to the neural network and the image that is determined to be a normal standard is output. In this specification, the output value is sometimes referred to as a "probability value" indicating the level of likelihood of being a defect.

[0060] Consider calculation methods for various probability values. Currently, a defect is set to be a "damage" on the surface of the housing of a hard disk drive. When the damage penetrates the coating on the aluminum plate, since the reflectivity of the aluminum plate is higher than that of the coating, the brightness value of the image of the damage is higher than that of the image in the case where there is no damage. When an image of a partial area including the damage is input to a convolutional neural network, image feature quantities related to the brightness value of the input image of the partial area can be obtained from an intermediate layer of the convolutional neural network. Each of the extracted image feature quantities is processed as a hash value of the input image of the partial area. Additionally, for example, when using any one of a plurality of images 14a stored in the storage device 14, each of which is an image obtained by photographing a plurality of articles determined to be qualified products, image feature quantities related to the brightness value of a partial area of the qualified product image corresponding to the position of the partial area determined to be a defect can also be extracted from an intermediate layer of the convolutional neural network. When comparing the hash value of the input image of the partial area with the hash value of the standard image of the partial area, it can be said that the closer the hash values are, the more similar the two are, and the farther the hash values are, the less similar the two are. Therefore, the larger the difference between the two hash values, the closer the assigned value is to 100, and the smaller the difference, the closer the assigned value is to 0, whereby the above-mentioned "probability value" can be obtained for each partial area determined to include a defect. Additionally, the probability value of the image of the partial area determined to be a defect can be adjusted to a value of at least 50% or more, for example. The above technique of extracting the image feature quantities extracted by the CNN as hash values is well-known. Therefore, detailed description thereof is omitted.

[0061] The class name and probability value of the defect output from the neural network are defect candidate detection results based on the learning result of the neural network.

[0062] However, even when an article is determined to include a defect, there may be cases where it is actually determined not to belong to a defect. For example, it is the following case: Although dirt adheres to the surface of the article, the degree thereof is small and does not substantially cause functional problems and appearance problems. In existing appearance inspections, a skilled inspector visually inspects the article, and thus determines that such dirt does not belong to a defect.

[0063] The appearance inspection system 1000 of the exemplary embodiment changes the defect determination criterion according to the relationship with a plurality of articles determined to be qualified products even when it is temporarily determined that an article includes a defect, and thus re-determines whether there is still a defect even so. In this specification, a case where an article that should originally be determined to be a qualified product is determined to have a defect is referred to as "over-detection". When over-detection occurs, the yield of the article decreases, and it has to be the subject of rework or the like, thereby increasing the cost of the article. In order to suppress such over-detection, based on the experience so far, the defect determination criterion is changed to expand the range determined to be a qualified product and thus suppress over-detection.

[0064] In addition, in order to suppress over-detection, a method of relaxing the defect determination criteria from the beginning has also been considered. However, if over-detection is suppressed too much, an article that should originally be determined as defective will be determined as a non-defective product, that is, undetected defects will occur, and the purpose of appearance inspection cannot be achieved.

[0065] The appearance inspection system 1000 of the exemplary embodiment first determines whether it is a non-defective product based on a criterion that is clearly determined as a non-defective product. In the case where it is determined that it is not a non-defective product and there are defects, the defect determination criteria are relaxed according to the relationship with a plurality of articles that have been determined as non-defective products up to that point.

[0066] The outline of the specific process is as follows.

[0067] When the appearance inspection system 1000 performs appearance inspection, the following data is stored in the storage device 14 in advance.

[0068] (a) A plurality of images obtained by respectively photographing a plurality of articles determined as non-defective products

[0069] (b) A reference table describing a plurality of thresholds

[0070] (c) An offset table describing a plurality of offset values

[0071] An explanation of the above (a) is given.

[0072] The articles determined as non-defective products are not limited to those initially determined as non-defective products. It also includes inspected objects that are temporarily determined to be defective and then determined to be non-defective products through, for example, visual determination by an inspector, and inspected objects whose results of the processing of the present disclosure described later are determined to be non-defective products. By including the latter two, the range of characteristics of the images determined as non-defective products can be expanded. Each of the "plurality of images" may include only the image of one article or may include the images of a plurality of articles. The images of each article determined as non-defective are used to change the criteria for determining whether there are defects.

[0073] An explanation of the above (b) is given.

[0074] The "plurality of thresholds" are respectively thresholds corresponding to each partial area when an image is divided into a plurality of partial areas. Each threshold is used as a criterion for determining whether each partial area contains an image of a defect when re-determining whether there are defects in an inspected object that is temporarily determined to be defective. In the present embodiment, the initial value of each of the "plurality of thresholds" is, for example, "0.5". The closer the threshold is to 1, the more only obvious defects, etc. are determined as "defects". That is, the closer the threshold is to 1, the more difficult it is to determine as a defect. Each of the plurality of thresholds can be updated to a higher threshold by each of the "plurality of offset values" described below.

[0075] In addition, a partial region may be a region composed of a plurality of consecutive pixels, or may be a region composed of one pixel, that is, a region defined by the size of one pixel. In the case of representing an article image in color, one pixel may be a region defined by the sizes of, for example, three sub-pixels of red, green, and blue arranged adjacent to each other.

[0076] An explanation will be given for the above (c).

[0077] "A plurality of offset values" are used to update a plurality of thresholds described in a reference table. When calculating the offset value of a certain partial region by the processing of this embodiment, the threshold of this partial region is updated according to the offset value so that the reference for determining a defect becomes higher. For example, the value obtained by adding the current threshold and the offset value is updated as the new threshold. Thereby, the reference for determining a defect can be further improved.

[0078] In addition, "a plurality of offset values" are not limited to the offset values for updating thresholds, and may include offset values that do not update thresholds, that is, offset values that maintain the current value. In the description of the embodiments to be described later, a mode having both an offset value (first type of offset value) that maintains the current value and an offset value (second type of offset value) for updating thresholds will be illustrated and described.

[0079] When using the offset value that maintains the current value, all partial regions are used as objects for calculation, so the program of the calculation process can be simplified. For example, when adding the offset value to the current threshold of each partial region, the following calculation process can be programmed: set 0 for the offset value that does not update the threshold, set a value greater than 0 for the offset value that updates the threshold, and add the offset value to the current threshold for all partial regions. On the other hand, when adding the offset value only to the threshold of a specific partial region, an operation for specifying the partial region as the object for updating the threshold is required, so the programming of the calculation process may be complicated.

[0080] For an article image of an image temporarily determined to contain a defect, the appearance inspection system 1000 uses a threshold that further improves the reference for determining a defect, and determines again whether each partial region of the article image contains a defective image.

[0081] When it is determined that none of the partial regions in the article image contains a defective image, the appearance inspection system 1000 registers the article image as one of the "plurality of images" shown in the above (a). On the other hand, when it is determined that any partial region in the article image continues to contain a defective image, the appearance inspection system 1000 determines that the article of the article image contains a defect.

[0082] According to the above processing, it is possible to suppress over-detection while preventing undetected cases.

[0083] In the present embodiment, the above-mentioned "defects" are divided into multiple categories, and it is determined whether each category is a defect. Multiple categories are exemplified, including "dirt", "foreign matter", "damage", "burr", "notch", "deformation", "color unevenness", "concavity and convexity", and "blur". One or more of the above-mentioned categories of "defects" may exist in the image of the object to be inspected. On the other hand, one or more of the above-mentioned categories of features that are once determined to be "defects" may exist in the image of the qualified product (the above (a)). Therefore, the reference table in the above (b) and the offset table in the above (c) only need to be set according to the categories of defects and used when determining the presence or absence of defects in each category. In the embodiments described later, an example corresponding to one category of defects is exemplified and described. Since it is complicated to describe all categories, the description is omitted.

[0084] The image processing device 100 that has acquired the image data performs the above-mentioned processing to perform the appearance inspection of the workpiece 70. Hereinafter, the content of this processing will be described in further detail.

[0085] 3. Appearance inspection processing

[0086] 3.1 Explanation of the idea of appearance inspection processing

[0087] Figure 6 It is a flowchart showing the process of appearance inspection processing.

[0088] In step S2, the arithmetic circuit 10 performs a predetermined defect detection process on the image data. When the image of the object to be inspected contains at least one image of a defect, the object to be inspected is extracted as a defect candidate. The "predetermined defect detection process" mentioned here refers to the defect detection process using the above-mentioned convolutional neural network. Through the defect detection process, when the image of the object to be inspected contains at least one image of a defect, the object to be inspected is extracted as a defect candidate. Figure 7 An example of the result of the defect detection process when there is damage in a certain partial area 72b is shown. In this example, the defect category output from the convolutional neural network is "damage", and the "probability value" indicating the level of the possibility of being a defect is 0.75. As a result, Figure 7 the workpiece 70 shown is extracted as a defect candidate. In addition, in Figure 7 , it is determined that there is a defect in only one partial area, but there may also be multiple defects. In this case, a defect category other than "damage" may be assigned. In this specification, the partial area determined to have a defect is sometimes referred to as a "defect candidate area".

[0089] Refer to again Figure 6。After the following step S4, the following processing is performed: For the inspected object temporarily determined to include a defect, the determination criterion for the defect is changed based on the relationship with a plurality of articles determined to be non-defective products, and it is re-determined whether there is still a defect even so.

[0090] In the following processing, the above-mentioned "reference table" and "offset table" prepared in advance are used. In the present embodiment, the "offset table" includes a first type of offset value that changes a part of a plurality of thresholds and a second type of offset value that does not change the remaining part. Using each offset value, the threshold value for determining whether the defect determination criterion set for each partial area is changed, or is not changed and remains as it is. In addition, the method for creating the offset table will be described later.

[0091] In step S4, the arithmetic circuit 10 cuts out a partial image from the image of the inspected object extracted as a defect candidate. The position of the cut-out partial image is the position of the partial area corresponding to the first type of offset value in the offset table 14c.

[0092] Figure 8 FIG. is a diagram schematically showing the offset table 14c (upper part) and the position of the partial image cut out from the image 72 of the workpiece 70 (lower part). The number of rows and columns of the offset table 14c arranged in a grid pattern is the same as the number of rows and columns that divide a plurality of partial areas. Thus, the position of each element of the offset table 14c can be made to correspond to the position of each partial area set in the image 72.

[0093] Refer to Figure 8 In the upper part of, there are a first type of offset value (a value other than 0) and a second type of offset value (a value of 0) in the offset table 14c. Among them, the images of the partial areas 72b, 72c, 72d, and 72e corresponding to the positions where the first type of offset value is assigned are cut out by the processing in step S4.

[0094] Refer to again Figure 6 。

[0095] In step S6, the arithmetic circuit 10 determines whether the image feature amount of the cut-out partial image is classified into an over-detection category. The "over-detection category" is a category in which the image feature amounts of the partial images of a plurality of images are classified at a ratio equal to or higher than a specified value at the position of the partial area corresponding to the first type of offset value.

[0096] The "over-detection category" is described in a more understandable way. First, prepare multiple images that are judged to be qualified products. The multiple images include images in which over-detection occurred in the previous appearance inspection process. Taking these multiple images as objects, cut out the partially detected areas (hereinafter referred to as "over-detection partial areas"). When performing the defect detection process of step S2 on the over-detection partial area, one or more image feature quantities related to the image of the input partial area can be obtained from the intermediate layer of the convolutional neural network. When performing a prescribed clustering process on each of the obtained image feature quantities, one or more clusters can be formed. Among them, the cluster with the largest number of elements can be said to be classified with image feature quantities that tend to be easily over-detected in multiple past images. The image feature quantities can vary according to the defect category in which over-detection occurred (hereinafter referred to as the "over-detection category"). The process of step S6 is a process of determining whether the partial image extracted in step S4 has image feature quantities belonging to the over-detection category.

[0097] In addition, when the number of image feature quantities extracted from the intermediate layer of the convolutional neural network is one, clustering can be performed in one-dimensional space according to the magnitude of the image feature quantity. When the number of image feature quantities is P (P: an integer of 2 or more), clustering can be performed in P-dimensional space according to the magnitude of the image feature quantity.

[0098] In step S8, when the image feature quantity of the cut partial image is classified as the over-detection category, the arithmetic circuit 10 uses the first type of offset value to change the threshold of the reference table at the position of the partial area. Thereby, the criterion for determining that the object to be inspected is a defective product is further improved.

[0099] The fact that the image feature quantity of the cut partial image is classified as the over-detection category can be said to mean that the image feature quantity has a tendency to be easily over-detected. The process of step S8 has the meaning of updating in such a way as to increase the threshold of the reference table at the position of the partial area to suppress such over-detection.

[0100] Figure 9 It is a diagram for explaining Figure 6 the process of step S8. Next, assume that there is a defect in the partial area 72b ( Figure 7 ). And in step S6, the image feature quantity of the partial image cut from the partial area 72b ( Figure 7 ) is classified as the over-detection category. For the sake of easy explanation, the partial image with the image feature quantity classified as the over-detection category is only the partial area 72b ( Figure 7 ).

[0101] The arithmetic circuit 10 adds the threshold values and offset values at the positions where the rows and columns are the same in the reference table 14b and the offset table 14c. In Figure 9 In the example of, in the reference table 14b, the value at the position 72-1 corresponding to the partial region 72b is "0.5". In the offset table 14c, the first type of offset value at the position 72-2 corresponding to the partial region 72b is "0.3". In addition, in the offset table 14c, the second type of offset value "0.0" is set for each position corresponding to a partial region other than the partial region 72b.

[0102] When adding the threshold value and the offset value at the position corresponding to the partial region 72b, it is "0.8". The value "0.8" is described at the position 72-3 in the updated reference table 14b-2 obtained by the addition. In addition, it is assumed that only the partial region 72b ( Figure 7 ) of the partial image having the image feature amount classified as the over-detection category, so the difference between the reference tables 14b and 14b-2 before and after the update is only the threshold value at the position 72-3. However, in the case where a plurality of image feature amounts are classified as the over-detection category, the threshold values at the positions of other partial regions can also be updated.

[0103] Refer to again Figure 6 .

[0104] In step S10, the arithmetic circuit 10 uses the reference table 14b-2 with the updated threshold value to determine whether at least one defective image is included in the image of the object to be inspected. Specifically, for the partial region 72b ( Figure 7 ) of the image determined to have a defect in step S2, the obtained probability value is compared with the threshold value of the reference table 14b-2. If the result of the comparison is that the probability value is equal to or greater than the threshold value, the arithmetic circuit 10 determines that a defect still exists. If the probability value is less than the threshold value or equal to the threshold value, the arithmetic circuit 10 determines that no defect exists.

[0105] Figure 10 is a diagram for explaining Figure 6 step S10. The arithmetic circuit 10 compares the probability value "0.75" that the image at the position of the partial region 72b contains a defect with the threshold value "0.8" of the reference table 14b-2 corresponding to that position. Since the probability value is less than the threshold value, the arithmetic circuit 10 determines that no defect exists in the partial region 72b of the image that was temporarily determined to contain a defect in step S2.

[0106] As described above, even in the case of being temporarily determined as a defect, the threshold value used as the determination criterion is updated only in a case similar to the over-detection in the result of the past non-defective product determination. As a result, it is possible to prevent the undetected defects while suppressing the over-detection of defects.

[0107] 3.2 Specific Instructions for Appearance Inspection Processing

[0108] A more specific example of appearance inspection processing will be described. In the following description, "attribute data" of the object to be inspected is introduced. The "attribute data" of the object to be inspected refers to data representing attributes related to the manufacturing conditions of each object to be inspected. As shown by the attribute data 14d of Figure 3 , attributes related to manufacturing conditions are, for example, the molds, production lines, manufacturing plants used when manufacturing each object to be inspected, and the manufacturing date of each object to be inspected. The attribute data can be at least one data selected from a set (group) of the above molds, production lines, manufacturing plants, manufacturing dates, etc. In addition, the attribute data can be represented, for example, by a combination of a string representing a name, a number, a symbol, and a numerical column. The attribute data is prepared for each object to be inspected. However, in the case where the manufacturing conditions of each object to be inspected are the same, one attribute data can be used as the attribute data for each object to be inspected.

[0109] The reason for using attribute data will be described. When a defect is confirmed in a certain object to be inspected, based on experience, it is known that other objects to be inspected with the same molds, production lines, and / or manufacturing plants, etc. will also be confirmed to have the same defect. Therefore, when re - judging whether there is a defect after temporarily determining that the object to be inspected has a defect, the same reference table and offset table are used for multiple objects to be inspected with the same attributes. Thus, the deviation of the result of the re - defect determination can be reduced.

[0110] As described above, after determining the attributes to be used, a set of reference tables and offset tables is prepared according to the types of attributes. The set of reference tables and offset tables can be prepared for each defect category.

[0111] Figure 11 is a flowchart showing the specific process of appearance inspection processing. This process is executed by the arithmetic circuit 10 using the convolutional neural network 12n in the memory 12 and various data in the storage device 14.

[0112] First, in step S20, the arithmetic circuit 10 acquires the image data of the object to be inspected. Consider various methods of acquiring image data. For example, the imaging device 30 captures an image of the object to be inspected to acquire image data and sends it to the interface device 22a. The arithmetic circuit 10 receives the image data received by the interface device 22a from the interface device 22a. Or, it can also be that the interface device 22a receives the image data of the object to be inspected from the imaging device 30, and after storing the image data in the storage device 14, the arithmetic circuit 10 acquires the image data from the storage device 14. In the latter example, the imaging device 30 may not be included as a part of the appearance inspection system 1000.

[0113] In step S22, the arithmetic circuit 10 acquires the attribute data of the object to be inspected.

[0114] In step S24, the arithmetic circuit 10 performs a defect detection process using the convolutional neural network 12n. Then, when a defect candidate region is detected within the image of the object to be inspected, the object to be inspected is extracted as a defect candidate. At this time, the defect category and the probability value are output for each defect candidate region.

[0115] In step S26, the arithmetic circuit 10 performs an offset table application process. The offset table application process is equivalent to Figure 9 the process shown. Through the offset table application process, the reference table 14b is updated by the offset table 14c, thereby obtaining a new reference table 14b-2. A specific description of the offset table application process will be described later.

[0116] In step S28, the arithmetic circuit 10 determines whether there is actually a defect in the defect candidate region by using the probability value output for each defect candidate region in step S24 and the threshold value of the updated reference table 14b-2. More specifically, the arithmetic circuit 10 compares the probability value at the position of the defect candidate region with the threshold value of the reference table 14b-2 corresponding to that position. When the probability value is equal to or greater than the threshold value, it is determined that there is still a defect in the object to be inspected, that is, the object to be inspected is a non-conforming product, and the process ends. On the other hand, when the probability value is less than the threshold value, the arithmetic circuit 10 determines that there is no defect, and the process proceeds to step S30.

[0117] In step S30, the arithmetic circuit 10 performs an update process of the database 14Z. The update process of the database 14Z is as follows: The image, attributes, and detection results of the object to be inspected that have been temporarily determined to be defect candidates are registered as conforming products in the storage device 14, and further, the offset table is updated or deleted according to whether there is a similar tendency in the image feature amount. For a specific description of the update process of the database 14Z, refer to Figure 16 the description later.

[0118] Above, Figure 11 the appearance inspection process ends.

[0119] In addition, after it is determined in step S28 that the object to be inspected is a non-conforming product, the process of the arithmetic circuit 10 ends, but it is also possible that, for example, an inspector visually determines whether there is a defect in the object to be inspected later.

[0120] 3.3 Database registration process (including similar tendency confirmation process and offset table generation / update process)

[0121] Next, refer to Figure 12A process of registering an offset table in the database 14Z of the storage device 14 will be described. Additionally, Figure 12 The process is to generate an offset table that is a prerequisite for executing Figure 11 step S30. In the present embodiment, a process of updating the database 14Z is also performed. Regarding the update process, refer to Figure 16 which will be described later.

[0122] Figure 12 is a flowchart showing the process of database registration processing. This process is executed by the arithmetic circuit 10.

[0123] In step S40, the arithmetic circuit 10 newly registers a qualified product image, attributes, and a detection result in the storage device 14. Thereby, the database 14Z is generated. The "qualified product image" refers to an image of an article determined to be a "qualified product". As already described, the article determined to be a "qualified product" is not limited to the article initially determined to be a qualified product. That is, it also includes an inspection object that was temporarily determined to be defective and then determined to be a qualified product, and an inspection object whose result of the processing of the present disclosure is determined to be a qualified product. Additionally, the "detection result" includes the detection range, defect category, and probability of the defect. For example, in the case of a rectangle, the detection range can be represented by the coordinates of the upper left corner and the lower right corner. Also, even for a qualified product, the state of over-detection varies depending on the sample, so it is registered as a separate detection result.

[0124] In step S42, the arithmetic circuit 10 performs a similar tendency confirmation process. The "similar tendency confirmation process" refers to the following process: clustering qualified product images based on the same image feature amounts of qualified product images having the same attributes, and when there is a cluster containing a specified proportion or more of qualified product images, saving the class classifier that achieved such clustering. Obtaining the class classifier means that for an image having a specific attribute and a specific image feature amount, the convolutional neural network 12n has performed over-detection. The similar tendency confirmation process refers to Figure 13 and Figure 14 which will be described later.

[0125] In step S44, the arithmetic circuit 10 determines whether there is a similar tendency based on the result of the similar tendency confirmation process performed in step S42. If there is a similar tendency, the process proceeds to step S44. If there is no similar tendency, the offset table is not generated and the process ends.

[0126] In step S46, the arithmetic circuit 10 generates an offset table and registers it in the database 14Z. The offset table is generated when over-detection is performed. After that, when it is determined that an image having the same attributes and image feature amounts is a defect, the determination criterion (threshold value) of the defect can be changed. The generation process of the offset table will be described later. Figure 15 This will be described later.

[0127] Through the above processing, when the specified conditions are met, the process of registering the offset table in the database 14Z ends. After the above processing, as Figure 3 shown, the offset table 14c is registered in the database 14Z.

[0128] Next, with reference to Figure 13 and Figure 14 the specific content of the similar tendency confirmation process will be described. The similar tendency confirmation process is as follows: When the newly registered non-defective product image in step S40 has a similar tendency in relation to the non-defective product images already registered in the database 14Z, an offset table is generated. Since the similar tendency confirmation process is performed according to the attributes of the object to be inspected, the attributes to be targeted are determined in advance when the process is executed. Figure 12 The flowchart shown in

[0129] Figure 13 is a flowchart showing the process of the similar tendency confirmation process. This process is executed by the arithmetic circuit 10.

[0130] In step S50, the arithmetic circuit 10 extracts n (n is an integer of 2 or more) of the latest non-defective product images among the non-defective product images having the target attributes from the database 14Z.

[0131] In step S52, the arithmetic circuit 10 cuts out the over-detection regions of one or more non-defective product images in which over-detection has occurred among the n extracted non-defective product images.

[0132] In step S54, the arithmetic circuit 10 extracts specified image feature amounts. The image feature amounts to be extracted are determined in advance. The image feature amounts can be, for example, feature amounts extracted by the convolutional neural network 12n, or GLCM feature amounts. The number of extracted image feature amounts is arbitrary.

[0133] In step S56, the arithmetic circuit 10 performs clustering processing using non-defective product images having the same attributes as the non-defective product images in which over-detection has occurred. As the clustering processing, for example, the K-means method can be used. Since the K-means method is a well-known technique, a specific description thereof is omitted. Other well-known clustering techniques other than the K-means method can also be used.

[0134] In step S58, the arithmetic circuit 10 determines whether the number of samples belonging to the cluster containing the over-detected qualified product images accounts for m% (m is a real number between 0 and 100, for example, 60) or more of the total number of samples. If the number of samples belonging to this cluster is m% or more of the total number of samples, it means that there is a tendency similar to the previous over-detection, and the process proceeds to step S60. If it is less than m%, it is inferred that there is no similar tendency and the over-detection occurred suddenly. Therefore, the process ends.

[0135] In step S60, the arithmetic circuit 10 saves the cluster classifier in the storage device 14.

[0136] Figure 14 It is for Figure 13 Figures that specifically illustrate steps S56 and S58. Each qualified product image is depicted using two-dimensional image feature quantities. For example, the horizontal axis is the image feature quantity related to brightness, and the vertical axis is the GLCM feature quantity related to contrast. As a result of performing clustering through a prescribed clustering technique, as Figure 14 shown, it is classified into three clusters A, B, and C. Among them, focus on cluster A.

[0137] Cluster A contains the point s obtained by depicting the image feature quantity of the image determined to be defective by the convolutional neural network 12n and multiple points a obtained by depicting the image feature quantities of each image in the group of qualified product images already registered in the database 14Z.

[0138] Among the total of n images, the image feature quantities of k images including the point s are depicted in cluster A. The arithmetic circuit 10 performs the calculation of Q = (k / n) * 100 to obtain the proportion of qualified product images classified into cluster A. Then, in Figure 12 step S58, the arithmetic circuit 10 determines whether Q≥m. When Q≥m, the image corresponding to the point s can be inferred to be over-detected in the same way as the qualified product images that had over-detection previously. Therefore, in Figure 13 step S60, the cluster classifier used for clustering is saved and used in the determination of the images of the inspected object newly performed later.

[0139] Next, refer to Figure 15 to explain the specific content of the generation / update process of the offset table. The generation / update process of the offset table is the process executed in Figure 12 step S46.

[0140] Figure 15 It is a flowchart showing the process of the generation / update process of the offset table. This process is executed by the arithmetic circuit 10.

[0141] In step S70, the arithmetic circuit 10 extracts n pieces of the latest qualified product images with the attributes of the object from the database 14Z. This process is the same as Figure 13 step S50, but "n" does not need to be the same value.

[0142] In step S72, the arithmetic circuit 10 generates an average image of the n extracted qualified product images. The pixel value at the position of the coordinate (x, y) of the average image is the average value of the pixel values of the coordinates (x, y) of the n extracted qualified product images.

[0143] In step S74, the arithmetic circuit 10 inputs the average image into the convolutional neural network 12n and generates an offset table using the output result. More specifically, the average image can be said to represent a typical qualified product image. Inputting this average image into the convolutional neural network 12n performs a defect detection process. Thus, defect candidate regions can be detected within the average image. This is because the convolutional neural network 12n also detects defect candidate regions for qualified product images with image feature amounts of points belonging to Figure 14 cluster A. The detection result may include one or more defect candidate regions. In addition, for each defect candidate region, a defect category and a probability value may also be output. The arithmetic circuit 10 generates an offset table that describes offset values (first type of offset values) corresponding to the magnitudes of the probability values in one or more partial regions. The generated offset table is, for example, the same as Figure 8 offset table 14c.

[0144] In step S76, the arithmetic circuit 10 extracts defect candidate regions for each defect category. This process means preparing an offset table for each defect category. When the offset table includes defect candidate regions, it can be said that the convolutional neural network 12n has over-detected for that defect category. Therefore, the defect candidate regions can also be referred to as "over-detected regions".

[0145] In step S78, the arithmetic circuit 10 sets the values of the regions other than the over-detected regions to 0 in the offset table for each defect category. Thus, second type of offset values are described in the offset table.

[0146] In step S80, the arithmetic circuit 10 saves the offset table for each defect category in the database. Thus, offset tables for each defect category are prepared in the database 14Z. In Figure 3 , the offset table 14c for one defect category (for example, defect category: damage) is registered in the database 14Z.

[0147] In addition, even if an offset table already exists, when new qualified product images are registered, a new offset table is generated and the existing offset table is rewritten. In such a case, Figure 15The processing can be called the update processing of the offset table.

[0148] 3.4 Database update processing

[0149] Next, with reference to Figure 16 the update processing of the database 14Z will be described. In addition, steps S40, S42, and S44 are the same as Figure 12 the processing of , so the description thereof will be omitted.

[0150] Figure 16 is a flowchart showing the process of the update processing of the database 14Z.

[0151] When it is determined in step S44 that there is a similar tendency, the process proceeds to step S96. When it is determined that there is no similar tendency, the process proceeds to step S98.

[0152] In step S96, the arithmetic circuit 10 updates the offset table. On the other hand, in step S98, the arithmetic circuit 10 deletes the offset table. The reason for deleting the offset table is to suppress the risk of impaired defect detection. If an offset is set, the risk of non-detection slightly increases. Therefore, when it is determined that there is no similar tendency, the offset table is deleted assuming no tendency of over-detection, thereby suppressing the risk of non-detection.

[0153] 3.5 Application processing of the offset table

[0154] Next, with reference to Figure 17 the specific content of the application processing of the offset table will be described. The application processing of the offset table is the processing performed in Figure 11 step S26 of .

[0155] Figure 17 is a flowchart showing the process of the application processing of the offset table. This process is executed by the arithmetic circuit 10.

[0156] In step S100, the arithmetic circuit 10 loads an offset table with the same attributes as those of the object to be inspected from the storage device. In addition, the attributes of the object to be inspected are determined according to the attribute data obtained in Figure 11 step S22 of .

[0157] In step S102, the arithmetic circuit 10 cuts out a partial area of the image of the object to be inspected. The position of the cut-out partial area corresponds to, for example, the position where the first type of offset value (non-zero value) is described in the offset table 14c ( Figure 8 ).

[0158] In step S104, the arithmetic circuit 10 inputs the image feature amount of the cut-out partial area to the cluster classifier saved through Figure 13 step S60 of .

[0159] In step S106, the arithmetic circuit 10 determines whether the image feature amount of a partial area is classified into a cluster including over-detection. The cluster including over-detection is cluster A in the example of Figure 14 . In the case where the image feature amount is classified into a cluster including over-detection, the process proceeds to step S108, and in the case where it is not classified into a cluster including over-detection, the process ends.

[0160] In step S108, the arithmetic circuit 10 adds a first type of offset value to the determination reference (threshold value) of the cut partial area. This process is equivalent to the process described with reference to Figure 9 .

[0161] Through the above processing, the threshold value of the reference table is updated to a larger value. As a result, the range determined as a qualified product is expanded, and thus over-detection can be suppressed.

[0162] 4. Modification Example

[0163] Regarding the image of the inspection object for determination, it can be obtained by the imaging device (camera) taking pictures of one or more articles in real time and the appearance inspection device 100 acquiring them. Alternatively, it can also be stored in a mass storage device after being temporarily taken, and the appearance inspection device 100 reads it out from the mass storage device to acquire it. An example of the latter will be described.

[0164] Figure 18 The structure of the appearance inspection system 1100 of the modification example is shown. The appearance inspection system 1100 includes an appearance inspection device 100 and a secondary storage device 310 connected to a communication network 300. The communication network 300 is, for example, a wide area communication network (WAN) such as the Internet or a local area communication network (LAN) laid within an enterprise or the like. In addition, an imaging device 30 is also connected to the communication network 300, but at the time point when the appearance inspection device 100 performs appearance inspection, the imaging device 30 is not an essential component of the appearance inspection system 1100.

[0165] The secondary storage device 310 is a so-called cloud storage. In this modification example, the secondary storage device 310 receives and stores the image data of the inspection object photographed and transmitted by the imaging device 30 via the communication network 300.

[0166] The interface device 22d of the appearance inspection device 100 receives the image data of the inspection object from the secondary storage device 310 via the communication network 300. The arithmetic circuit 10 acquires the image data received by the interface device 22d and stores it in the storage device 14, for example, to execute the above processing.

[0167] The appearance inspection system and computer program of the present disclosure can be appropriately applied to the appearance inspection of articles or components in manufacturing sites such as factories.

[0168] Description of Reference Numerals

[0169] 1000: Appearance inspection system; 10: Arithmetic circuit; 12: Memory; 14: Storage device; 16: Communication circuit; 18: Image processing circuit; 22a - 22c: Interface device; 30: Imaging device; 120: Input device; 130: Monitor.

Claims

1. An appearance inspection system that determines whether an object to be inspected is qualified by using an image of the object to be inspected, wherein, the appearance inspection system includes: a storage device that stores a plurality of images obtained by photographing a plurality of articles determined to be qualified products, a reference table describing a plurality of thresholds, and an offset table describing a plurality of offset values; an interface device that receives image data of the object to be inspected; and an arithmetic circuit, the image of the object to be inspected and the plurality of images each include a plurality of partial regions, each threshold of the plurality of thresholds in the reference table and each offset value of the plurality of offset values in the offset table are set corresponding to the plurality of partial regions respectively, the plurality of thresholds are respectively benchmarks for determining that the image of the object to be inspected included in the corresponding plurality of partial regions represents a defect, the plurality of offset values include a first type of offset value that changes a part of the plurality of thresholds and a second type of offset value that does not change the remaining part, the arithmetic circuit performs the following processes: Process (a), performing a predetermined defect detection process on the image data, and extracting the object to be inspected as a defect candidate when the image of the object to be inspected includes at least one defective image; Process (b), cutting out a partial image from the image of the object to be inspected extracted as the defect candidate, wherein the position of the cut partial image is the position of the partial region corresponding to the first type of offset value in the offset table; Process (c), determining whether the image feature amount of the cut partial image is classified into an over-detection category, wherein the over-detection category is a category in which the image feature amounts of the partial images of the plurality of images at the position of the partial region corresponding to the first type of offset value are classified into over-detection at a ratio of a specified value or more; Process (d), when the image feature amount of the cut partial image is classified into the over-detection category, using the first type of offset value to change the threshold in the reference table at the position of the partial region, and further raising the benchmark for determining that the image of the object to be inspected represents a defect; and Process (e), using the reference table with the changed threshold to determine whether the image of the object to be inspected includes at least one defective image.

2. The appearance inspection system according to claim 1, wherein, the interface device further receives attribute data representing an attribute related to the manufacturing conditions of the object to be inspected, the storage device stores the reference table and the offset table according to each pre-classified attribute, the process (b) includes the following processes: Process (b1), the arithmetic circuit reads out a same-attribute offset table having the same attribute as the attribute of the object to be inspected from a plurality of types of offset tables stored in the storage device according to the attribute data of the object to be inspected; and Process (b2), the arithmetic circuit cuts out a partial image from the image of the object to be inspected extracted as the defect candidate at the position of the partial region corresponding to the first type of offset value included in the read same-attribute offset table.

3. The appearance inspection system according to claim 2, wherein, the attribute related to the manufacturing conditions is at least one selected from the group consisting of a mold, a production line, and a manufacturing factory used when manufacturing the object to be inspected.

4. The appearance inspection system according to any one of claims 1 to 3, wherein, in the defect detection process performed by the process (a), the operation circuit uses the first image feature amount of the image data to detect an image of the at least one defect. When the image feature amount used in the process (c) is set as the second image feature amount, the first image feature amount and the second image feature amount are the same.

5. The appearance inspection system according to any one of claims 1 to 3, wherein, the image feature amount in the process (c) is an image feature amount related to a gray-level co-occurrence matrix of the image.

6. The appearance inspection system according to any one of claims 1 to 3, wherein, the defect detection process in the process (a) is a process for detecting multiple types of defects, and the process (a) includes a process of extracting the defect candidates according to the types of the at least one detected defect.

7. The appearance inspection system according to claim 6, wherein, the storage device stores the reference table and the offset table for each type of the defect, and the process (b) includes the following processes: Process (b3), the operation circuit reads out the offset table of the same type corresponding to the type of the at least one defect from the multiple types of offset tables stored in the storage device; and Process (b4), the operation circuit cuts out a partial image from the image of the object to be inspected extracted as the defect candidate at the position of the partial area corresponding to the first type of offset value included in the read offset table of the same type.

8. The appearance inspection system according to any one of claims 1 to 3, wherein, the multiple partial areas are areas each represented by one pixel, and each threshold of the multiple thresholds of the reference table and each offset value of the multiple offset values of the offset table are set corresponding to each pixel.

9. The appearance inspection system according to any one of claims 1 to 3, wherein, the appearance inspection system further includes the following process (f): when an image of the object to be inspected in the process (e) does not include an image of the at least one defect, the image of the object to be inspected is additionally stored in the storage device as the multiple images obtained by separately photographing the multiple articles determined to be qualified products.

10. The appearance inspection system according to claim 9, wherein, the appearance inspection system further includes the following process (g): after the process (f), the offset table stored in the storage device is rewritten as an offset table in which the second type of offset value is changed.

11. The appearance inspection system according to claim 10, wherein, the process (f) includes the following processes: Process (f1), determining whether there is a similar over-detection result; Process (f2), in a case where the determination result indicates the existence of similar over-detection results, rewrite the second type of offset value of the offset table. The process (f1) is as follows: In a case where the image feature amounts of the partial images of the plurality of images at the position of the partial region corresponding to the first type of offset value are classified into the over-detection category at a ratio equal to or higher than the specified ratio, it is determined that there are similar over-detection results.

12. The appearance inspection system according to claim 11, wherein, The process (f1) is as follows: In a case where a predetermined clustering process is performed on the image feature amounts of the partial images of the plurality of images at the position of the partial region corresponding to the first type of offset value and the execution result indicates that they are classified into the same cluster at a ratio equal to or higher than the specified ratio, it is determined that there are similar over-detection results.

13. The appearance inspection system according to claim 12, wherein, This appearance inspection system further has a category classifier used in the predetermined clustering process. The process (c) is a process of using the category classifier to determine whether the image feature amount of the cut partial image is classified into the over-detection category.

14. The appearance inspection system according to claim 11, wherein, The process (f) includes the following process (f3): In a case where the determination result indicates the non-existence of similar over-detection results, discard the offset table in which the second type of offset value has been changed.

15. The appearance inspection system according to any one of claims 10 to 14, wherein, The interface device also receives attribute data indicating an attribute related to the manufacturing conditions of the object to be inspected. The storage device stores the plurality of images obtained by respectively photographing the plurality of objects determined to be qualified products, the reference table, and the offset table in association with pre-classified attributes. The process (g) includes the following processes: Process (g1), select a group of images having the same attribute as the attribute data received by the interface device from the plurality of images stored in the storage device; Process (g2), generate at least one reference image based on the group of images; Process (g3), perform the predetermined defect detection process on the image data of the at least one reference image. In a case where it is determined that the at least one reference image contains one or more defect images, generate an over-detection table associating a region indicating the position of each defect image with a value indicating the degree of over-detection; Process (g4), in the over-detection table, a first value indicating the degree of over-detection occurring at the position of each defect image is described, and a second value indicating that no over-detection occurs at positions other than the position of each defect image is also described; and Process (g5), use the over-detection table to rewrite the offset table stored in the storage device into an offset table in which the second type of offset value has been changed.

16. The appearance inspection system according to any one of claims 1 to 3, wherein, This appearance inspection system further has an imaging device. The imaging device captures the inspected object to obtain the image data and outputs it.

17. The appearance inspection system according to claim 16, wherein, the imaging device sends the image data of the inspected object to the interface device, and the arithmetic circuit obtains the image data received by the interface device.

18. The appearance inspection system according to claim 16, wherein, the interface device receives the image data of the inspected object from the imaging device, the storage device stores the received image data, and the arithmetic circuit obtains the image data of the inspected object from the storage device.

19. The appearance inspection system according to claim 16, wherein, the appearance inspection system further has a secondary storage device connected to a communication network, the imaging device sends the image data of the inspected object to the secondary storage device via the communication network, the secondary storage device stores the received image data, the interface device receives the image data of the inspected object from the secondary storage device via the communication network, and the arithmetic circuit obtains the image data received by the interface device.

20. The appearance inspection system according to any one of claims 1 to 3, wherein, the predetermined defect detection process in the process (a) is a process of using a learned neural network to determine whether the image of the inspected object contains an image of the at least one defect, and the neural network is constructed by a machine learning process in which images of respective articles of a plurality of articles including qualified products and unqualified products and pass / fail data indicating whether each of the articles is a qualified product or an unqualified product are used as teacher data.

21. The appearance inspection system according to claim 20, wherein, the neural network is constructed by a machine learning process in which images of respective articles of a plurality of articles including qualified products and unqualified products, pass / fail data indicating whether each of the articles is a qualified product or an unqualified product, and defect type data indicating types of defects of the unqualified products are used as teacher data.

22. A computer program product having a computer program recorded thereon, the computer program being executed by an arithmetic circuit of an appearance inspection system that determines whether an inspected object is qualified by using an image of the inspected object, wherein, the appearance inspection system includes: a storage device that stores a plurality of images respectively obtained by photographing a plurality of articles determined to be qualified products, a reference table describing a plurality of thresholds, and an offset table describing a plurality of offset values; an interface device that receives image data of the inspected object; and the arithmetic circuit, the image of the inspected object and the plurality of images each include a plurality of partial regions, each of the plurality of thresholds in the reference table and each of the plurality of offset values in the offset table are set corresponding to the plurality of partial regions respectively, and the plurality of thresholds are respectively benchmarks for determining that the image of the inspected object included in the corresponding plurality of partial regions represents a defect. The plurality of offset values include a first type of offset value that causes a part of the plurality of thresholds to change and a second type of offset value that does not cause the remaining part to change. The computer program causes the arithmetic circuit to perform the following processing: Processing (a): performing a predetermined defect detection process on the image data, and extracting the object to be inspected as a defect candidate when an image of at least one defect is included in the image of the object to be inspected; Processing (b): cutting out a partial image from the image of the object to be inspected extracted as the defect candidate, wherein the position of the cut partial image is the position of the partial area corresponding to the first type of offset value in the offset table; Processing (c): determining whether the image feature amount of the cut partial image is classified into an over-detection category, wherein the over-detection category is a category in which the image feature amounts of the partial images of the plurality of images at the position of the partial area corresponding to the first type of offset value are classified as over-detected at a ratio equal to or higher than a specified value; Processing (d): when the image feature amount of the cut partial image is classified into the over-detection category, using the first type of offset value to change the threshold value of the reference table at the position of the partial area, and further increasing the reference for determining that the image of the object to be inspected represents a defect; and Processing (e): determining whether an image of at least one defect is included in the image of the object to be inspected by using the reference table with the changed threshold value.

Images