AI-based product surface inspection device and method

Abstract

An AI-based product surface inspection device (100), comprising: a sensor unit (110; 210) that photographs a product to generate image data and measures at least one of a color, a saturation, a brightness, a transparency, and a reflectance of the product; and a detection unit (120; 230) that detects a defect on the product by inputting the image data to a convolutional neural network (CNN) trained to detect a defect on a product surface, wherein the number of convolutional layers (231) of the convolutional neural network is determined based on a defect detection difficulty determined from at least one of the color, saturation, brightness, transparency, and reflectance of the product and a defect type.

Description

Technical FieldThe present disclosure relates to a technique for inspecting a surface of a product, and more particularly to an apparatus and method for product surface inspection that detects a defect on products having different features using a previously trained artificial neural network.Prior ArtRecently, studies on a method for detecting product defects by image analysis using an artificial neural network have been made. However, the artificial neural network requires a large amount of training data for image analysis regardless of a learning method. However, in many manufacturing environments, it is difficult to obtain enough training data to train an artificial neural network.Moreover, the performance of the artificial neural network varies depending on training data, so sophisticated training data is required. When the training data classified by faulty standards at the manufacturing site is used, the performance of the artificial neural network may be significantly degraded.Accordingly, it is necessary to develop an apparatus that recognizes a defect of the product having a feature different from the training data at different manufacturing locations using an artificial neural network that learns various defect types occurring on a product surface from training data precisely designed in advance.DisclosureTechnical ProblemAn object is to provide a product surface inspection apparatus and method that recognize a defect on products having different characteristics using a previously trained artificial neural network.Technical SolutionAccording to one aspect, an apparatus for inspecting a product surface includes: a sensor unit that photographs a product to generate image data and measures at least one of a color, saturation, brightness, transparency, and reflectance of the product; and a recognition unit that recognizes a defect on a product by inputting the image data to a convolutional neural network trained to recognize a defect on a product surface, wherein the number of convolutional layers of the convolutional neural network can be determined based on at least one of the color, saturation, brightness, transparency, and reflectance of the product.The convolutional neural network may perform the learning by receiving a predetermined size of training image data for a first defect type to split the training image data into rasters having a predetermined size.The training image data is configured by a three-channel image obtained by dividing an image for the same product with respect to RGB (Red-Green-Blue color space), and a size of the training data is 448× 448 and a size of the raster is 7× 7. The convolutional neural network may perform learning as many times as the number obtained by dividing a size of training image data by a size of the raster.The convolutional neural network may learn N defect types.The number of convolutional layers of the convolutional neural network may be determined based on a similarity between at least one of the color, saturation, brightness, transparency, and reflectance of the product used as the training data of the convolutional neural network and at least one of the color, saturation, brightness, transparency, and reflectance of the product.The number of convolutional layers of the convolutional neural network can be increased inversely proportional to the similarity.The product surface inspection device may further include: a preprocessor that converts the image data based on at least one of the color, saturation, brightness, transparency, and reflectance of the product to input the converted data to the recognition unit.The preprocessor may perform automatic cropping to extract a shape of the product from the image data to extract at least one feature of a brightness and a shadow of the product based on a shape of the automatically cropped product.The preprocessor may convert at least one of the color, saturation, and brightness of the image data with respect to at least one of the color, saturation, and brightness of the product used as the training data of the convolutional neural network.The preprocessor may determine a frequency of an image sharpening filter based on transparency of the product and convert the image data by applying the image sharpening filter to the image data.The recognition unit may recognize a position of a defect, a size of the defect, and a type of the defect on the product, and when a predetermined number or more of defects of the same position, the same size, and the same type sequentially occur, it may be determined that the defect is not a defect.The recognition unit may display a bounding box on the image data based on the position of the defect and the size of the defect present on the product, and output image data displayed with the bounding box and the type of the defect.In one aspect, a product surface inspection method may include: generating image data by photographing a product; measuring at least one of a color, saturation, brightness, transparency, and reflectance of the product; and detecting a defect on the product by inputting the image data to a convolutional neural network trained to detect the defect on a product surface, wherein the number of convolutional layers of the convolutional neural network is determined based on at least one of a color, saturation, brightness, transparency, and reflectance of the product.Advantageous EffectsThe defect on products having different characteristics at different production sites can be effectively and quickly detected using a pre-trained artificial neural network specified for product surface inspection with a small amount of image data.DESCRIPTION OF THE DRAWINGSFIG. 1 is a configuration diagram of an AI-based product surface inspection device according to an exemplary embodiment. FIG. 2 is a configuration diagram of an AI-based product surface inspection device according to an exemplary embodiment. FIG. 3 is an exemplary view for explaining an operation of an AI-based product surface inspection device according to an exemplary embodiment. FIG. 4 is a flow chart illustrating an AI-based method for product surface inspection according to an exemplary embodiment.Best ModeHereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the exemplary embodiment of the present disclosure, a detailed description of known configurations or functions included herein will be omitted when it is determined that the detailed description may obscure the subject matter of the present disclosure. Moreover, the terms used in the specification are defined in consideration of the functions of the present disclosure, and may vary depending on the intention or usual operation of a user or operator. Accordingly, the terms in this entire specification must be defined based on details.Exemplary embodiments of an apparatus and a method for product surface inspection are described in detail below with reference to drawings. Hereinafter, the AI-based apparatus and method for product surface inspection may be abbreviated as a product surface inspection apparatus and a product surface inspection method, respectively.FIG. 1 is a configuration diagram of a product surface inspection device according to an exemplary embodiment.Referring to FIG. 1, the product surface inspection apparatus 100 may include a sensor unit 110 and a recognition unit 120.According to an exemplary embodiment, the sensor unit 110 may photograph a product to generate image data. For example, the sensor unit 110 may include a camera sensor that photographs a product.According to the exemplary embodiment, the sensor unit 110 may measure at least one of a color, saturation, brightness, transparency, and reflectance of the product. For example, the sensor unit 110 may include a separate sensor to measure at least one of a color, saturation, brightness, transparency, and reflectance, or analyze at least one of the color, saturation, brightness, transparency, and reflectance by analyzing image data.According to the exemplary embodiment, the recognition unit 120 inputs image data to a convolutional neural network (CNN) trained to recognize a defect on a product surface to recognize a defect present on the product. According to an exemplary embodiment, the convolutional neural network may be configured by a plurality of convolutional layers that execute a function of extracting a feature from a shaped product.According to the exemplary embodiment, the convolutional neural network may perform learning by receiving a predetermined size of training image data for a first defect type to split the training image data into rasters having a predetermined size. For example, the training image data is configured by a three-channel image obtained by dividing an image for the same product with respect to the RGB, and a size of the training data is 448× 448 and a size of the raster is 7× 7.According to the exemplary embodiment, the convolutional neural network for product surface inspection may be trained by dividing layers to extract a feature of the defect based on the image data obtained by photographing the product. At this time, a width and a height of a defective part in the image data are decreased and a channel is increased the deeper the layer is in the convolutional neural network.According to the exemplary embodiment, the convolutional neural network may perform learning as many times as the number obtained by dividing a size of training image data by a size of the raster. For example, when it is assumed that a pixel size of image data including a defect is 448× 448×3 (width×height×channel) and a pixel size of a defect image is 7×7, the convolutional neural network can learn 1024×4 times different features for each channel.According to an exemplary embodiment, convolutional neural network N may learn defect types.For example, the defect type of the product may be a short shot, a black stripe, a crack, a burr (a seam), a flow mask, a spatter, a sink mark, a silver stripe, a deformation, a weld, a cloudy surface, or a delamination.MODE FOR CARRYING OUT THE INVENTIONReferring to FIG. 2, a recognition unit 230 may include a plurality of convolutional layers 231 and a plurality of fully connected layers.According to the exemplary embodiment, the number of convolutional layers of the convolutional neural network may be determined based on at least one of a color, saturation, brightness, transparency, and reflectance of the product. Depending on the color, saturation, brightness, transparency, or reflectance of the product, the difficulty in detecting a defect on the product surface may be worthwhile. For example, a silver stripe is easily seen on the product surface in a dark-colored and opaque product, while a silver stripe is less likely to be incident on the product surface in a light-colored and transparent product. Accordingly, the number of convolutional layers can be adjusted in consideration of a defect recognition difficulty or a processing speed of the convolutional neural network.According to the exemplary embodiment, the convolutional neural network may be trained using training data about a predetermined product, and a product on which actual recognition is performed may have a different feature from a product of the training data. Accordingly, the configuration of the convolutional neural network can be adjusted by considering a feature of a product on which a defect is detected.For example, the number of convolutional layers of the convolutional neural network may be determined based on a similarity between at least one of a color, saturation, brightness, transparency, and reflectance of the product used as the training data of the convolutional neural network and at least one of a color, saturation, brightness, transparency, and reflectance of a product.For example, if the feature of the product on which the recognition is performed resembles a product of the training data used to train the convolutional neural network, the convolutional neural network may recognize a defect of the product with a relatively higher probability. On the other hand, when the feature of the product on which the recognition is performed is different from the product of the training data used for training the convolutional neural network, the convolutional neural network can recognize a defect of the product with a relatively lower probability. Accordingly, the number of convolutional layers configuring the convolutional neural network may be adjusted by comparing features of a product on which the defect is detected and the product of the training data used to train the convolutional neural network.According to the exemplary embodiment, the number of convolutional layers of the convolutional neural network may be increased inversely proportional to the similarity. For example, when the similarity is high, the probability of recognition of the defect can be relatively increased, so that the number of convolutional layers of the convolutional neural network is decreased to increase the processing efficiency. On the other hand, when the similarity is low, the probability of recognition of the defect is relatively decreased, so that the number of convolutional layers of the convolutional neural network is increased to improve recognition performance.According to the exemplary embodiment, the product surface inspection device may further include a preprocessor that converts image data based on at least one of the color, saturation, brightness, transparency, and reflectance of the product to input the converted data to the recognition unit.Referring to FIG. 2, the preprocessor 220 may receive and convert image data generated in the sensor unit 210 to transmit the converted data to the recognition unit 230. To this end, the preprocessor 220 may receive information on at least one of the color, the saturation, and the brightness of the product from the sensor unit 210. Alternatively, the preprocessor 220 may analyze the input image data to analyze at least one of the color, saturation, and brightness of the product.According to the exemplary embodiment, the preprocessor 220 may perform automatic cropping to extract a shape of the product from the image data to extract at least one feature of a brightness and a shadow of the product based on a shape of the automatically cropped product.According to the exemplary embodiment, the preprocessor 220 may convert at least one of the color, saturation, and brightness of the image data with respect to at least one of the color, saturation, and brightness of the product used as the training data of the convolutional neural network.For example, if the feature of the product from which the defect is detected and the feature of the product of the training data used to train the convolutional neural network are different, the performance of the detection unit 230 may be degraded. Accordingly, the preprocessor 220 may convert at least one of the color, saturation, and brightness of the product with respect to at least one of the color, saturation, and brightness of the product of the training data. For example, if it is assumed that the probability of defect recognition of the product is decreased at high brightness, the preprocessor 220 may generate a converted image by decreasing the brightness of the image data.According to the exemplary embodiment, the preprocessor 220 may determine a frequency of an image sharpening filter based on transparency of the product and convert the image data by applying the image sharpening filter to the image data.For example, in a transparent product, a spot on the product surface cannot be easily distinguished from a normal part. Accordingly, in order to emphasize a fine difference, a high-frequency feature of the image data needs to be emphasized. To this end, the preprocessor 220 may further emphasize the high-frequency power of the image data using the sharpening filter. For example, the sharpening filter may be expressed as a high frequency pass filter or a high frequency emphasis filter.According to an exemplary embodiment, the recognition unit 120 recognizes a position of a defect, a size of the defect, and a type of the defect present on the product. When a predetermined number or more of defects of the same position, size, and type continuously occur, it can be determined that the defect is not a defect.According to the exemplary embodiment, the recognition unit 120 may display a bounding box on the image data based on the position of the defect and the size of the defect present in the product, and output image data displayed with the bounding box and the type of the defect. For this purpose, the product surface inspection device may further comprise an interface for outputting image data.Referring to FIG. 3A, when the sensor unit 110 generates image data of the product to recognize the defect of the product, illumination light may be reflected on the product or other surrounding objects may be reflected. In particular, in the case of a glossy product with a smooth surface, the influence of the reflection can be considerable.For example, the recognition unit 120 may erroneously recognize a portion such as a spot as a defect due to the reflection. Accordingly, when the position of the defect, the size, and the type are continuously repeated, the recognition unit 120 may determine that the defect has been generated due to the external influence. In this case, the recognition unit 120 may exclude the defect determined as a defect due to the external influence from the recognition.Referring to FIG. 3B, when the sensor unit 110 generates image data of the product to recognize the defect of the product, light passing through a transparent product or an image of another surrounding object may be photographed. Particularly in the case of a product having high transparency, the influence of transmission of a background may be considerable.For example, the recognition unit 120 may erroneously recognize a portion such as an object on the background and a spot as a defect. Accordingly, when the position of the defect, the size, and the type are continuously repeated, the recognition unit 120 may determine that the defect has been caused by the external influence. In this case, the recognition unit 120 may exclude the defect determined as a defect due to the external influence from the recognition.FIG. 4 is a flowchart illustrating a product surface inspection method according to an exemplary embodiment.According to an exemplary embodiment, the product surface inspection device may photograph a product to generate image data in step 410. For example, the product surface inspection device may include a camera sensor that photographs a product.According to the exemplary embodiment, the product surface inspection device may measure at least one of a color, saturation, brightness, transparency, and reflectance of the product in step 420.According to the exemplary embodiment, the product surface inspection device may measure at least one of a color, saturation, brightness, transparency, and reflectance of the product. For example, the product surface inspection apparatus may include a separate sensor to measure at least one of the color, saturation, brightness, transparency, and reflectance, or to analyze at least one of the color, saturation, brightness, transparency, and reflectance by analyzing image data.According to the exemplary embodiment, the product surface inspection device may input image data to a convolutional neural network (CNN) trained to detect a defect on a product surface to detect a defect present on the product in step 430.According to an exemplary embodiment, the convolutional neural network is configured by a plurality of convolutional layers that perform a function of extracting a feature from a shaped product and a plurality of fully connected layers.According to the exemplary embodiment, the number of convolutional layers of the convolutional neural network may be determined based on at least one of a color, saturation, brightness, transparency, and reflectance of the product. For example, the number of convolutional layers of the convolutional neural network may be determined based on a similarity between at least one of the color, saturation, brightness, transparency, and reflectance of the product used as the training data of the convolutional neural network and at least one of the color, saturation, brightness, transparency, and reflectance of the product.According to the exemplary embodiment, the product surface inspection device may convert image data based on at least one of a color, saturation, brightness, transparency, and reflectance of the product. For example, the product surface inspection device may convert at least one of a color, saturation, and brightness of the image data with respect to at least one of the color, saturation, and brightness of the product used as the training data of the convolutional neural network. As another example, the product surface inspection device may determine a frequency of an image sharpening filter based on transparency of the product and convert the image data by applying the image sharpening filter to the image data.According to an exemplary embodiment, the product surface inspection apparatus may recognize a position of a defect, a size of the defect, and a type of the defect on the product. When a predetermined number or more of defects of the same position, size, and type occur continuously, it can be determined that the defect is not a defect.An aspect of the present disclosure may also be implemented as computer readable code written on a computer readable recording medium. Code and code segments implementing the program can be easily derived by a computer programmer in the art. The computer readable recording medium may include any type of recording device that stores data readable by a computer system. Examples of the computer readable recording medium may include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, and the like. The computer readable recording medium is distributed in computer systems connected via a network to be written and executed in a distributed manner with computer readable code.Heretofore, the present disclosure has been described with reference to the exemplary embodiments. It will be apparent to those skilled in the art that the present disclosure may be practiced in modified form without departing from a significant feature of the present disclosure. Accordingly, the scope of the present disclosure is not limited to the above-described embodiment, but should be construed to include various embodiments within the scope corresponding to the description of the claims.Industrial applicabilityThe present disclosure is applicable to the industry of process automation.

Claims

A AI-based product surface inspection apparatus (100) comprising: a sensor unit (110; 210) that photographs a product to generate image data and measures at least one of a color, saturation, brightness, transparency, and reflectance of the product; and a recognition unit (120; 230) that recognizes a defect on the product by inputting the image data to a convolutional neural network, CNN, trained to recognize a defect on a product surface, wherein the number of convolutional layers (231) of the convolutional neural network is determined based on a defect recognition difficulty determined based on at least one of the color, saturation, brightness, transparency, and reflectance of the product and a defect type.The AI-based product surface inspection apparatus (100) according to claim 1, wherein the convolutional neural network performs a learning operation by receiving a predetermined size of training image data for a first defect type to split the training image data into rasters having a predetermined size.The AI-based product surface inspection apparatus (100) according to claim 2, wherein the training image data is configured by a three-channel image obtained by dividing an image for the same product with respect to RGB, and a size of the training data is 448 × 448, and a size of the raster is 7 × 7.The AI-based product surface inspection apparatus (100) according to claim 2, wherein the convolutional neural network performs the learning as many times as the number obtained by dividing the size of training image data by the size of the raster.The AI-based product surface inspection apparatus (100) according to claim 2, wherein the convolutional neural network learns N defect types.The AI-based product surface inspection apparatus (100) according to claim 1, wherein the number of convolutional layers (231) of the convolutional neural network is further determined based on a similarity between at least one of the color, saturation, brightness, transparency, and reflectance of the product used as the training data of the convolutional neural network and at least one of the color, saturation, brightness, transparency, and reflectance of the product.The AI-based product surface inspection apparatus (100) according to claim 6, wherein the number of convolutional layers (231) of the convolutional neural network is increased inversely in proportion to the similarity.The AI-based product surface inspection device (100) according to claim 1, further comprising: a preprocessor (220) that converts the image data based on at least one of the color, the saturation, the brightness, the transparency, and the reflectance of the product to input the converted data to the recognition unit (120; 230).The AI-based product surface inspection apparatus (100) according to claim 8, wherein the preprocessor (220) performs automatic cropping to extract a shape of the product from the image data to extract at least one feature of brightness and shadow of the product based on the shape of the automatically cropped product.The AI-based product surface inspection device (100) according to claim 8, wherein the preprocessor (220) converts at least one of the color, saturation, and brightness of the image data with respect to at least one of the color, saturation, and brightness of the product used as training data of the convolutional neural network.The AI-based product surface inspection apparatus (100) according to claim 8, wherein the preprocessor (220) determines a frequency of an image sharpening filter based on the transparency of the product and converts the image data by applying the image sharpening filter to the image data.The AI-based product surface inspection apparatus (100) according to claim 1, wherein the recognition unit (120; 230) recognizes a position of the defect, a size of the defect, and a type of the defect on the product, and when a predetermined number or more of defects of the same position, the same size, and the same type sequentially occur, it is determined that the defect is not a defect.The AI-based product surface inspection apparatus (100) according to claim 12, wherein the recognition unit (120; 230) displays a bounding box on the image data based on the position of the defect and the size of the defect present on the product, and outputs the image data displayed with the bounding box and the type of the defect.A AI-based product surface inspection method comprising: generating (410) image data by photographing a product; measuring (420) at least one of a color, saturation, brightness, transparency, and reflectance of the product; and detecting (430) a defect on the product by inputting the image data to a convolutional neural network trained to detect the defect on a product surface, wherein the number of convolutional layers of the convolutional neural network is determined based on a defect detection difficulty determined from at least one of the color, saturation, brightness, transparency, and reflectance of the product and a defect type.

Images