Surface defect detection method based on perceptual auto-encoder, and related apparatus

Abstract

The present invention belongs to the field of integrated circuit defect detection. Provided is a surface defect detection method based on a perceptual auto-encoder. In the present invention, pixel-wise loss is replaced with perceptual loss to solve the problems of blurred output images and severe edge distortion caused by single-pixel loss; in addition, the method has a strong transferability and can thus be well transferred to a new target to be detected. Moreover, in the present invention, a sliding window method is replaced with a feature pyramid, by means of which it is not necessary to perform a large amount of redundant computation by sliding on an original image, thereby effectively reducing memory and computation costs. Provided in the present invention is a surface defect detection apparatus based on a perceptual auto-encoder, the apparatus including specific modules for implementing an operating method. Provided in the present invention are a computer device and a storage medium for a surface defect detection method based on a perceptual auto-encoder, the computer device and the storage medium being used for implementing the specific steps of the operating method.

Description

A surface defect detection method and related device based on perceptual autoencoder Technical Field

[0001] The present invention belongs to the field of integrated circuit defect detection, and in particular to a surface defect detection method based on a perceptual autoencoder and a related device. Background Art

[0002] In the field of automated defect detection technology for military circuits, machine vision methods are generally used for defect detection. Based on the different feature extraction methods, it is divided into traditional appearance detection methods and intelligent detection methods. Among the existing traditional automatic detection methods, detection mainly relies on specific optical design. Typical methods include structured light defect detection and design of light source reflection surface mark detection. These methods have poor portability and require independent design for different products. Existing intelligent detection methods can be divided into two categories based on whether supervision is required: methods that require supervision are more accurate, but require a large amount of labeled product data and cannot adapt to industrial detection scenarios; unsupervised methods do not require independent design or manual labeling and can adapt to the application scenarios of military circuit surface defect detection.

[0003] To improve the transferability of inspection across different products, AutoEncoder (AE) was employed. AE (AutoEncoder) is an unsupervised technology based on a backpropagation algorithm and optimization methods. It uses the input data itself as supervision to guide a neural network in learning a mapping relationship. As a novel feature extraction method, AE can perform both feature extraction and image generation in an unsupervised manner. Therefore, AE's feature extraction capabilities can be leveraged to extract features of defect-free products, generate templates, and compare them. This allows for rapid migration to different military circuit surface inspection scenarios, significantly improving the versatility of surface inspection methods.

[0004] Currently, AE applied in other fields generally adopts pixel-by-pixel loss as the loss function, such as MSE (mean square error). The information involved in pixel-by-pixel loss is the information of a single pixel, ignoring the structural information across pixels, which is not conducive to the surface detection of military circuits. In this process, due to the characteristics of pixel-by-pixel loss, the network tends to output blurred images, resulting in severe edge distortion, and poor detection effect for products with more edges such as military circuits. At the same time, pixel-by-pixel loss treats each pixel equally, and in the military circuit detection scenario, pixels representing defects are far more important than non-defective pixels, which makes pixel-by-pixel loss difficult to adapt to the military circuit surface defect detection scenario. If AE uses perceptual loss, it can reduce the loss of edge information, thereby increasing the similarity of the synthesized template and increasing the accuracy of detection.

[0005] Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a surface defect detection method based on perceptual autoencoder and related devices.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A surface defect detection method based on a perceptual autoencoder comprises the following steps:

[0009] (1) Constructing an encoder-decoder to form an unsupervised perceptual autoencoder network, wherein the encoder contains a feature pyramid;

[0010] Inputting a defect-free sample image into the encoder, the encoder extracts features of different scales from the input image, and then performs convolution on the features to obtain a feature map. The decoder performs a deconvolution operation on the feature map to output an initial reconstructed image;

[0011] (2) Inputting the initial reconstructed image and the original image into the perceptual computing network, the perceptual computing network calculates feature maps of the two respectively, calculates perceptual loss based on the feature maps of the two, and optimizes the perceptual autoencoder network based on the perceptual loss and the back propagation algorithm until the perceptual loss reaches a minimum value, thereby completing the training of the perceptual autoencoder network;

[0012] (3) Collect the image to be tested of the military product circuit to be tested, input the image to be tested into the trained perceptual autoencoder network, and output a defect-free output image as a template image;

[0013] Then, the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops. At this point, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region.

[0014] If the structural similarity of the region is greater than the preset similarity threshold, no output is given, indicating that there is no defect.

[0015] Furthermore, in step (1), the sample image is a grayscale image or a color image of a military circuit.

[0016] Furthermore, in step (2), the perceptual loss is the mean square error between the initial reconstructed image feature map and the original image feature map.

[0017] Furthermore, in step (2), the functional expression of the perceptual loss is:

[0018] Where, Loss perrepresents the perceptual loss, ρ represents the perceptual computing network, X is the input image, which contains n components after vectorization, a is the self-encoder operation, X k is the kth component of X, a(X) is the reconstructed version of the autoencoder output, a(X) k is the kth component after vectorization of the remodeled version, ρ(X k ),ρ(a(X) k ) are the input image and reconstructed pattern feature vector extracted using the perception network, respectively.

[0019] Furthermore, in step (3), the similarity threshold is preset to 0.2;

[0020] The preset minimum accuracy is 25 pixels.

[0021] Furthermore, in step (3), the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm, and the specific operation is as follows:

[0022] (301), dividing the template image and the image to be detected into four images, and calculating the structural similarity between the segmented regions and the corresponding positions of the template image;

[0023] (302) selecting a region whose regional structural similarity is lower than a preset similarity threshold, and recursively performing quadtree segmentation on the region;

[0024] (303), compare the area of ​​the segmented region with the preset minimum precision, if it is greater, go to (302);

[0025] If the area of ​​the region is less than the preset minimum precision, go to (304);

[0026] (304), stop segmentation, compare the structural similarity of the segmented region with the preset similarity threshold; if it is less than or equal to, output the region as a defective region;

[0027] If it is greater, there is no output, indicating there is no defect.

[0028] A surface defect detection system based on a perceptual autoencoder, comprising a perceptual autoencoder network building module, a training module, and a detection module;

[0029] The perceptual autoencoder network construction module is used to construct an encoder-decoder to form an unsupervised perceptual autoencoder network, and the encoder contains a feature pyramid;

[0030] Inputting a defect-free sample image into the encoder, the encoder extracts features of different scales from the input image, and then performs convolution on the features to obtain a feature map. The decoder performs a deconvolution operation on the feature map to output an initial reconstructed image;

[0031] The training module is configured to input the initial reconstructed image and the original image into a perceptual computing network, the perceptual computing network calculating feature maps of the two images respectively, calculating a perceptual loss based on the feature maps of the two images, and optimizing the perceptual autoencoder network based on the perceptual loss and a back-propagation algorithm until the perceptual loss reaches a minimum value, thereby completing the training of the perceptual autoencoder network;

[0032] The detection module is used to collect an image of a military product circuit to be detected, input the image to be detected into a trained perceptual autoencoder network, and output a defect-free output image as a template image;

[0033] Then, the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops. At this point, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region.

[0034] If the structural similarity of the region is greater than the preset similarity threshold, no output is given, indicating that there is no defect.

[0035] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the surface defect detection method based on perceptual autoencoder of the present invention are implemented.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the surface defect detection method based on perceptual autoencoder described in the present invention.

[0037] Beneficial effects of the present invention

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This paper proposes a surface defect detection method based on a perceptual autoencoder. This method uses perceptual loss instead of pixel-by-pixel loss, resolving the issues of blurred output images and severe edge distortion associated with single-pixel loss. Furthermore, this method is highly transferable and can be readily adapted to new inspection targets. Furthermore, the present invention uses a feature pyramid instead of a sliding window approach. This eliminates the need for redundant computations by sliding the feature pyramid over the original image, effectively reducing memory and computational costs.

[0040] The present invention provides a surface defect detection device based on a perceptual autoencoder, which includes a specific module for completing the above-mentioned working method.

[0041] The present invention provides a computer device and storage medium for a surface defect detection method based on a perceptual autoencoder, which are used to implement the specific steps of the above working method. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of an autoencoder network;

[0043] FIG2 is a schematic diagram of the training process of the perceptual autoencoder of the present invention;

[0044] FIG3 is a schematic diagram of the detection process of the perceptual autoencoder of the present invention;

[0045] Figure 4 is a schematic diagram of the quadtree algorithm detection process;

[0046] FIG5 is a schematic diagram of an input and generated template according to an embodiment. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0048] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] The present invention is described in further detail below with reference to the accompanying drawings:

[0050] Example 1

[0051] The specific scenario targeted by the present invention is: performing surface defect detection on manufactured military circuits during the processing and manufacturing of military circuits.

[0052] The present invention provides a surface defect detection method based on a perceptual autoencoder, comprising the following steps:

[0053] Step 1: Build an unsupervised perceptual autoencoder network, including a feature pyramid:

[0054] Specifically, an unsupervised perceptual autoencoder network is constructed. (See Figure 1.) The basic architecture of an unsupervised perceptual autoencoder network is an encoder-decoder. The encoder performs convolution on the preprocessed input image to extract features, generating a feature map. The decoder then performs deconvolution on the feature map to reconstruct the image. It should be noted that the network outputs feature maps of different depths, forming a feature pyramid. Deconvolution is performed simultaneously on these feature maps at different depths to produce multiple output images. The purpose of this step is to build an unsupervised perceptual autoencoder network and obtain the initial reconstructed image.

[0055] The present invention uses a feature pyramid instead of the sliding window method, which does not require the sliding window method to perform a large amount of redundant calculations, thus saving computational costs. Furthermore, it does not require labeling of the input image, thus saving a lot of manpower.

[0056] Step 2: Use the reconstructed image to calculate the perceptual loss and optimize the perceptual autoencoder network:

[0057] The reconstructed image obtained in step 1 is used to calculate the perceptual loss. Specifically, the reconstructed image and the original image are input into a pre-trained perceptual computing network. The perceptual computing network calculates the perceptual loss of both and uses an optimization algorithm to optimize until the perceptual loss is minimized.

[0058] First, a grayscale image of the surface of a non-defective military circuit is captured and recorded as the original image. This original image is then fed into a perceptual autoencoder network, which outputs an initial reconstructed image. Both the original image and the initial reconstructed image are fed into a perceptual computing network to obtain a perceptual loss. This perceptual loss is calculated by feeding the original image and the initial reconstructed image into a pre-trained perceptual computing network, which computes feature images for both. The mean squared error (MSE) between these two feature images is then calculated to obtain the perceptual loss.

[0059] Specifically, in the initial reconstructed image OUTPUT of size W×H×N output by the perceptual autoencoder network, OUTPUT corresponds to the original image INPUT one-to-one at the pixel level. In this embodiment, both the original image and the initial reconstructed image are 512×512×1. The perceptual computing network is used to calculate the feature maps of the original image and the initial reconstructed image respectively through convolution. Each pixel in the feature map of the original image corresponds to multiple pixels of the original image. These multiple pixels are called receptive fields, and the same is true for the initial reconstructed image. The receptive field size is determined by R m-1 =[(R m -1)×Str m ]+K m Calculate, where R m-1 With R mThey represent the receptive fields of the deeper mth feature map and the shallower (m-1)th feature map, Str m represents the convolution step size, K m Represents the convolution kernel size, and the receptive field size can be iteratively calculated by this formula. In this implementation, the receptive field size of the final layer of the ALEX network is 195.

[0060] Among them, the functional expression of perceptual loss is:

[0061] In the above formula, Loss per represents the perceptual loss, ρ represents the perceptual computing network, X is the input vector (matrix) of size n, X i is the i-th component of the vector (matrix), and a is the autoencoder operation.

[0062] Then, the perceptual loss and back-propagation algorithm are used to optimize the perceptual autoencoder network so that it can obtain the characteristics of military circuits and use them for detection.

[0063] It is worth noting that color images can also be synthesized using perceptual loss, but more training samples are required to extract features. In this embodiment, grayscale images are used for training, and the number of training samples is 63.

[0064] Since the feature map has a large receptive field, the structural features are retained, so the mean square error of the feature map, that is, the perceptual loss is more accurate than the mean square error.

[0065] Step 3: Use the trained network to synthesize a defect-free template and use the quadtree algorithm for detection

[0066] As shown in Figure 3, after completing training in step 2, a grayscale image of the military circuit to be inspected is collected, defined as the image to be inspected. This image to be inspected is then fed into the trained perceptual autoencoder network to produce a defect-free output image, defined as the template image. A quadtree algorithm is used to recursively segment the template image and the image to be inspected into four equal parts, until the structural similarity of the segmented regions falls below a preset similarity threshold or the area of ​​the segmented regions falls below a preset minimum precision.

[0067] Specifically, the input image to be detected is of size W×H×N, and the network outputs a template image of size W×H×N. In this embodiment, the size of the image to be detected and the template image are not fixed, but both are single-channel, that is, W×H×1. The W×H×1 image to be detected is input to the quadtree segmentation module, and the quadtree divides the image to be detected into The four images are defined as regions A, B, C, and D. Structural similarity is calculated between A, B, C, and D and their corresponding locations in the template image. If the structural similarity is greater than a preset similarity threshold (0.2 in this example), quadtree segmentation is recursively performed on the region. After several segmentations, if the area of ​​the region falls below a set threshold (25 pixels in this example), segmentation is stopped and the corresponding region is output. After executing the quadtree algorithm, the resulting region is the area that differs between the two images and meets the preset accuracy, i.e., the defect area. If no output is given, it indicates that there is no defect.

[0068] 4 , which is a schematic diagram of a detection process using a quadtree algorithm, the present invention uses a quadtree algorithm to recursively divide the template image and the image to be detected into four equal parts. The specific operations are as follows:

[0069] (301), dividing the template image and the image to be detected into four images, and calculating the structural similarity between the segmented regions and the corresponding positions of the template image;

[0070] (302) selecting a region whose regional structural similarity is lower than a preset similarity threshold, and recursively performing quadtree segmentation on the region;

[0071] (303), compare the area of ​​the segmented region with the preset minimum precision, if it is greater, go to (302);

[0072] If the area of ​​the region is less than the preset minimum precision, go to (304);

[0073] (304), stop segmentation, compare the structural similarity of the segmented region with the preset similarity threshold; if it is less than or equal to, output the region as a defective region;

[0074] If it is greater, there is no output, indicating there is no defect.

[0075] The present invention uses quadtree segmentation to compare image differences, so there is no need to strictly align the images, and the preset accuracy can be used to limit the output area size.

[0076] Example 2

[0077] This embodiment provides a military product circuit surface defect detection system based on perception and unsupervised detection, including a perception autoencoder network construction module, a training module, and a detection module;

[0078] The perceptual autoencoder network construction module is used to construct an encoder-decoder to form an unsupervised perceptual autoencoder network, and the encoder contains a feature pyramid;

[0079] Inputting a defect-free sample image into the encoder, the encoder extracts features of different scales from the input image, and then performs convolution on the features to obtain a feature map. The decoder performs a deconvolution operation on the feature map to output an initial reconstructed image;

[0080] The training module is configured to input the initial reconstructed image and the original image into a perceptual computing network, the perceptual computing network calculating feature maps of the two images respectively, calculating a perceptual loss based on the feature maps of the two images, and optimizing the perceptual autoencoder network based on the perceptual loss and a back-propagation algorithm until the perceptual loss reaches a minimum value, thereby completing the training of the perceptual autoencoder network;

[0081] The detection module is used to collect an image of a military product circuit to be detected, input the image to be detected into a trained perceptual autoencoder network, and output a defect-free output image as a template image;

[0082] Then, the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops. At this point, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region.

[0083] If the structural similarity of the region is greater than the preset similarity threshold, no output is given, indicating that there is no defect.

[0084] Example 3

[0085] The present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The device is characterized in that when the processor executes the computer program, the steps of the surface defect detection method based on perceptual autoencoder of the present invention are implemented.

[0086] Example 4

[0087] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the surface defect detection method based on perceptual autoencoder of the present invention.

[0088] See FIG5 , which is a schematic diagram of input and generated templates according to an embodiment.

[0089] After testing, the present invention achieved a detection accuracy of 78.03 for the front side and 76.34 for the back side, and an F1 score of 86.66 for the front side and 86.58 for the back side on a dip-8 chip package surface dataset containing 200 images.

[0090] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A surface defect detection method based on perceptual autoencoder, characterized in that: The following steps are involved: (1) constructing an encoder-decoder to form an unsupervised perceptual autoencoder network, wherein the encoder contains a feature pyramid; The defect-free sample image is input into the encoder, the encoder extracts features of different scales from the input image, and then performs convolution on the features to obtain a feature map, and the decoder performs a deconvolution operation on the feature map to output an initial reconstructed image; (2) Inputting the initial reconstructed image and the original image into the perceptual computing network, the perceptual computing network respectively calculates feature maps of the two, calculates perceptual loss based on the feature maps of the two, and optimizes the perceptual autoencoder network based on the perceptual loss and the back propagation algorithm until the perceptual loss reaches a minimum value, thereby completing the training of the perceptual autoencoder network; (3) Collecting images of military circuits to be inspected, inputting the images to be inspected into the trained perceptual autoencoder network, and outputting defect-free output images as template images; Then, the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum accuracy, and the segmentation is stopped; At this time, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defective region; If the structural similarity of the region is greater than the preset similarity threshold, there is no output, indicating that there is no defect.

2. The surface defect detection method based on perceptual autoencoder according to claim 1 is characterized in that: In step (1), the sample image is a grayscale image or a color image of a military product circuit.

3. The surface defect detection method based on perceptual autoencoder according to claim 1 is characterized in that: In step (2), the perceptual loss is the mean square error between the initial reconstructed image feature map and the original image feature map.

4. The surface defect detection method based on perceptual autoencoder according to claim 3 is characterized in that: In step (2), the functional expression of the perceptual loss is: In the formula, Loss per represents the perceptual loss, ρ represents the perceptual computing network, X is the input image, which contains n components after vectorization, a is the autoencoder operation, and X k is the kth component of X, a(X) is the reconstructed version of the autoencoder output, a(X) k is the kth component after vectorization of the remodeled version, ρ(X k )、ρ(a(X) k ) are the input image and reconstructed model feature vector extracted by perceptual network respectively.

5. The surface defect detection method based on perceptual autoencoder according to claim 1 is characterized in that: In step (3), the similarity threshold is preset to 0.2; The preset minimum precision is 25 pixels.

6. The surface defect detection method based on perceptual autoencoder according to claim 1 is characterized in that: In step (3), the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm, and the specific operation is as follows: (301), dividing the template image and the image to be detected into four images, and calculating the structural similarity between the divided regions and the corresponding positions of the template image; (302), selecting a region whose regional structural similarity is lower than a preset similarity threshold, and recursively performing quadtree segmentation on the region; (303), compare the area of ​​the segmented region with the preset minimum precision, if it is greater, go to (302); If the area of ​​the region is less than the preset minimum precision, go to (304); (304), stopping segmentation, and comparing the structural similarity of the segmented regions with the preset similarity threshold; If it is less than or equal to, the area is output as a defective area; If it is greater, there is no output, indicating there is no defect.

7. A surface defect detection system based on perceptual autoencoder, characterized in that: It includes a perceptual autoencoder network building module, a training module, and a detection module; The perceptual autoencoder network building module is used to construct an encoder-decoder to form an unsupervised perceptual autoencoder network, and the encoder contains a feature pyramid; The defect-free sample image is input into the encoder, the encoder extracts features of different scales from the input image, and then performs convolution on the features to obtain a feature map, and the decoder performs a deconvolution operation on the feature map to output an initial reconstructed image; The training module is used to input the initial reconstructed image and the original image into the perceptual computing network, the perceptual computing network calculates feature maps of the two respectively, calculates the perceptual loss based on the feature maps of the two, and optimizes the perceptual autoencoder network based on the perceptual loss and the back propagation algorithm until the perceptual loss reaches a minimum value, thereby completing the training of the perceptual autoencoder network; The detection module is used to collect the image to be detected of the military product circuit to be detected, input the image to be detected into the trained perceptual autoencoder network, and output a defect-free output image as a template image; Then, the template image and the image to be detected are recursively divided into four equal parts using a quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset maximum similarity threshold. When the precision is small, the segmentation is stopped; at this time, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defective region; If the structural similarity of the region is greater than the preset similarity threshold, there is no output, indicating that there is no defect.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the surface defect detection method based on perceptual autoencoder described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the surface defect detection method based on perceptual autoencoder described in any one of claims 1 to 6 are implemented.

Images