A surface defect detection method based on a perception autoencoder and a related device

Abstract

This invention proposes a surface defect detection method based on a perceptual autoencoder, belonging to the field of integrated circuit defect detection. This invention uses perceptual loss instead of pixel-by-pixel loss, solving the problems of blurred output images and severe edge distortion caused by single-pixel loss. Furthermore, this method has strong transferability, capable of effectively transferring to new targets. On the other hand, this invention uses a feature pyramid instead of the sliding window method, eliminating the need for extensive redundant calculations by sliding the feature pyramid across the original image, effectively reducing memory and computational costs. This invention also provides a surface defect detection device based on a perceptual autoencoder, comprising specific modules for implementing the above-described method. Finally, this invention provides a computer device and storage medium for implementing the specific steps of the above-described method.

Description

Technical Field

[0001] This invention belongs to the field of integrated circuit defect detection, and in particular to a surface defect detection method and related device based on a sensing autoencoder. Background Technology

[0002] In the field of automated defect detection technology for military circuits, machine vision methods are generally used for defect detection. Based on different feature extraction methods, these methods are divided into traditional appearance inspection methods and intelligent inspection methods. Existing traditional automated inspection methods mainly rely on specific optical designs for detection. Typical methods include structured light defect detection and design-based light source reflective surface marking detection. These methods have poor transferability and require independent design for different products. Existing intelligent inspection methods can be further divided into two categories based on whether supervision is required: supervised methods have high accuracy but require a large amount of labeled product data, making them unsuitable for industrial inspection scenarios; unsupervised methods do not require independent design or manual labeling and can adapt to the application scenarios of surface defect detection for military circuits.

[0003] To improve the transferability of inspection across different products, AutoEncoder (AE) was used. AE is an unsupervised technique based on backpropagation and optimization methods, using the input data itself as supervision to guide the neural network in learning a mapping relationship. As a novel feature extraction method, AE can complete product image feature extraction and image generation under unsupervised conditions. Therefore, its feature extraction capabilities can be used to extract features from defect-free products, generate templates, and compare them, enabling rapid transfer to different military circuit surface inspection scenarios, greatly improving the versatility of surface inspection methods.

[0004] Currently, most Advanced Image Processing (AE) applications in other fields use pixel-wise loss as the loss function, such as MSE (Mean Squared Error). Pixel-wise loss only deals with information about individual pixels, ignoring structural information across pixels, which is detrimental to the surface inspection of military circuits. In this process, due to the characteristics of pixel-wise loss, the network tends to output a blurred image, leading to severe edge distortion and poor detection performance for products with many edges, such as military circuits. Furthermore, pixel-wise loss treats each pixel equally, while in the military circuit inspection scenario, pixels representing defects are far more important than defect-free pixels, making pixel-wise loss unsuitable for detecting surface defects in military circuits. If AE uses perceptual loss, it can reduce the loss of edge information, thereby increasing the similarity of the synthesized template and improving detection accuracy. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a surface defect detection method and related apparatus based on a sensor autoencoder.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] A surface defect detection method based on a perceptual autoencoder includes the following steps:

[0008] (1) Construct an encoder-decoder network to form an unsupervised perceptual autoencoder network, wherein the encoder contains a feature pyramid;

[0009] A defect-free sample image is input into the encoder, which extracts features at different scales from the input image and then performs convolution on the features to obtain a feature map. The decoder performs deconvolution on the feature map and outputs the initial reconstructed image.

[0010] (2) Input the initial reconstructed image and the original image into the perceptual computing network. The perceptual computing network calculates the feature maps of the two images respectively, calculates the perceptual loss based on the feature maps of the two images, and optimizes the perceptual autoencoder network based on the perceptual loss and the backpropagation algorithm until the perceptual loss reaches the minimum value, thus completing the training of the perceptual autoencoder network.

[0011] (3) Collect the image of the military circuit to be tested, input the image to be tested into the trained perceptual autoencoder network, and output a defect-free output image as a template image.

[0012] Then, the template image and the image to be detected are recursively segmented into four equal parts using the quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops; at this point, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region.

[0013] If the structural similarity of the region is greater than the preset similarity threshold, there will be no output, indicating that there is no defect.

[0014] Furthermore, in step (1), the sample image is a grayscale image or a color image of a military-grade circuit.

[0015] Furthermore, in step (2), the perceptual loss is the mean square error between the initial reconstructed image feature map and the original image feature map.

[0016] Furthermore, in step (2), the functional expression for the perceptual loss is:

[0017]

[0018] In the formula, Loss per Let ρ represent the perceptual loss, ρ represent the perceptual computation network, X be the input image (vectorized to have n components), and a be the autoencoder operation. kLet X be the k-th component of X, and let a(X) be the remodeled version of the autoencoder output. k For the k-th component after vectorization of the remodeled version, ρ(X) k ), ρ(a(X) k ) are the input image and the remodeled feature vector extracted using a perceptual network, respectively.

[0019] Furthermore, in step (3), the preset similarity threshold is 0.2;

[0020] The preset minimum precision is 25 pixels.

[0021] Furthermore, in step (3), the template image and the image to be detected are recursively segmented into four equal parts using the quadtree algorithm. The specific operation is as follows:

[0022] (301) Segment the template image and the image to be detected into four images, and calculate the structural similarity between the segmented regions and the corresponding positions in the template image;

[0023] (302) Select regions whose regional structural similarity is lower than a preset similarity threshold, and then recursively perform quadtree segmentation on the regions;

[0024] (303) Compare the area of ​​the segmented region with the preset minimum precision. If it is greater, go to (302).

[0025] If the area of ​​the region is less than the preset minimum precision, then go to (304);

[0026] (304) Stop segmentation, compare the structural similarity of the segmented region with the preset similarity threshold; if it is less than or equal to the threshold, the region is output as a defect region.

[0027] If the value is greater than 1, there will be no output, indicating that there is no defect.

[0028] A surface defect detection system based on a perceptual autoencoder includes a perceptual autoencoder network construction module, a training module, and a detection module;

[0029] The perceptual autoencoder network construction module is used to construct an unsupervised perceptual autoencoder network consisting of an encoder and a decoder, wherein the encoder contains a feature pyramid.

[0030] A defect-free sample image is input into the encoder, which extracts features at different scales from the input image and then performs convolution on the features to obtain a feature map. The decoder performs deconvolution on the feature map and outputs the initial reconstructed image.

[0031] The training module is used to input the initial reconstructed image and the original image into the perceptual computing network. The perceptual computing network calculates the feature maps of the two images respectively, calculates the perceptual loss based on the feature maps, and optimizes the perceptual autoencoder network based on the perceptual loss and the backpropagation algorithm until the perceptual loss reaches the minimum value, thus completing the training of the perceptual autoencoder network.

[0032] The detection module is used to acquire the image of the military circuit to be inspected, input the image to be inspected into the trained perceptual autoencoder network, and output a defect-free output image as a template image.

[0033] Then, the template image and the image to be detected are recursively segmented into four equal parts using the quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops; at this point, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region.

[0034] If the structural similarity of the region is greater than the preset similarity threshold, there will be no output, indicating that there is no defect.

[0035] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the surface defect detection method based on a perceptual autoencoder according to the present invention.

[0036] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the surface defect detection method based on a perceptual autoencoder described in this invention.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This invention proposes a surface defect detection method based on a perceptual autoencoder. It uses perceptual loss instead of pixel-by-pixel loss, solving the problems of blurred output images and severe edge distortion caused by single-pixel loss. Furthermore, this method exhibits strong transferability, effectively adapting to new targets. On the other hand, this invention uses a feature pyramid instead of the sliding window method. The feature pyramid eliminates the need for extensive redundant calculations by sliding across the original image, effectively reducing memory and computational costs.

[0039] The present invention provides a surface defect detection device based on a sensor autoencoder, which includes specific modules for performing the above-mentioned working method.

[0040] This invention provides a computer device and storage medium for a surface defect detection method based on a perceptual autoencoder, which implements the specific steps of the above-mentioned working method. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of an autoencoder network;

[0042] Figure 2 This is a schematic diagram of the training process of the perceptual autoencoder of the present invention;

[0043] Figure 3 This is a schematic diagram of the detection process of the self-encoder of the present invention;

[0044] Figure 4 This is a schematic diagram of the quadtree algorithm detection process;

[0045] Figure 5 This is a schematic diagram of the input and generated templates in one embodiment. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] The present invention will now be described in further detail with reference to the accompanying drawings:

[0049] Example 1

[0050] The specific scenario addressed by this invention is: during the manufacturing process of military circuits, surface defect detection is performed on the manufactured military circuits.

[0051] This invention provides a surface defect detection method based on a perceptual autoencoder, comprising the following steps:

[0052] Step 1: Construct an unsupervised, perceptual autoencoder network, including a feature pyramid:

[0053] Specifically, construct an unsupervised perceptual autoencoder network, see [link to documentation]. Figure 1 The basic architecture of an unsupervised perceptual autoencoder network is an encoder-decoder combination. The encoder performs convolutional feature extraction on the preprocessed input image to obtain a feature map; the decoder then performs deconvolution on the feature map to obtain the reconstructed image. It should be noted that the network outputs feature maps of different depths, forming a feature pyramid. Deconvolution is then performed simultaneously on these feature maps to obtain multiple output images. The purpose of this step is to build the unsupervised perceptual autoencoder network and obtain the initial reconstructed image.

[0054] This invention uses a feature pyramid instead of the sliding window method, eliminating the need for redundant calculations and saving computational costs. Furthermore, it eliminates the need for input image annotation, saving significant manpower.

[0055] Step 2: Calculate the perceptual loss using the reconstructed image and optimize the perceptual autoencoder network:

[0056] The reconstructed image obtained in step one is used to calculate the perceptual loss. Specifically, the reconstructed image and the original image are input into a pre-trained perceptual computing network. The perceptual computing network calculates the perceptual loss of both images and uses an optimization algorithm to optimize until the perceptual loss is minimized.

[0057] First, a grayscale image of the surface of a defect-free military circuit is acquired and recorded as the original image. Then, the acquired original image is input into a perceptual autoencoder network, which outputs an initial reconstructed image. The original image and the initial reconstructed image are then input into a perceptual computation network to obtain the perceptual loss. The perceptual loss is calculated by inputting the original image and the initial reconstructed image into a pre-trained perceptual computation network, which calculates the feature images of both images and calculates the mean square error between them.

[0058] Specifically, in the initial reconstructed image OUTPUT output by the perceptual autoencoder network, which is W×H×N in size, OUTPUT corresponds one-to-one with the original image INPUT at the pixel level. In this embodiment, both the original image and the initial reconstructed image are 512×512×1. A perceptual computing network is used to calculate feature maps for the original image and the initial reconstructed image through convolution. Each pixel in the feature map of the original image corresponds to multiple pixels in the original image; these multiple pixels are called the receptive field, and the same applies to the initial reconstructed image. The size of the receptive field is determined by R. m-1 =[(R m -1)×Str m ]+K m Calculate, where R m-1 With R mStr represents the receptive field of the m-th feature map (which is deeper) and the (m-1)-th feature map (which is shallower), respectively. m K represents the convolution stride. m The kernel size is represented by the above formula. The receptive field size can be calculated iteratively. In this embodiment, the receptive field size of the final layer of the ALEX network is 195.

[0059] The functional expression for the perceptual loss is:

[0060]

[0061] In the above formula, Loss per Let ρ represent the perceptual loss, ρ represent the perceptual computation network, and X be an input vector (matrix) of size n. i Let be the i-th component of the vector (matrix), and 'a' be the autoencoder operation.

[0062] Then, the perceptual autoencoder network is optimized using perceptual loss and backpropagation algorithms to enable it to acquire the characteristics of military circuits and use them for detection.

[0063] It is worth noting that color images can also be synthesized using perceptual loss, but more training samples are needed to extract features. In this embodiment, grayscale images are used for training, with 63 training samples.

[0064] Because the receptive field of the feature map is large, it preserves structural features. Therefore, the mean squared error of the feature map, i.e., the perceptual loss, is more accurate than the mean squared error.

[0065] Step 3: Synthesize defect-free templates using the trained network and use the quadtree algorithm for detection.

[0066] See details Figure 3 After training is completed in step two, a grayscale image of the military circuit to be inspected is acquired and defined as the image to be inspected. The image to be inspected is input into the trained perceptual autoencoder network to obtain a defect-free output image, defined as the template image. The template image and the image to be inspected are recursively segmented into four equal parts using a quadtree algorithm until the structural similarity of the segmented regions is lower than a preset similarity threshold, or the area of ​​the segmented regions is less than a preset minimum precision.

[0067] Specifically, the input image to be detected is of size W×H×N, and the network outputs a template image of size W×H×N. In this embodiment, the sizes of the image to be detected and the template image are not fixed, but both are single-channel, i.e., W×H×1. The W×H×1 image to be detected is input into the quadtree segmentation module, and the quadtree segments the image to be detected into... Four images are defined as regions A, B, C, and D. The structural similarity between A, B, C, and D and their corresponding positions in the template image is calculated. If the structural similarity is greater than a preset similarity threshold (0.2 in this example), quadtree segmentation is recursively performed on that region. Segmentation continues until, after several segmentations, the area of ​​the region is less than a set threshold (25 pixels in this example), at which point the segmentation stops, and the corresponding region is output. The region obtained after executing the quadtree algorithm is the region that differs between the two images and meets the preset precision, i.e., the defect region. If no output is given, it indicates that there is no defect.

[0068] See Figure 4 , Figure 4 This is a schematic diagram of the quadtree algorithm detection process. The present invention utilizes the quadtree algorithm to recursively segment the template image and the image to be detected into four equal parts. The specific operation is as follows:

[0069] (301) Segment the template image and the image to be detected into four images, and calculate the structural similarity between the segmented regions and the corresponding positions in the template image;

[0070] (302) Select regions whose regional structural similarity is lower than a preset similarity threshold, and then recursively perform quadtree segmentation on the regions;

[0071] (303) Compare the area of ​​the segmented region with the preset minimum precision. If it is greater, go to (302).

[0072] If the area of ​​the region is less than the preset minimum precision, then go to (304);

[0073] (304) Stop segmentation, compare the structural similarity of the segmented region with the preset similarity threshold; if it is less than or equal to the threshold, the region is output as a defect region.

[0074] If the value is greater than 1, there will be no output, indicating that there is no defect.

[0075] This invention uses quadtree segmentation to compare image differences, so there is no need to strictly align the images, and the output area size can be limited using preset precision.

[0076] Example 2

[0077] This embodiment provides a sensor-based unsupervised detection system for surface defects in military circuits, including a sensory autoencoder network construction module, a training module, and a detection module;

[0078] The perceptual autoencoder network construction module is used to construct an unsupervised perceptual autoencoder network consisting of an encoder and a decoder, wherein the encoder contains a feature pyramid.

[0079] A defect-free sample image is input into the encoder, which extracts features at different scales from the input image and then performs convolution on the features to obtain a feature map. The decoder performs deconvolution on the feature map and outputs the initial reconstructed image.

[0080] The training module is used to input the initial reconstructed image and the original image into the perceptual computing network. The perceptual computing network calculates the feature maps of the two images respectively, calculates the perceptual loss based on the feature maps, and optimizes the perceptual autoencoder network based on the perceptual loss and the backpropagation algorithm until the perceptual loss reaches the minimum value, thus completing the training of the perceptual autoencoder network.

[0081] The detection module is used to acquire the image of the military circuit to be inspected, input the image to be inspected into the trained perceptual autoencoder network, and output a defect-free output image as a template image.

[0082] Then, the template image and the image to be detected are recursively segmented into four equal parts using the quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops; at this point, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region.

[0083] If the structural similarity of the region is greater than the preset similarity threshold, there will be no output, indicating that there is no defect.

[0084] Example 3

[0085] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the surface defect detection method based on a perceptual autoencoder of the present invention.

[0086] Example 4

[0087] A computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the surface defect detection method based on a perceptual autoencoder of the present invention.

[0088] See Figure 5 , Figure 5 This is a schematic diagram of the input and generated templates in one embodiment.

[0089] Tests showed that the present invention achieved a detection accuracy of 78.03 for the front side and 76.34 for the back side on a dataset containing 200 images of the dip-8 chip package surface, and F1 scores of 86.66 for the front side and 86.58 for the back side.

[0090] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A surface defect detection method based on a perceptual autoencoder, characterized in that, Includes the following steps: (1) Construct an encoder-decoder network to form an unsupervised perceptual autoencoder network, wherein the encoder contains a feature pyramid; A defect-free sample image is input into the encoder, which extracts features at different scales from the input image and then performs convolution on the features to obtain a feature map. The decoder performs deconvolution on the feature map and outputs the initial reconstructed image. (2) Input the initial reconstructed image and the original image into the perceptual computing network. The perceptual computing network calculates the feature maps of the two images respectively. Based on the feature maps of the two images, the perceptual loss is calculated. Based on the perceptual loss and the backpropagation algorithm, the perceptual autoencoder network is optimized until the perceptual loss reaches the minimum value, and the training of the perceptual autoencoder network is completed. (3) Collect the image of the military circuit to be tested, input the image to be tested into the trained perceptual autoencoder network, and output a defect-free output image as a template image; Then, the template image and the image to be detected are recursively segmented into four equal parts using the quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops. At this time, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region; If the structural similarity of the region is greater than the preset similarity threshold, there is no output, indicating that there is no defect; The template image and the image to be detected are recursively segmented into four equal parts using a quadtree algorithm. The specific operation is as follows: (301) Segment the template image and the image to be detected into four images, and calculate the structural similarity between the segmented regions and the corresponding positions in the template image; (302) Select regions whose regional structural similarity is lower than a preset similarity threshold, and then recursively perform quadtree segmentation on the regions; (303) Compare the area of ​​the segmented region with the preset minimum precision. If it is greater, go to (302). If the area of ​​the region is smaller than the preset minimum precision, then go to (304). (304) Stop segmentation, compare the structural similarity of the segmented region with the preset similarity threshold; if it is less than or equal to the threshold, the region is output as a defect region; If the value is greater than 1, there will be no output, indicating that there is no defect.

2. The surface defect detection method based on a perceptual autoencoder according to claim 1, characterized in that, In step (1), the sample image is a grayscale image or a color image of a military circuit.

3. The surface defect detection method based on a perceptual autoencoder according to claim 1, characterized in that, In step (2), the perceptual loss is the mean square error between the initial reconstructed image feature map and the original image feature map.

4. The surface defect detection method based on a perceptual autoencoder according to claim 3, characterized in that, In step (2), the functional expression for the perceptual loss is: In the formula, Indicates perceived loss. Represents a perceptual computing network. Given an input image, after vectorization, it contains... n One portion, For autoencoder operation, X k for The k One portion, A remodeled version output by an autoencoder. The vectorized version of the remodeled version k One portion, , These are the input image extracted using a perceptual network and the remodeled feature vector, respectively.

5. The surface defect detection method based on a perceptual autoencoder according to claim 1, characterized in that, In step (3), the preset similarity threshold is 0.2; The preset minimum precision is 25 pixels.

6. A surface defect detection system based on a perceptual autoencoder, characterized in that, It includes a perceptual autoencoder network construction module, a training module, and a detection module; The perceptual autoencoder network construction module is used to construct an unsupervised perceptual autoencoder network consisting of an encoder and a decoder, wherein the encoder contains a feature pyramid. A defect-free sample image is input into the encoder, which extracts features at different scales from the input image and then performs convolution on the features to obtain a feature map. The decoder performs deconvolution on the feature map and outputs the initial reconstructed image. The training module is used to input the initial reconstructed image and the original image into the perceptual computing network. The perceptual computing network calculates the feature maps of the two images respectively, calculates the perceptual loss based on the feature maps, and optimizes the perceptual autoencoder network based on the perceptual loss and the backpropagation algorithm until the perceptual loss reaches the minimum value, thus completing the training of the perceptual autoencoder network. The detection module is used to acquire the image of the military circuit to be inspected, input the image to be inspected into the trained perceptual autoencoder network, and output a defect-free output image as a template image. Then, the template image and the image to be detected are recursively segmented into four equal parts using the quadtree algorithm until the area of ​​the segmented region below the preset similarity threshold is less than or equal to the preset minimum precision, at which point the segmentation stops. At this time, if the structural similarity of the region is less than or equal to the preset similarity threshold, it is output as a defect region; If the structural similarity of the region is greater than the preset similarity threshold, there is no output, indicating that there is no defect; The template image and the image to be detected are recursively segmented into four equal parts using a quadtree algorithm. The specific operation is as follows: (301) Segment the template image and the image to be detected into four images, and calculate the structural similarity between the segmented regions and the corresponding positions in the template image; (302) Select regions whose regional structural similarity is lower than a preset similarity threshold, and then recursively perform quadtree segmentation on the regions; (303) Compare the area of ​​the segmented region with the preset minimum precision. If it is greater, go to (302). If the area of ​​the region is smaller than the preset minimum precision, then go to (304). (304) Stop segmentation, compare the structural similarity of the segmented region with the preset similarity threshold; if it is less than or equal to the threshold, the region is output as a defect region; If the value is greater than 1, there will be no output, indicating that there is no defect.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the surface defect detection method based on a perceptual autoencoder as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the surface defect detection method based on a perceptual autoencoder as described in any one of claims 1-5.

Images