Semiconductor packaging defect detection system based on machine vision

Abstract

The invention discloses a semiconductor packaging defect detection system based on machine vision, and relates to the technical field of semiconductor packaging defect detection, and the system comprises a multispectral collection module which uses a multispectral camera and a light source to collect multiband images of a packaging surface from different angles, and carries out the preprocessing of the multiband images; the image fusion module is used for fusing the preprocessed wave band images into a comprehensive image by adopting an image fusion algorithm; the illumination adjusting module is used for carrying out illumination quality analysis on the comprehensive image, adjusting light source parameters based on a quality analysis result and re-collecting a multi-band image; the method has the beneficial effects that through the comprehensiveness of a multispectral image, the detail enhancement capability of image fusion, the stability of dynamic illumination adjustment, the feature extraction capability of deep learning and the anomaly detection capability of an auto-encoder, the robustness of the image fusion is improved; the defects of unstable image quality, insufficient feature extraction capability, inaccurate error judgment and the like in the prior art are comprehensively overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of semiconductor package defect detection, and particularly to a semiconductor package defect detection system based on machine vision. Background Art

[0002] With the rapid development of semiconductor technology and the high integration of electronic product design, semiconductor packaging technology, as a key link for chip protection and function realization, has received extensive attention. The quality of semiconductor packaging is directly related to the reliability, performance, and lifespan of chips. Therefore, the detection of packaging defects during the production process is of great significance. Traditional semiconductor package defect detection methods mainly rely on manual inspection or machine vision detection technology based on single-spectrum images. Manual inspection, due to relying on the experience of operators, has low efficiency and is easily affected by subjective factors, and cannot meet the requirements of modern manufacturing for high-precision and high-efficiency detection. Although the detection technology based on single-spectrum images has achieved a certain degree of automation, due to the limitations of a single spectrum, its detection ability for complex surface defects (such as micro-cracks, foreign objects, bubbles, or irregular reflection areas) is weak. Especially under a combination of multiple materials and complex lighting conditions, the accuracy and robustness of the detection results are insufficient. Summary of the Invention

[0003] In view of the above existing problems, the present invention is proposed.

[0004] Therefore, the present invention provides a semiconductor package defect detection system based on machine vision to solve the problems of insufficient multi-spectral information fusion, unstable lighting quality, and low defect determination accuracy in semiconductor package defect detection.

[0005] To solve the above technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a semiconductor package defect detection system based on machine vision, which includes a multi-spectral acquisition module that uses a multi-spectral camera and a light source to collect multi-band images of the package surface from different angles and preprocesses the multi-band images;

[0007] an image fusion module that fuses the preprocessed multi-band images into a comprehensive image using an image fusion algorithm;

[0008] a lighting adjustment module that analyzes the lighting quality of the comprehensive image, adjusts the light source parameters based on the quality analysis results, and re-collects multi-band images;

[0009] a feature extraction module that obtains the features of the comprehensive image using a convolutional neural network to obtain a convolutional feature map;

[0010] a normal feature reconstruction module that reconstructs the original feature map of a normal package using an autoencoder network;

[0011] A reconstruction error calculation module calculates the reconstruction error between the original feature map and the convolutional feature map, sets an error threshold, and determines whether the package is defective.

[0012] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: the multi-spectral camera and the light source are used to collect multi-band images of the package surface from different angles, including,

[0013] A multi-spectral camera covering visible light, near-infrared, and ultraviolet light bands is selected and a special multi-spectral lens is installed;

[0014] An adjustable multi-band LED light source is selected, and the multi-band LED light source is deployed around the multi-spectral camera in a circular array;

[0015] A mechanical rotating platform is configured to image the package at different angles. The control of the rotating platform is synchronously connected to the multi-spectral camera and the multi-band LED light source, and the multi-spectral camera and the multi-band LED light source are triggered to image at each angular position;

[0016] Images are sequentially collected at all angles to obtain multi-band images.

[0017] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: the preprocessing of the multi-band images includes,

[0018] The Gaussian noise model and the Gaussian filter are respectively used to denoise the multi-band images to obtain denoised images;

[0019] Histogram equalization is performed on the denoised images of each band to obtain images with enhanced contrast;

[0020] The images after denoising and histogram equalization are stored in units of bands.

[0021] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: the image fusion algorithm is used to fuse the preprocessed images of each band into a comprehensive image, including,

[0022] The wavelet transform-based image fusion method is used to perform wavelet decomposition on the images of each band to obtain low-frequency components and high-frequency components;

[0023] For the low-frequency components, weighted average fusion is used, and for the high-frequency components, the maximum selection method is used for fusion. At each pixel position, the maximum value of the high-frequency components in all bands is selected to retain the most significant details;

[0024] After the fusion of the low-frequency component and the high-frequency component, inverse wavelet transform is performed for reconstruction, and the fused low-frequency component and high-frequency component are combined into a complete fused image;

[0025] The Laplacian operator is used to enhance the details of the fused image;

[0026] The Laplacian-enhanced image is superimposed on the original fused image to obtain the final comprehensive image with enhanced details.

[0027] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: the illumination quality analysis of the comprehensive image includes,

[0028] Perform brightness analysis on the fused comprehensive image, calculate the gray value of each pixel of the comprehensive image, which is defined as the brightness of the comprehensive image, then calculate the average value of the brightness of all pixels of the comprehensive image, which is defined as the global brightness, set the global brightness threshold, and when the global brightness exceeds this threshold, the light source brightness needs to be adjusted;

[0029] Calculate the local brightness standard deviation of the comprehensive image, divide the comprehensive image into several sub-regions, and when the local brightness standard deviation is too large, it indicates uneven illumination, and the angle and brightness of the light source need to be adjusted;

[0030] Analyze the highlight and shadow regions in the comprehensive image through histogram analysis, calculate the proportion of pixels whose brightness values in the comprehensive image are within the extreme value range, set the proportion threshold, and when the proportion exceeds this threshold, the angle and brightness of the light source need to be adjusted;

[0031] Analyze the color temperature distribution of the comprehensive image, detect the color temperature by analyzing the RGB channel values of the comprehensive image, calculate the ratio of the RGB channels, set the ratio threshold, and when the ratio of the RGB channels is not within this threshold range, the color temperature of the light source needs to be adjusted.

[0032] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: the adjustment of the light source parameters and the re-acquisition of multi-band images based on the quality analysis results include,

[0033] When any one of the conditions that the global brightness exceeds the range, the illumination is uneven, there are too many highlights and shadows, and the color temperature is inappropriate is met in the illumination quality analysis results, the brightness, angle, and color temperature of the light source are adaptively adjusted;

[0034] After the adjustment of the brightness, angle, and color temperature of the light source is completed, multi-band images are re-acquired;

[0035] The re-acquired multi-band images are preprocessed again and fused again to obtain a new comprehensive image;

[0036] Perform illumination quality analysis on the new composite image to verify whether the adjusted illumination meets the preset requirements. If the illumination quality still does not meet the requirements, continue to repeat the adjustment process until the requirements for illumination uniformity, brightness range, and color temperature are met, and the final composite image is obtained.

[0037] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: obtaining the features of the composite image by using a convolutional neural network to obtain a convolutional feature map, including,

[0038] Using a pre-trained convolutional neural network as a feature extractor, inputting the composite image into the feature extractor to extract low-level features, middle-level features, and high-level features;

[0039] The low-level features are the edges, corners, and textures of the composite image; the middle-level features are the textures, shapes, and basic structures of the composite image; the high-level features are the semantics and global information of the composite image;

[0040] Perform linear transformations on the low-level features, middle-level features, and high-level features respectively, so that features at different levels have the same dimension;

[0041] Adopt a global context attention mechanism to calculate the global context representation of each level of features, and obtain the global context vectors corresponding to the low-level features, middle-level features, and high-level features through global average pooling. The expression is:

[0042]

[0043] Among them, g(x) is the context vector, F l (x) is the feature map of the l-th layer, H l is the height of the feature map of the l-th layer, W l is the width of the feature map of the l-th layer, F(x) i,j is the value of the feature map at the (i, j) position;

[0044] Calculate the attention weights for the global context vectors of each level. Through a two-layer fully connected network, map the context vectors of each level to a scalar that is the attention weight for the features of that level;

[0045] Weightedly fuse each level of features according to the attention weights of each level of features to obtain the final convolutional feature map.

[0046] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: reconstructing the original feature map of the normal package by using an autoencoder network, including,

[0047] Input the convolutional feature map into the encoder, and gradually compress the dimension of the convolutional feature map through multiple fully connected layers, and finally output the latent space representation;

[0048] The latent space representation is input into the decoder, which gradually expands the feature dimension through multiple fully connected layers until the original input dimension is restored to obtain the original feature map.

[0049] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: calculating the reconstruction error between the original feature map and the convolutional feature map includes,

[0050] The mean square error is used to calculate the reconstruction error between the original feature map and the convolutional feature map, and the formula is:

[0051]

[0052] where L(x) is the reconstruction error, F(x) i is the i-th element of the original feature map, is the i-th element of the convolutional feature map, and n is the total number of elements of the feature map.

[0053] As a preferred solution of the semiconductor package defect detection system based on machine vision according to the present invention, wherein: setting an error threshold to determine whether the package has defects includes,

[0054] Applying the trained autoencoder network to the normal packages in the validation set, calculating the reconstruction error of each package, recording the reconstruction errors of all normal samples, and forming a reconstruction error distribution;

[0055] Calculating the mean and standard deviation of the reconstruction error, and statistically analyzing the distribution characteristics of the reconstruction error;

[0056] Setting an error threshold based on the distribution characteristics of the reconstruction error. When the reconstruction error is greater than this threshold, it indicates that the surface package has defects.

[0057] The beneficial effects of the present invention are as follows: Through the comprehensiveness of multi-spectral images, the detail enhancement ability of image fusion, the stability of dynamic light adjustment, the feature extraction ability of deep learning, and the anomaly detection ability of autoencoders, the present invention comprehensively solves the deficiencies of the prior art in terms of unstable image quality, insufficient feature extraction ability, inaccurate error determination, etc., and greatly improves the accuracy and efficiency of semiconductor package defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0059] Figure 1 Schematic diagram of the semiconductor package defect detection system based on machine vision in Embodiment 1. Specific implementation manners

[0060] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the specific implementation manners of the present invention in detail with reference to the accompanying drawings of the specification.

[0061] In the following description, many specific details are set forth to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0062] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it an individual or alternative embodiment that is mutually exclusive with other embodiments.

[0063] Embodiment 1, referring to Figure 1 , which is the first embodiment of the present invention. This embodiment provides a semiconductor package defect detection system based on machine vision, including the following steps:

[0064] A multispectral acquisition module, using a multispectral camera and a light source, acquires multi-band images of the package surface from different angles and preprocesses the multi-band images;

[0065] Select a multispectral camera that can cover the visible light, near-infrared, and ultraviolet light bands, and install a special multispectral lens;

[0066] Select an adjustable multi-band LED light source that can emit visible light, near-infrared light, and ultraviolet light respectively. The light source is deployed around the multispectral camera and arranged in a circular array to ensure uniform illumination coverage of the package surface. The angle of the light source is adjustable to adapt to the reflection characteristics of different package materials and avoid overexposure caused by direct reflection;

[0067] Those skilled in the art set the initial illumination parameters based on the reflection characteristics of the package material, and use the adjustable function of the light source system to set the light source angle to 30 - 45 degrees to avoid interference of direct reflected light on the image. At the same time, adjust the distance between the light source and the package surface to ensure the best illumination coverage;

[0068] Configure a hardware trigger to ensure that the emission timing of the light source is strictly synchronized with the exposure timing of the multispectral camera. When the trigger sends a signal, the light source and the camera start simultaneously to ensure that the images of each band are accurately captured at the same moment;

[0069] According to the reflection characteristics and band differences of the encapsulation material, an appropriate exposure time is set for each band. Since the light intensity and material reflectivity in different bands are different, the setting of the exposure time should consider the reflected light intensity and the light sensitivity of the multispectral camera to ensure consistent image brightness in each band;

[0070] When imaging in different bands, the light source emits light in different wavelength ranges. Set the timing control of the trigger to ensure more precise switching time of the light source in different bands and avoid mutual interference between bands. The turn-on time of the light source in each band is a function of the exposure time to ensure that the light source switches to the next band immediately after the camera exposure is completed;

[0071] Configure a mechanical rotating platform to image the encapsulation body at different angles. The control of the rotating platform is synchronously connected to the multispectral camera and the light source, and the multispectral camera and the light source are triggered to image at each angular position;

[0072] Collect images sequentially at all angles until the complete 360 degrees are covered to obtain multi-band images;

[0073] An image fusion module uses an image fusion algorithm to fuse the preprocessed multi-band images into a comprehensive image;

[0074] Use the Gaussian noise model and the Gaussian filter to denoise the multi-band images respectively to obtain denoised images;

[0075] Perform histogram equalization on the denoised images of each band to obtain images with enhanced contrast;

[0076] After denoising and histogram equalization processing, the images of all bands have been enhanced, and the images are stored in units of bands;

[0077] Adopt the image fusion method of wavelet transform to decompose the images of each band by wavelet to obtain low-frequency components and high-frequency components. The low-frequency components contain the overall contour and large structure information of the image, and the high-frequency components contain the details and edge information of the image.

[0078] For the low-frequency components, weighted average fusion is adopted, and the weights are determined by the quality of the images of each band. For the high-frequency components, the maximum value selection method is used for fusion. At each pixel position, the maximum value of the high-frequency components in all bands is selected to retain the most significant details;

[0079] After completing the fusion of the low-frequency components and the high-frequency components, perform wavelet inverse transform reconstruction to combine the fused low-frequency components and high-frequency components into a complete fused image;

[0080] The Laplacian operator is used to enhance the details of the fused image, further highlighting the defect information on the encapsulation surface and inside the material, enhancing the edge features in the image, and clearly showing the tiny defects on the encapsulation surface;

[0081] The Laplacian-enhanced image is superimposed on the original fused image to obtain the final comprehensive image with enhanced details, which is stored in a standard image format;

[0082] The illumination adjustment module analyzes the illumination quality of the comprehensive image, adjusts the light source parameters based on the quality analysis results, and re-acquires multi-band images;

[0083] Perform brightness analysis on the fused comprehensive image, calculate the gray value of each pixel in the comprehensive image, define the gray value of each pixel as the brightness of the comprehensive image. The gray value of each pixel in the comprehensive image adopts the existing calculation method, which will not be elaborated here. Then calculate the average value of the brightness of all pixels in the comprehensive image, which is defined as the global brightness. The formula is:

[0084]

[0085] Among them, B g is the global brightness, M is the width of the comprehensive image, N is the height of the comprehensive image, and B(x, y) is the brightness of the comprehensive image.

[0086] Set the global brightness threshold according to the dynamic range of the multispectral camera, the requirement for the detail performance of the image, and through experimental calibration. When the global brightness exceeds this threshold, the light source brightness needs to be adjusted;

[0087] Calculate the local brightness standard deviation of the comprehensive image. Divide the image into several sub-regions. The brightness standard deviation of each sub-region represents the uniformity of the illumination in that region. When the local brightness standard deviation is too large, it means the illumination is uneven and the angle and brightness of the light source need to be adjusted;

[0088] Analyze the highlight and shadow regions in the comprehensive image through histogram analysis, and calculate the proportion of pixels whose brightness values in the comprehensive image are within the extreme value range;

[0089] Take experimental photos of the comprehensive image under different illumination conditions, analyze the influence of highlights and shadows in the comprehensive image on the detection target, set a threshold according to the experimental results. When the pixel proportion exceeds this threshold, the angle and brightness of the light source need to be adjusted;

[0090] Analyze the color temperature distribution of the comprehensive image, detect the color temperature by analyzing the RGB channel values of the comprehensive image, and calculate the ratio of the RGB channels;

[0091] Determine the optimal color temperature range through experimental calibration, set a ratio threshold. When the ratio of the RGB channels is not within this threshold range, the color temperature of the light source needs to be adjusted;

[0092] When any of the conditions is met in the light quality analysis results, such as the global brightness exceeding the range, uneven illumination, excessive highlights and shadows, or inappropriate color temperature, the brightness, angle, and color temperature of the light source are adaptively adjusted.

[0093] When the light brightness is not within the preset range, the brightness of the light source is adjusted. When the illumination is uneven, the angle of the light source is adjusted to achieve more uniform illumination. The goal of the angle adjustment is to reduce the standard deviation of the local brightness. The adjustment strategy of the light source angle can be based on the gradient of the local brightness difference. The pitch angle and rotation angle of the light source are adjusted. If the detected color temperature exceeds the preset range, the color temperature of the light source is adjusted to return it to the target color temperature range.

[0094] After the adjustment of the light source brightness, angle, and color temperature is completed, multi-band images are re-acquired.

[0095] The re-acquired multi-band images are pre-processed again and fused again to obtain a new composite image.

[0096] The light quality of the re-acquired composite image is analyzed to verify whether the adjusted illumination meets the preset requirements. If the light quality still does not meet the requirements, the adjustment process is continued until the requirements of illumination uniformity, brightness range, and color temperature are met, and the final composite image is obtained.

[0097] By analyzing the light quality of the composite image, the brightness, angle, and color temperature parameters of the light source are adjusted in real time to ensure the consistency of the acquired multi-band images under different lighting conditions. The analysis of multiple lighting indicators such as global brightness, local brightness standard deviation, highlight and shadow ratio, and color temperature distribution makes the adjustment of the light source parameters more accurate and intelligent. At the same time, through cyclic adjustment and verification, the situation of affecting defect detection due to uneven illumination or over-brightness or over-darkness is avoided.

[0098] The feature extraction module uses a convolutional neural network to obtain the features of the composite image and obtains a convolutional feature map.

[0099] A pre-trained convolutional neural network is used as the feature extractor. The composite image is input into the feature extractor to extract low-level features, middle-level features, and high-level features.

[0100] In the convolutional layer close to the input of the convolutional neural network, the detailed information, edge, corner, and texture features of the composite image are captured, extracted in the first few layers of the convolutional neural network, and output as low-level features.

[0101] In the middle layer of the convolutional neural network, the texture, shape, and basic structure of the image are captured, and middle-level features are extracted through multiple convolution, activation, and pooling operations. They are more abstract than low-level features but still retain certain local information.

[0102] In the deeper layers of a convolutional neural network, near the output layer, it captures the global structure, categories, and object - related information of an image. The output is high - level features, which have highly abstract semantic information and are important for understanding the global semantics in the image;

[0103] Therefore, low - level features synthesize the edges, corners, and textures of an image, middle - level features synthesize the textures, shapes, and basic structures of an image, and high - level features synthesize the semantics and global information of an image;

[0104] Perform linear transformations on low - level features, middle - level features, and high - level features respectively, so that features at different levels have the same dimension;

[0105] Adopt a global context attention mechanism to calculate the global context representation of features at each level. Obtain the global context vectors corresponding to low - level, middle - level, and high - level features through global average pooling. The expression is:

[0106]

[0107] where \(g(x)\) is the global context vector, \(F l (x)\) is the feature map of the \(l\) - th layer, \(H l \) is the height of the \(l\) - th layer feature map, \(W l \) is the width of the \(l\) - th layer feature map, \(F(x) i,j \) is the value of the feature map at the \((i, j)\) position;

[0108] Calculate the attention weights for the global context vectors at each level. Through a two - layer fully - connected network, map the context vectors at each level to a scalar, and this scalar is the attention weight of the features at that level;

[0109] The first - layer fully - connected layer maps the global context vector to an intermediate dimension, and performs non - linear transformation through linear transformation and activation function. The formula is,

[0110] \(z(x)=\text{ReLU}(W 1 g(x)+b 1 )\);

[0111] where \(z(x)\) is the intermediate - dimension representation, \(W 1 \) is the weight matrix of the first - layer fully - connected layer, \(g(x)\) is the global context vector, \(b 1 \) is the bias term of the first layer;

[0112] The second - layer fully - connected layer: maps the intermediate - dimension representation to a scalar value, and then obtains the attention weight of this layer through the activation function. The formula is,

[0113] \(\alpha(x)=\text{sigmoid}(W 2 z(x)+b 2 )\);

[0114] Among them, α(x) is the attention weight of this layer, and W 2 is the weight matrix of the second fully connected layer, z(x) is the intermediate dimensional representation, and b 2 is the bias term of the second layer;

[0115] For the features of each layer, the corresponding attention weights are calculated through a two-layer fully connected network;

[0116] According to the attention weights of the features of each layer, they are weighted and fused to obtain the final convolutional feature map;

[0117] A pre-trained convolutional neural network is used to extract features from the comprehensive image, obtaining low-level, middle-level, and high-level features, which comprehensively cover multi-level information such as the edges, textures, shapes, and semantics of the package surface. The global context representations of different-level features are calculated through the global context attention mechanism, and each feature is weighted and fused according to the attention weights, ensuring the effective combination of low-level detail features and high-level semantic features. This feature extraction method not only enhances the expression ability of deep features but also enables the system to more accurately capture the complex defect characteristics of the package body.

[0118] The normal feature reconstruction module uses an autoencoder network to reconstruct the original feature map of the normal package body;

[0119] The convolutional feature map is input into the encoder, and the dimension of the convolutional feature map is gradually compressed through multiple fully connected layers, and finally the latent space representation is output;

[0120] The latent space representation is input into the decoder, and the decoder gradually expands the feature dimension through multiple fully connected layers until the original input dimension is restored to obtain the original feature map.

[0121] The reconstruction error calculation module calculates the reconstruction error between the original feature map and the convolutional feature map, sets an error threshold, and judges whether the package body has defects;

[0122] The mean square error is used to calculate the reconstruction error between the original feature map and the convolutional feature map, and the formula is:

[0123]

[0124] Among them, L(x) is the reconstruction error, and F(x) i is the i-th element of the original feature map, is the i-th element of the convolutional feature map, and n is the total number of elements of the feature map;

[0125] The training objective of the autoencoder is to minimize the reconstruction error so that it can effectively reconstruct the features of normal samples, and its loss function uses the mean of the reconstruction error;

[0126] Train the autoencoder using normal sample data to ensure that the model learns the feature patterns of normal samples;

[0127] Update the parameters of the autoencoder using the Adam optimizer;

[0128] Through multiple iterations and backpropagation, minimize the reconstruction error and optimize the parameters of the model;

[0129] Apply the trained autoencoder network to the normal packages in the validation set, calculate the reconstruction error of each package, record the reconstruction errors of all normal samples, and form a reconstruction error distribution;

[0130] Calculate the mean and standard deviation of the reconstruction error, and statistically analyze the distribution characteristics of the reconstruction error. The formula is

[0131]

[0132] where μ is the mean of the reconstruction error, L(x j ) is the reconstruction error of the j-th sample, m is the number of normal samples in the validation set, and σ is the standard deviation of the reconstruction error;

[0133] Set the error threshold based on the distribution characteristics of the reconstruction error. The formula is

[0134] τ = μ + k·σ;

[0135] where τ is the error threshold, μ is the mean of the reconstruction error, σ is the standard deviation of the reconstruction error, and k is a hyperparameter;

[0136] If the reconstruction error is greater than the error threshold, it indicates that there is a defect in the surface mount package.

[0137] Through feature reconstruction and calculation of the reconstruction error, accurate determination of package defects is achieved, solving the problem of high false positive rate in traditional defect detection and improving the detection ability for small and irregular defects.

[0138] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A semiconductor packaging defect detection system based on machine vision, characterized in that: include, The multispectral acquisition module uses a multispectral camera and a light source to collect multi-band images of the package surface from different angles and pre-process the multi-band images; The image fusion module uses the image fusion algorithm to fuse the preprocessed multi-band images into a comprehensive image; The illumination adjustment module performs illumination quality analysis on the comprehensive image, adjusts the light source parameters based on the quality analysis results, and re-collects multi-band images; The feature extraction module uses a convolutional neural network to obtain the features of the comprehensive image and obtain a convolution feature map; The normal feature reconstruction module uses an autoencoder network to reconstruct the original feature map of the normal encapsulation; The reconstruction error calculation module calculates the reconstruction error between the original feature map and the convolution feature map, sets the error threshold, and determines whether the package has defects.

2. The machine vision-based semiconductor packaging defect detection system according to claim 1, characterized in that: The multi-spectral camera and light source are used to collect multi-band images of the package surface from different angles, including: Select a multispectral camera that covers visible light, near infrared and ultraviolet light bands, and install a dedicated multispectral lens; An adjustable multi-band LED light source is selected and deployed in a ring array around the multi-spectral camera; A mechanical rotating platform is configured to image the package at different angles. The control of the rotating platform is synchronously connected with the multi-spectral camera and the multi-band LED light source, and the multi-spectral camera and the multi-band LED light source are triggered to perform imaging at each angle position. Images are collected sequentially at all angles to obtain multi-band images.

3. The machine vision-based semiconductor packaging defect detection system according to claim 2, characterized in that: The preprocessing of the multi-band image includes: The Gaussian noise model and Gaussian filter are used to denoise the multi-band image to obtain the denoised image; Perform histogram equalization on the denoised image of each band to obtain an image with enhanced contrast; The images after denoising and histogram equalization are stored in band units.

4. The machine vision-based semiconductor packaging defect detection system according to claim 3, characterized in that: The image fusion algorithm is used to fuse the pre-processed band images into a comprehensive image, including: The image fusion method of wavelet transform is adopted to decompose the images of each band by wavelet transform to obtain low-frequency components and high-frequency components. For low-frequency components, weighted average fusion is used, and for high-frequency components, maximum value selection is used. At each pixel position, the maximum value of the high-frequency component in all bands is selected to retain the most significant details. After the fusion of low-frequency components and high-frequency components is completed, inverse wavelet transform reconstruction is performed to combine the fused low-frequency components and high-frequency components into a complete fused image; The Laplacian operator is used to enhance the details of the fused image; The Laplace enhanced image is superimposed with the original fused image to obtain the final composite image with enhanced details.

5. The machine vision-based semiconductor packaging defect detection system according to claim 4, characterized in that: The light quality analysis of the comprehensive image includes: Perform brightness analysis on the fused composite image, calculate the gray value of each pixel in the composite image, which is defined as the composite image brightness, and then calculate the average brightness of all pixels in the composite image, which is defined as the global brightness. Set the global brightness threshold. When the global brightness exceeds the threshold, the light source brightness needs to be adjusted. Calculate the local brightness standard deviation of the integrated image and divide the integrated image into several sub-areas. When the local brightness standard deviation is too large, it means that the lighting is uneven and the angle and brightness of the light source need to be adjusted. By analyzing the highlight and shadow areas in the composite image through histogram, the proportion of pixels whose brightness values ​​of the composite image are in the extreme range is calculated, and a ratio threshold is set. When the ratio exceeds the threshold, the angle and brightness of the light source need to be adjusted; Analyze the color temperature distribution of the comprehensive image, detect the color temperature by analyzing the RGB channel values ​​of the comprehensive image, calculate the ratio of the RGB channels, set the ratio threshold, and when the ratio of the RGB channels is not within the threshold range, it is necessary to adjust the color temperature of the light source.

6. The machine vision-based semiconductor packaging defect detection system according to claim 5, characterized in that: The light source parameters are adjusted based on the quality analysis results and the multi-band images are re-collected, include, When the lighting quality analysis results show that the global brightness is out of range, the lighting is uneven, there are too many highlights and shadows, or the color temperature is inappropriate, if any of the conditions are met, the brightness, angle, and color temperature of the light source are adaptively adjusted; After the brightness, angle and color temperature of the light source are adjusted, the multi-band image is collected again; The re-collected multi-band images are re-processed and fused again to obtain a new composite image; Perform lighting quality analysis on the new composite image to verify whether the adjusted lighting meets the preset requirements. If the lighting quality still does not meet the requirements, continue to repeat the adjustment process until the lighting uniformity, brightness range, and color temperature requirements are met to obtain the final composite image.

7. The machine vision-based semiconductor packaging defect detection system according to claim 6, characterized in that: The convolutional neural network is used to obtain the features of the comprehensive image to obtain a convolutional feature map, including: Use a pre-trained convolutional neural network as a feature extractor, input the comprehensive image into the feature extractor, and extract low-level features, mid-level features, and high-level features; The low-level features are the edges, corners, and textures of the comprehensive image; the middle-level features are the texture, shape, and basic structure of the comprehensive image; and the high-level features are the semantics and global information of the comprehensive image. Perform linear transformations on low-level features, middle-level features, and high-level features respectively, so that features at different levels have the same dimension; The global context attention mechanism is used to calculate the global context representation of each level feature, and the global context vector corresponding to the low-level features, middle-level features and high-level features is obtained through global average pooling. The expression is: Among them, g(x) is the context vector, F l (x) is the feature map of the lth layer, H l is the height of the feature map of the lth layer, W l is the width of the feature map of the lth layer, F(x) i,j is the value of the feature map at position (i, j); Calculate the attention weight for the global context vector of each level, and map the context vector of each level to a scalar of the attention weight of the feature at that level through a two-layer fully connected network; The features at each level are weighted and fused according to their attention weights to obtain the final convolutional feature map.

8. The machine vision-based semiconductor packaging defect detection system according to claim 7, characterized in that: The original feature map of the normal encapsulation body reconstructed by the autoencoder network includes: The convolution feature map is input into the encoder, and the dimension of the convolution feature map is gradually compressed through multiple layers of fully connected layers, and finally the latent space representation is output; The latent space representation is input into the decoder, and the decoder gradually expands the feature dimension through multiple layers of fully connected layers until the original input dimension is restored to obtain the original feature map.

9. The machine vision-based semiconductor packaging defect detection system according to claim 8, characterized in that: The calculation of the reconstruction error between the original feature map and the convolution feature map includes: The mean square error is used to calculate the reconstruction error between the original feature map and the convolution feature map. The formula is: Where L(x) is the reconstruction error, F(x) i is the i-th element of the original feature map, is the i-th element of the convolution feature map, and n is the total number of elements in the feature map.

10. The machine vision-based semiconductor packaging defect detection system according to claim 9, characterized in that: Set the error threshold to determine whether the package is defective, including: Apply the trained autoencoder network to the normal encapsulations in the validation set, calculate the reconstruction error of each encapsulation, record the reconstruction errors of all normal samples, and form a reconstruction error distribution; Calculate the mean and standard deviation of the reconstruction error and statistically analyze the distribution characteristics of the reconstruction error; An error threshold is set based on the distribution characteristics of the reconstruction error. When the reconstruction error is greater than the threshold, the surface package has defects.

Images