A surface mount inspection system in a semiconductor package

Abstract

The application discloses a surface mount detection system in a semiconductor package, and relates to the field of automatic detection, comprising a data acquisition module, which acquires temperature information, topographic images and reflected light images of the surface of the semiconductor package in real time, and collects historical data of the semiconductor package; a pretreatment module, which pretreats the acquired temperature information, topographic images and reflected light images; a data fusion module, which fuses the pretreated information and images to form a multimodal feature vector; a three-dimensional reconstruction module, which generates a three-dimensional image of the semiconductor package through a three-dimensional reconstruction algorithm and a multimodal data set; and a defect identification module, which establishes a defect identification model based on the multimodal feature vector and outputs a defect identification result. Through multi-scale analysis of the CNN, the application can capture defects at different scales, and dynamically adjusts the weight of each mode through a self-attention mechanism, so that the detection strategy can be flexibly adjusted according to the actual situation, and the accuracy and flexibility of defect identification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automated detection, and in particular to a surface mount detection system in semiconductor packaging. BACKGROUND

[0002] With the rapid development of the semiconductor industry, integrated circuit packaging technology is constantly improving, especially in the field of surface mount technology, the packaging size is constantly shrinking, and the packaging density and complexity are gradually increasing. Advanced packaging technologies such as BGA, CSP and Flip-Chip have become core components in modern electronic devices.

[0003] The existing surface mount detection technology has several shortcomings. First, single modal detection technology can only obtain single aspect features of the packaging surface, and it is difficult to meet the simultaneous multi-dimensional detection requirements. In addition, in the existing defect recognition method, a fixed image processing algorithm is usually used for defect extraction and classification, and there is a lack of deep analysis and use of historical data, and it is difficult to fully mine the implicit patterns in normal samples. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a surface mount detection system in semiconductor packaging to solve the problem that single modal detection technology can only obtain single aspect features of the packaging surface, and it is difficult to meet the simultaneous multi-dimensional detection requirements. In addition, in the existing defect recognition method, a fixed image processing algorithm is usually used for defect extraction and classification, and there is a lack of deep analysis and use of historical data, and it is difficult to fully mine the implicit patterns in normal samples.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a surface mount detection system in semiconductor packaging, comprising,

[0008] a data acquisition module, which acquires temperature information, topography images and reflected light images of the surface of the semiconductor packaging in real time, and collects historical data of the semiconductor packaging at the same time;

[0009] a preprocessing module, which pre-processes the acquired temperature information, topography images and reflected light images;

[0010] a multi-modal data fusion module, which fuses the pre-processed information and images to form a multi-modal feature vector;

[0011] a three-dimensional reconstruction module, which generates a three-dimensional image of the semiconductor packaging through a three-dimensional reconstruction algorithm and the multi-modal feature vector;

[0012] The defect identification module establishes a defect identification model based on the multi-modal feature vector set, and outputs a defect identification result;

[0013] The defect classification module classifies the defect identification result output by the defect identification model, and generates a defect classification result;

[0014] The defect positioning module labels the classified defect position on the packaged three-dimensional image, and generates a preliminary detection report;

[0015] The adaptive adjustment module automatically adjusts the parameters of the detection equipment according to the preliminary detection report, and performs secondary detection on the key area to generate a secondary detection result;

[0016] The feedback analysis module inputs the detection data adjusted by the adaptive feedback into the defect identification model for analysis and confirmation again, and finally generates a complete detection report.

[0017] As a preferred scheme of the surface mount detection system in the semiconductor package, the information and the reflected light image of the semiconductor package surface under different polarizing angles are collected in real time, specifically including the following steps,

[0018] The temperature change of the semiconductor package surface is recorded in real time by a thermal imaging device, the high-definition image of the package surface is collected in real time by an optical detection device, and the reflected light image under different polarizing angles is collected by a polarizing detection device.

[0019] As a preferred scheme of the surface mount detection system in the semiconductor package, the information and the image collected are preprocessed, specifically including the following steps,

[0020] The collected temperature change information, high-definition image of the package surface and reflected light image are denoised;

[0021] The pictures from different devices are aligned, registered and enhanced;

[0022] The data and the image are format-converted.

[0023] As a preferred scheme of the surface mount detection system in the semiconductor package, the preprocessed information and image are fused to form a multi-modal feature vector, specifically including the following steps,

[0024] The edge features of the optical image are extracted using a Sobel operator;

[0025] The texture features related to the polarizing angle in the polarized image are extracted using a Gabor filter;

[0026] The temperature gradient features of the temperature of the semiconductor package surface are extracted;

[0027] The edge feature, the texture feature and the temperature gradient feature are fused to form a multi-modal feature vector.

[0028] As a preferred scheme of the surface mount detection system in the semiconductor package, the three-dimensional image of the semiconductor package is generated by a three-dimensional reconstruction algorithm and a multi-modal data set, specifically including the following steps,

[0029] The fused multi-modal feature vector is mapped into a three-dimensional voxel grid, and the multi-modal feature vector is a voxel, each voxel representing a small cube in three-dimensional space;

[0030] The data of different modalities is divided into voxels in three-dimensional space, and each voxel contains comprehensive physical information of its coordinate position in three-dimensional space;

[0031] Using volume rendering technology, each voxel is assigned a color and transparency according to its attribute value, and a three-dimensional visualization image of the package is generated.

[0032] As a preferred scheme of the surface mount detection system in the semiconductor package, the three-dimensional image of the semiconductor package is generated by a three-dimensional reconstruction algorithm and a multi-modal data set, specifically including the following steps,

[0033] The multi-modal data of the normal sample data is extracted from the historical data, and the convolutional neural network is used to extract the features of each normal sample modal data;

[0034] In the feature extraction stage of the convolutional neural network, multi-scale convolution is added, different scale convolution kernels are used to capture features of different scales, and the features of different scales are fused;

[0035] All multi-modal features of normal samples are aggregated to obtain a set of reference feature vectors;

[0036] The multi-modal feature vectors of the normal samples are statistically analyzed dimension by dimension, and the mean and standard deviation of each dimension are calculated;

[0037] Real-time data is extracted in the same way as normal samples in historical data, multi-scale convolution is used in the feature extraction process, and self-attention mechanism is used in the fusion process, the weights are dynamically adjusted according to the importance of each modality, and the final weighted fusion feature vector is generated;

[0038] The weighted fusion feature vector is compared with the reference feature vector to calculate the standardized deviation of the real-time data;

[0039] The standard deviations of each real-time data are aggregated by absolute deviation to generate the overall deviation value of the point;

[0040] Based on historical data, set the deviation threshold using cross-validation;

[0041] When the overall deviation value is greater than the deviation threshold, it is determined that the real-time data of the position has defects and defects.

[0042] As a preferred scheme of the surface mount detection system in the semiconductor package, wherein: the recognition result output by the defect recognition model is classified to generate a defect classification result, specifically including the following steps,

[0043] Collecting the features of all known defect types to form a defect feature library;

[0044] Extracting the features of the anomaly and defect output by the defect recognition model to form a feature vector of the anomaly and defect;

[0045] Calculate the similarity between the feature vector of the anomaly and defect and each known defect in the defect feature library by Euclidean distance, which is expressed as:

[0046]

[0047] Where dist(F,F a ) represents the similarity between the new defect F and the known defect F a , F represents the feature vector of the new defect, F a represents the feature vector of the a-th known defect in the defect feature library, n represents the total dimension of the feature vector, F k represents the k-th feature component of the new defect feature vector F, F a,k represents the k-th feature component of the a-th known defect F a in the defect feature library, and k represents the summation index;

[0048] According to the similarity between the new defect F and the known defect F a , the nearest neighbor method is used to determine the category of the new defect;

[0049] After determining the category of the new defect, output the defect classification result.

[0050] As a preferred scheme of the surface mount detection system in the semiconductor package, wherein: the classified defect position is labeled on the three-dimensional image of the package, and a preliminary detection report is generated, specifically including the following steps,

[0051] The position of the defect is labeled on the three-dimensional image in a visual manner, and the defect position is highlighted in different colors, shapes and markers in the three-dimensional package image;

[0052] And generate a preliminary detection report containing package identification information, defect statistics, defect detailed information and image and label.

[0053] As a preferred scheme of the surface mount detection system in the semiconductor package, wherein: the parameters of the detection equipment are automatically adjusted according to the preliminary detection report, and secondary detection is carried out on the key area to generate a secondary detection result, specifically including the following steps,

[0054] The resolution, detection frequency, light source and angle, and detection method of the detection equipment are automatically adjusted according to the preliminary detection report;

[0055] The position of the defect marked on the three-dimensional image is defined as the key area;

[0056] The key area is detected again by the equipment after adjusting the parameters, and a secondary detection result is generated.

[0057] As a preferred scheme of the surface mount detection system in the semiconductor package, wherein: the detection data adjusted by adaptive feedback is input into the defect recognition model for analysis and confirmation again, and a complete detection report is finally generated, specifically including the following steps,

[0058] The secondary detection result after automatic adjustment is input into the defect recognition model, and all detected defects are finally confirmed, and the defect classification result is updated;

[0059] After feedback analysis and confirmation, a complete detection report containing all detected defect information, detailed analysis of defects, and defect classification results is generated.

[0060] The present application has the following advantages: by fusing the preprocessed data, a multi-modal feature vector is formed, and the extraction of multiple features of the package surface is realized. The fused feature vector can more comprehensively reflect the state of the package. This fusion process enhances the analysis capability, enabling comprehensive judgment based on multiple data sources, ensuring that potential defects are not missed, and improving the comprehensiveness and accuracy of the detection. In addition, by establishing a defect recognition model based on the multi-modal feature vector, using CNN to extract features, and combining multi-scale convolution and self-attention mechanism for analysis, the defect recognition result is output. At the same time, through the comparison of the benchmark vector of historical data and real-time data, abnormalities and defects are identified in a timely manner. Through multi-scale analysis of CNN, defects at different scales can be captured, and the weights of each modality are dynamically adjusted through the self-attention mechanism, ensuring that the detection strategy can be flexibly adjusted according to the actual situation, improving the accuracy and flexibility of defect recognition. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0062] Figure 1 System diagram of surface mount detection system in semiconductor package in embodiment 1.

[0063] Figure 2 Schematic diagram of surface mount detection in semiconductor package in embodiment 1. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0065] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other ways different from those described herein without departing from the scope of the present application, and those skilled in the art can make similar extensions without departing from the concept of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0066] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0067] Embodiment 1, refer to Figure 1 and Figure 2 , the first embodiment of the present application provides a surface mount detection system in semiconductor package, comprising the following steps:

[0068] The data acquisition module acquires the temperature information, topographic image and reflected light image of the surface of the semiconductor package in real time, and collects the historical data of the semiconductor package at the same time;

[0069] The function of the thermal imaging device is to detect the temperature distribution of different regions of the surface of the semiconductor package, and to monitor the local overheating or uneven heat dissipation phenomenon that may occur in the packaging process in real time. The thermal imaging device scans the packaging surface through an infrared sensor, captures its thermal radiation information, and converts it into temperature data;

[0070] Optical inspection equipment is mainly used to obtain high-resolution images of the surface of semiconductor packages, capturing the appearance features, surface quality and physical structure of the packages. Optical inspection equipment can scan in the visible light band and generate high-definition images, helping to detect surface defects such as cracks, scratches, foreign matter, etc.

[0071] Polarized light inspection equipment acquires reflected light images of the package surface at different polarized angles by controlling the polarization angle of the light source. Different polarized angles can reveal the details of the texture, stress distribution and material structure of the package surface. Polarized imaging technology can effectively reveal microscopic stress concentration, bubbles or foreign matter and other problems that are difficult to detect in ordinary optical images.

[0072] Further explanation, by using thermal imaging equipment, optical inspection equipment and polarized light inspection equipment in combination, the system can obtain comprehensive information of the package surface from three different dimensions of temperature, appearance and microstructure. The unique data provided by each device complements each other.

[0073] The pre-processing module pre-processes the collected temperature information, topographic images and reflected light images;

[0074] Through denoising algorithm, the data and images are processed to eliminate or reduce the influence of noise.

[0075] Aligning images from different sources to the same coordinate system so that they can correspond to the same physical point, and performing enhancement processing such as contrast enhancement, edge detection enhancement, etc. to highlight key information in the images.

[0076] The content of format conversion includes standardization of image format (such as conversion from different image formats such as JPEG, PNG to a unified format), adjustment of data resolution and conversion of color space (such as conversion of RGB images to grayscale images or other color spaces).

[0077] Further explanation, pre-processing reduces false positives and false negatives, improves the reliability of detection, and improves the compatibility and efficiency of processing data, reducing the processing bottleneck caused by format incompatibility.

[0078] The data fusion module fuses the pre-processed information and images to form a multi-modal feature vector;

[0079] Sobel operator is a widely used filter for edge detection, which can effectively extract edge information in images by calculating the gradient of images in horizontal and vertical directions. The operator is particularly sensitive to areas with significant brightness changes in the image (i.e. edge parts), and is commonly used to detect surface defects such as object contours, cracks, scratches, etc.

[0080] Sobel operator is applied to the high-definition images collected by the optical detection device. Sobel operator calculates the gradient of the image in horizontal and vertical directions respectively, and generates an edge intensity image by combining the gradient information of the two directions.

[0081] Gabor filter is a frequency domain filter commonly used in texture feature extraction of images. It can capture both spatial and frequency information of images, especially suitable for extracting directional texture and periodic structure in images. For polarized images, the light reflection characteristics at different polarization angles can reveal the texture features and internal stress distribution of the packaging surface material.

[0082] Gabor filter is applied to the reflected light images collected by the polarized detection device. The core of Gabor filter is to decompose the image in direction and frequency, so as to effectively capture the texture features in the polarized image.

[0083] Temperature gradient of semiconductor packaging surface refers to the rate of temperature change between different regions of the packaging surface. Temperature gradient information can reveal the thermal distribution of the packaging surface under working conditions, and help to find potential problems caused by thermal stress, poor heat dissipation or excessive heat concentration.

[0084] Temperature distribution map of the packaging surface is obtained by thermal imaging device, and temperature gradient is calculated.

[0085] After calculating the edge features, texture features and temperature gradient features respectively, they are combined into a multi-modal feature vector through feature fusion algorithm. This feature vector contains key information from different modalities, which can provide more accurate data support for subsequent defect identification and three-dimensional reconstruction.

[0086] Further, multi-modal feature fusion can integrate the data advantages of different modalities, thereby forming a more comprehensive and detailed description of the packaging surface. Edge features, texture features and temperature gradient features each provide key information in terms of physical appearance, material structure and thermal performance, and their fusion has the ability to accurately detect defects from multiple angles. Through such feature fusion, the accuracy and coverage of defect detection can be greatly improved, and the limitations of single modal data can be effectively avoided.

[0087] The three-dimensional reconstruction module generates a three-dimensional image of the semiconductor package through a three-dimensional reconstruction algorithm and a multi-modal feature vector.

[0088] The packaging surface is spatially divided into a plurality of small cubes (i.e. voxels).

[0089] The fused multi-modal feature vector is mapped to these voxels.

[0090] Each voxel contains coordinate information in the three-dimensional space it belongs to, as well as multi-modal feature information (such as edge, texture, and temperature data) at that location.

[0091] According to the spatial coordinates of each voxel, the feature values of the multi-modal data (such as edge intensity, texture direction, and temperature value) are associated with the voxel.

[0092] In three-dimensional space, voxels at different locations contain different physical information.

[0093] For example, a voxel may contain a high temperature gradient and obvious edge features, suggesting that there may be thermal stress concentration and surface cracks at that location.

[0094] According to the attribute values of each voxel (such as edge intensity, texture feature, or temperature gradient), it is assigned a color and transparency.

[0095] For example, voxels with high temperature can be rendered in red, voxels with high edge intensity can be rendered in bright white, and voxels with obvious texture features can be assigned a specific color or directional indication. The setting of transparency can help the inspector to view the internal structure of the package in the three-dimensional image without being blocked by the surface information.

[0096] Further explanation, through volume rendering technology, the three-dimensional voxel grid is converted into an intuitive three-dimensional image, and the physical properties of the package surface can be intuitively understood through color and transparency. This three-dimensional visualization image enhances the efficiency and effectiveness of defect identification, especially in the detection of complex package structures and subtle defects, the three-dimensional image can provide more detailed information.

[0097] The defect identification module establishes a defect identification model based on the multi-modal feature vector and outputs a defect identification result.

[0098] From historical data, extract multi-modal data of normal samples. Multi-modal data usually includes different types of sensory data, such as optical images, polarized images, temperature data, etc. Each modality of data can provide physical information from different angles, which helps to analyze the package surface comprehensively.

[0099] Defects on the package surface can occur at different sizes and scales, so single-scale feature extraction may not be sufficient to capture defects comprehensively. Therefore, multi-scale convolution is introduced in the feature extraction stage of CNN. By using convolution kernels of different sizes, features can be extracted from different scales, and the expression is:

[0100]

[0101] where f i,jrepresents the feature vector of the jth modality of the ith sample, i represents the index of the sample, the ith sample, j represents the index of the modality, the jth modality, Concat() represents merging different feature vectors into a larger vector, usually by column or by row splicing, CNN represents a convolutional neural network, k1 represents the size of the convolution kernel, indicating the size of the first convolution kernel used in the CNN, x i,j represents the original input data of the jth modality of the ith sample, if the input is image data, x i,j may be an image, k2 represents the size of the convolution kernel, indicating the size of the second convolution kernel used in the CNN, which is usually different from k1.

[0102] Small-scale convolution kernel, used to capture subtle features on the encapsulated surface, such as tiny cracks or fine scratches.

[0103] Large-scale convolution kernel, used to capture features in larger areas, such as large temperature gradients or large-area texture changes.

[0104] Fusion of features at different scales ensures that data from each modality can fully capture features at different scales.

[0105] Aggregate features of all normal samples. By aggregating the features of these normal samples, a set of baseline feature vectors is generated, which represent the typical features of normal samples in different modalities, and its expression is:

[0106]

[0107] where F 基准 represents the baseline feature vector obtained by averaging the feature vectors of all normal samples, N represents the total number of normal samples, f i represents the final fusion feature vector of the ith sample.

[0108] Statistical analysis of feature values in each dimension, calculate the mean and standard deviation. The expression for calculating the mean is:

[0109]

[0110] where μ d represents the mean of the dth dimension, representing the average of the feature values of all normal samples in the dth dimension, μ represents the mean, usually used to describe the central tendency of a group of data, d represents the dth dimension of the feature vector, a sample's feature vector may have multiple dimensions, each dimension representing a certain feature, f i,d represents the feature value of the dth dimension of the final fusion feature vector f i .

[0111] The expression for calculating the standard deviation is:

[0112]

[0113] where σ d represents the standard deviation of the dth dimension.

[0114] For real-time data, multi-modal feature extraction is performed in the same way as normal samples. When fusing features, the system introduces a self-attention mechanism to dynamically adjust the weight of each modality. The expression is:

[0115]

[0116] where f 实时 represents the fusion feature vector of real-time data, M represents the number of modalities of this real-time data, α j represents the weight coefficient of the jth modality to the fusion feature vector f 实时 , j represents the index of the modality, the jth modality, f 实时,j represents the real-time fusion feature vector of the jth modality.

[0117] To measure the difference between real-time data and the baseline feature vector, the normalized deviation of each feature dimension is calculated. The expression is:

[0118]

[0119] where z d represents the value of the dth dimension of the real-time feature f 实时,d after normalization, f 实时,d represents the feature value of the real-time fusion feature f 实时 in the dth dimension.

[0120] The normalized deviation vector of real-time data is aggregated to generate an overall deviation value. The expression is:

[0121]

[0122] where D represents the overall deviation value, D 总 represents the total number of dimensions of the feature vector of real-time data, |z d | represents the normalized deviation of the dth feature dimension.

[0123] Cross-validation is used with historical data to determine the deviation threshold T. This threshold is used to distinguish between normal and abnormal data.

[0124] During cross-validation, different thresholds are used to test historical data, and finally a threshold is selected that can maximize the distinction between normal and abnormal data.

[0125] The overall deviation value D of the real-time data is compared with the deviation threshold T.

[0126] If the overall deviation exceeds the threshold, the system determines that the real-time data at that location is abnormal and defective, and outputs the detection result.

[0127] To further explain, multi-scale convolution can capture defect information of different sizes and scales, ensuring comprehensive detection. Furthermore, by calculating the standardized deviation and overall deviation value, the degree of anomaly in real-time data can be quantified, enabling accurate defect identification.

[0128] The defect classification module classifies the defect identification results output by the defect identification model and generates defect classification results.

[0129] A large number of known defects were collected through data analysis, and features were extracted from each sample to form a feature library. Each feature vector in the feature library represents a certain known defect.

[0130] Extract the feature vectors of anomalies or defects output by the defect identification model.

[0131] This feature vector is a description of the defect itself, containing multi-dimensional information such as shape, texture, color, location, and size.

[0132] The similarity between the feature vectors of anomalies and defects and each known defect in the defect feature database is calculated using Euclidean distance. The expression is as follows:

[0133]

[0134] Where, dist(F,F) a ) represents the difference between the new defect F and the known defect F. a The similarity between them, F represents the feature vector of the new defect, F a F represents the feature vector of the a-th known defect in the defect feature library, where n represents the total dimension of the feature vectors. k Let F represent the k-th feature component of the new defect feature vector F. a,k This represents the a-th known defect F in the defect feature library. a The k-th feature component, where a represents the index of the defect feature library and k represents the summation index;

[0135] The similarity between the new defect and each known defect is quantified by calculating the Euclidean distance between the new defect and each known defect.

[0136] The known defect with the smallest distance is selected using the nearest neighbor method.

[0137] Output the classification result of the defect based on the category of the new defect determined by the nearest neighbor method.

[0138] a defect positioning module that labels the classified defect positions on the packaged three-dimensional image and generates a preliminary detection report;

[0139] The detected defect positions are labeled on the three-dimensional model through the coordinate system. Each defect is assigned its accurate coordinates in three-dimensional space (such as (x, y, z)).

[0140] Through analysis of the defect feature vector, the specific location of the defect on the packaged object (such as the packaging surface, internal structure, etc.) is determined.

[0141] According to the type or severity of the defect, different colors are used to highlight the defect. For example, red can represent a serious defect, yellow a minor defect, and green a negligible defect.

[0142] Different types of defects are labeled using different geometric shapes (such as circles, squares, triangles, etc.). The choice of shape can be based on the properties of the defect (such as cracks, holes, bubbles, etc.).

[0143] Next to each defect, a label or tag can be attached to display the defect number, type, size, location, etc.

[0144] After labeling, a preliminary detection report is generated containing packaging identification information, defect statistics, defect detailed information, and image and labeling.

[0145] The packaging identification information refers to product information and packaging information.

[0146] The defect statistics refer to the total number of defects, type distribution, and severity.

[0147] The defect detailed information refers to the defect number, defect position, defect description, and severity.

[0148] The image and labeling refer to three-dimensional image screenshots and defect labeling diagrams.

[0149] Further explanation, through the visualization of three-dimensional images, the spatial position and shape of defects can be more intuitively displayed. Traditional two-dimensional images are difficult to reflect the specific position of defects on the internal or complex surface of an object, while three-dimensional visualization can provide clearer spatial perception. Through high-light display in three-dimensional space, defects can be more effectively labeled, avoiding omission or misjudgment. Especially for defects inside the package, three-dimensional display can better reflect their position and severity. The use of different colors, shapes, and labels facilitates the differentiation of different types and severity of defects, making the detection results more explicit.

[0150] An adaptive adjustment module automatically adjusts the parameters of the detection equipment based on the preliminary detection report and conducts secondary detection on key areas to generate secondary detection results.

[0151] If the preliminary detection shows that some defects are large and obvious, a lower resolution is chosen to save detection time and resources. For small or difficult-to-identify defects, the resolution is automatically increased to obtain higher-quality images in the secondary detection.

[0152] In the preliminary detection, if the defect type in some areas is complex (such as dynamic defects or rapidly changing defects), the detection frequency needs to be increased to capture more images for analysis. For areas that do not require high-frequency detection, the detection frequency is appropriately reduced to reduce data volume and save bandwidth and storage resources.

[0153] Based on the defect type in the preliminary detection report, the intensity, color, angle, and other parameters of the light source are automatically adjusted. For example, for small surface cracks, side light sources may be needed to enhance reflection; for internal bubbles, the position of the light source may need to be adjusted for penetration detection. The most suitable light source settings are automatically selected in the secondary detection to ensure that defects can be captured more clearly.

[0154] Based on the preliminary detection results, different detection methods can be selected to optimize secondary detection according to actual conditions. For example: optical detection, ultrasonic detection, X-ray detection, and electromagnetic detection.

[0155] Key areas are defined based on the severity of defects, the density of defects, and the difficulty of detection.

[0156] After parameter adjustment, secondary detection is conducted on key areas. At this time, higher resolution, higher frequency, or switching to more suitable detection methods (such as X-ray, ultrasonic, etc.) are used to obtain more detection data.

[0157] Further explanation, the purpose of secondary detection is to further confirm the defects found in preliminary detection and obtain more detailed features of defects, to ensure that there is no omission or misjudgment.

[0158] Secondary detection results are generated based on secondary detection.

[0159] The secondary detection results include detailed defect descriptions, correction of preliminary detection errors, high-precision positioning, defect classification, and priority adjustment.

[0160] Further, the function of automatically adjusting equipment parameters can greatly reduce the need for manual intervention. According to the preliminary detection report, the parameters are automatically optimized to ensure that the equipment can run in the best state during the secondary detection, reducing the time and errors of manual adjustment of the equipment. By automatically focusing on key areas, it can focus on the areas that need the most detailed detection while maintaining overall detection efficiency, avoiding global redundant detection.

[0161] The feedback analysis module inputs the detection data adjusted by adaptive feedback into the defect identification model for analysis and confirmation, and finally generates a complete detection report.

[0162] During preliminary detection and secondary detection, some false defects or noise may be detected, which may be caused by equipment errors, changes in lighting conditions, etc. The defect identification model can remove these false positives by further analyzing the detection results, ensuring that the final confirmed defects are real. At the same time, some defects may not be detected in the preliminary detection, but are identified after the secondary detailed detection. The defect identification model will confirm these newly added defects to ensure that important defects are not missed.

[0163] The model may reevaluate the severity of some defects based on new detection results and parameter adjustments. For example, some parts that were considered minor defects in preliminary detection may be reevaluated as serious defects after secondary detection.

[0164] A complete detection report containing defect overview, defect detailed analysis, and updated classification results is generated.

[0165] The defect overview includes the total number of defects, defect classification statistics, and defect severity distribution.

[0166] The defect detailed analysis includes defect location, defect feature description, and defect impact analysis.

[0167] The updated classification results include classification update descriptions.

[0168] Further, through the analysis of the secondary detection results by the defect identification model, defects can be more accurately identified and confirmed, avoiding false positives and omissions. This greatly improves the reliability of detection, making the final confirmed defects more real and accurate.

[0169] To sum up, the application realizes the extraction of multiple features of the packaging surface by forming a multi-modal feature vector through the fusion of pre-processed data. The fused feature vector can more comprehensively reflect the state of the packaging. This fusion process enhances the analysis capability, enabling comprehensive judgment based on multiple data sources, ensuring that potential defects are not missed, and improving the comprehensiveness and accuracy of the detection. In addition, by establishing a defect recognition model based on the multi-modal feature vector, using CNN to extract features, and combining multi-scale convolution and self-attention mechanism for analysis, the defect recognition result is output. At the same time, through the comparison of the benchmark vector of historical data and real-time data, abnormalities and defects can be identified in a timely manner. Through the multi-scale analysis of CNN, defects at different scales can be captured, and the weights of each modality can be dynamically adjusted through the self-attention mechanism, ensuring that the detection strategy can be flexibly adjusted according to the actual situation, improving the accuracy and flexibility of defect recognition.

[0170] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, which should be covered by the scope of the claims of the present application.

Claims

1. A surface mount inspection system in a semiconductor package, characterized by: The application relates to a semiconductor package defect detection method and device. The data acquisition module acquires temperature information, topography images and reflected light images of a semiconductor package surface in real time through a detection device, and collects historical data of the semiconductor package; The preprocessing module pre-processes the acquired temperature information, topography images and reflected light images; The data fusion module fuses the pre-processed temperature information, topography images and reflected light images to form a multi-modal feature vector, and specifically comprises the following steps, the edge features of the topography images are extracted by using a Sobel operator; the texture features related to the polarization angle in the reflected light images are extracted by using a Gabor filter; temperature gradient features of the temperature information of the semiconductor package surface are extracted; the edge features, texture features and temperature gradient features are fused to form a multi-modal feature vector; The three-dimensional reconstruction module generates a three-dimensional image of the semiconductor package through a three-dimensional reconstruction algorithm and the multi-modal feature vector; The defect recognition module establishes a defect recognition model based on the multi-modal feature vector and outputs a defect recognition result; The defect classification module classifies the defect recognition result output by the defect recognition model to generate a defect classification result; The defect positioning module labels the classified defect positions on the three-dimensional image of the package and generates a preliminary detection report; The adaptive adjustment module automatically adjusts the parameters of the detection device according to the preliminary detection report, and performs secondary detection on the temperature information, topography images and reflected light images of the key areas to generate a secondary detection result; The feedback analysis module inputs the detection data adjusted through the adaptive feedback into the defect recognition model for analysis and confirmation, and finally generates a complete detection report.

2. The surface mount inspection system in a semiconductor package of claim 1, wherein: The detection device acquires information of the semiconductor package surface and reflected light images of the semiconductor package surface under different polarization angles in real time, and specifically comprises the following steps, The thermal imaging device records the temperature change of the semiconductor package surface in real time as temperature information, the optical detection device acquires a high-definition image of the package surface in real time as a topography image, and the polarization detection device acquires reflected light images under different polarization angles.

3. The surface mount inspection system in a semiconductor package of claim 2, wherein: The acquired temperature information, topography images and reflected light images are pre-processed, and specifically comprise the following steps, The acquired temperature information, topography images and reflected light images of the package surface are denoised; The topography images and reflected light images are aligned, registered and enhanced; The temperature information, topography images and reflected light images are format-converted.

4. The surface mount inspection system in a semiconductor package of claim 3, wherein: The three-dimensional reconstruction algorithm and the multi-modal data set are used to generate a three-dimensional image of the semiconductor package, and specifically comprise the following steps, The fused multi-modal feature vector is mapped into a three-dimensional voxel grid, and the multi-modal feature vector is a voxel, each voxel representing a small cube in the three-dimensional space; The data of different modalities are divided into voxels in the three-dimensional space, and each voxel contains comprehensive physical information of the coordinate position in the three-dimensional space; The volume rendering technology is used to specify the color and transparency of each voxel according to the attribute value of the voxel, and a three-dimensional visualization image of the package is generated.

5. The surface mount inspection system in a semiconductor package of claim 4, wherein: The multi-modal feature vector is used to establish a defect recognition model, and a defect recognition result is output, and specifically comprises the following steps, Multi-modal data of normal sample data is extracted from historical data, and a convolutional neural network is used to extract features from the data of each normal sample modality; In the feature extraction stage of the convolutional neural network, multi-scale convolution is added to capture features at different scales through convolution kernels of different scales, and the features at different scales are fused; All multi-modal features of normal samples are aggregated to obtain a set of baseline feature vectors; Statistical analysis is performed on the multi-modal feature vectors of normal samples dimension by dimension, and the mean and standard deviation of each dimension are calculated; Real-time data is extracted in the same way as normal samples in historical data, and multi-scale convolution is used in the feature extraction process, and a self-attention mechanism is used in the fusion process to dynamically adjust the weight according to the importance of each modality, generating a final weighted fusion feature vector; By comparing the weighted fusion feature vector with the baseline feature vector, the standard deviation of the real-time data is calculated; The standard deviations of each real-time data are aggregated by absolute deviation to generate an overall deviation value; Based on historical data, the deviation threshold is set using cross-validation; When the overall deviation value is greater than the deviation threshold, it is determined that the real-time data at that position is abnormal and defective.

6. The surface mount inspection system in a semiconductor package of claim 5, wherein: The recognition results output by the defect recognition model are classified to generate a defect classification result, including the following steps, Collect the features of all known defect types to form a defect feature library; Extract the features of the anomaly and defect output by the defect recognition model to form a feature vector of the anomaly and defect; Calculate the similarity between the feature vector of the anomaly and defect and each known defect in the defect feature library by Euclidean distance, which is expressed as: ； wherein, represents the similarity between the new defect and the known defects, represents the feature vector of the new defect, represents the feature vector of the i-th known defect in the defect feature library, represents the i-th feature component of the new defect feature vector represents the i-th feature component of the i-th known defect in the defect feature library represents the summation index;​​​​​​​​ According to the similarity between the new defect and the known defects , the class of the new defect is determined using the nearest neighbor method; After determining the category of the new defect, output the defect classification result.

7. The surface mount inspection system in a semiconductor package of claim 6, wherein: Label the classified defect positions on the three-dimensional image of the package and generate a preliminary detection report, including the following steps, Label the defect positions on the three-dimensional image through visualization, and highlight the defect positions in different colors, shapes and markers in the three-dimensional package image; And generate a preliminary detection report containing package identification information, defect statistics, defect details and images and annotations.

8. The surface mount inspection system in a semiconductor package of claim 7, wherein: According to the preliminary detection report, automatically adjust the parameters of the detection equipment, and perform secondary detection on the temperature information, topographic image and reflected light image of the key area to generate a secondary detection result, including the following steps, According to the preliminary detection report, automatically adjust the resolution, detection frequency and light source and angle of the detection equipment; Define the positions of the defects labeled on the three-dimensional image as key areas; According to the adjusted parameters, the equipment detects the temperature information, topographic image and reflected light image of the key area again, and generates a secondary detection result.

9. The surface mount inspection system in a semiconductor package of claim 8, wherein: Input the detection data adjusted by adaptive feedback into the machine learning model for analysis and confirmation, and finally generate a complete detection report, including the following steps, Input the secondary detection result adjusted automatically into the defect recognition model, and finally confirm all detected defects, while updating the defect classification result; After feedback analysis and confirmation, a complete test report containing all the detected defect information, detailed analysis of defects, and defect classification results is generated.

Images