An automatic detection method for chip appearance defects based on machine vision

Abstract

The present invention discloses an automatic detection method for chip appearance defects based on machine vision, which relates to the technical field of chip appearance detection. The present invention combines a dynamic lighting strategy and multi-modal imaging technology to collect multi-view images of the surface and interior of the chip under multi-angle and multi-spectral conditions, enhancing the saliency of low-contrast defects and making up for the limitations of single-modal imaging in dealing with defects of complex structures; through the Retinex algorithm to correct uneven lighting, the CLAHE technology to enhance local contrast, and multi-scale decomposition to highlight edge and detail features, significantly improving the accuracy of feature extraction; based on the lightweight MobileNet model, combining multi-scale feature extraction and dynamic threshold to generate the region of interest ROI, realizing sub-pixel level positioning and classification of low-contrast defects, effectively coping with the interference of complex backgrounds on the detection accuracy; at the same time, through a cross-modal feature fusion framework, further integrating terahertz, infrared and visible light features to improve the accuracy of defect classification.

Description

Technical Field

[0001] The present invention relates to the technical field of chip appearance detection, and in particular to an automatic detection method for chip appearance defects based on machine vision. Background Art

[0002] During the chip manufacturing process, low-contrast defects such as bubbles, delamination, and cracks often appear inside the packaging material, which will have an important impact on the electrical performance, reliability, and service life of the chip. Therefore, automated defect detection technology is particularly important in the chip manufacturing process.

[0003] Traditional chip appearance defect detection methods are mostly based on non-contact machine vision technology. Chip surface images are obtained through optical imaging, and defect recognition is carried out by combining feature extraction and rule algorithms. To a certain extent, significant defects are detected, but the recognition effect for low-contrast defects is not good. The features of low-contrast defects are not obvious enough, and the difference from background noise is small. Traditional methods are easily affected by environmental illumination, lens noise, and surface texture complexity during the image processing stage, resulting in blurred boundaries or even difficulty in identifying defect areas; in addition, traditional methods rely strongly on defect features and cannot effectively adapt to defect types with diverse shapes or irregular distributions.

[0004] In dealing with low-contrast defects, traditional methods mainly optimize the imaging system and enhance the image processing algorithm to improve the saliency of the target area. However, these optimization methods have limitations in terms of cost and real-time performance and cannot meet the dual requirements of detection accuracy and speed; therefore, there is an urgent need for an automatic detection method for chip appearance defects based on machine vision to solve such problems. Summary of the Invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides an automatic detection method for chip appearance defects based on machine vision to solve the problems that the features of low-contrast defects are not obvious enough, the difference from background noise is small, and traditional methods are easily affected by environmental illumination, lens noise, and surface texture complexity during the image processing stage, resulting in blurred boundaries of defect areas and difficulty in identification.

[0007] To solve the above technical problems, the present invention provides the following technical solutions:

[0008] An embodiment of the present invention provides an automatic detection method for chip appearance defects based on machine vision, which includes,

[0009] Step S1, in the pipeline environment of chip manufacturing, image the surface and interior of the chip through an industrial camera and a multispectral imaging device. During the imaging process, dynamically adjust the lighting strategy, collect images under multi-angle and multi-intensity conditions, and generate multi-view chip images;

[0010] Step S2, preprocess the chip images, including lighting uniformity, contrast enhancement, and separation of multi-scale features;

[0011] Step S3, input the images preprocessed in Step S2 into a deep learning model for feature extraction and classification, and generate a region of interest ROI; the deep learning model here uses a lightweight MobileNet model;

[0012] Step S4, perform a refined analysis on the generated region of interest ROI;

[0013] Step S5, transfer the analysis results of Step S4 to an edge computing device for real-time evaluation and production closed-loop: based on the defect type, measured size, and position obtained from the analysis in Step S4, divide the risk level, prioritize the detection and review of high-risk areas, and then send the detection results back to the production pipeline for adjusting process parameters or marking chips that require additional processing.

[0014] As a preferred solution of the automatic chip appearance defect detection method based on machine vision described in the present invention, wherein: in the step of imaging the surface and interior of the chip through an industrial camera and a multispectral imaging device,

[0015] Perform multi-view dynamic imaging, and the imaging formula is:

[0016] I k =φ(S,λ k ,θ k ,L k ),

[0017] wherein, I k represents the k-th image collected, S represents the target area on the surface or interior of the chip, λ k represents the k-th spectral band used for imaging, θ k represents the viewing angle parameter during imaging, L k represents the dynamic lighting intensity during imaging, φ(·) represents the imaging response function, and by modeling the spectral characteristics, spatial angles, and dynamic lighting intensity, convert the surface and interior information of the chip into a two-dimensional image;

[0018] Adopt timing control to adjust the lighting strategy, and the lighting adjustment formula is:

[0019] L k =L0·(1 + β·sin(ω k t)),

[0020] Among them, L k is the current light intensity, L0 is the initial light intensity, β is the light intensity change amplitude coefficient, ω k represents the frequency regulation factor related to the wavelength band λ k , and t is the current imaging time point;

[0021] Multi-angle imaging fusion is performed, and the fusion formula is:

[0022]

[0023] Among them, I fused represents the multi-angle imaging image after fusion, w k represents the weight parameter related to the angle and spectrum, and N is the total number of imaging wavelength bands and viewing angles.

[0024] As a preferred solution of the automatic chip appearance defect detection method based on machine vision according to the present invention, in which: in step S2, the preprocessing method includes:

[0025] Based on the Retinex algorithm, correct the brightness imbalance phenomenon caused by uneven illumination,

[0026] Adopt the contrast-limited adaptive histogram equalization (CLAHE) technique to selectively enhance the saliency of low-contrast regions,

[0027] And use the multi-scale decomposition method to decompose the image into high-frequency and low-frequency components. The high-frequency components are used to highlight the edge features of low-contrast defects, and the low-frequency components retain the global structure information.

[0028] As a preferred solution of the automatic chip appearance defect detection method based on machine vision according to the present invention, in which: the step of preprocessing the chip image is

[0029] Perform illumination homogenization processing, and adjust it based on the Retinex algorithm. The adjustment formula is:

[0030]

[0031] Among them, I norm (x, y) represents the image after illumination correction, I raw (x, y) represents the original image, G(x, y) is the multi-scale Gaussian kernel, and * represents the convolution operation;

[0032] Enhance the local contrast based on the CLAHE technique. The enhancement formula is:

[0033] I enh (x, y) = CLAHE(I norm (x, y), c, g),

[0034] Among them, I enh (x, y) is the enhanced image, c is the contrast limit parameter, g is the grid size parameter, and CLAHE represents the local enhancement process.

[0035] Obtain features through high-frequency and low-frequency component decomposition:

[0036] I low (x, y) = G(x, y) * I enh (x, y), I high (x, y) = I enh (x, y) - I low (x, y),

[0037] Among them, I low (x, y) is the low-frequency component, retaining the global structure information, I high (x, y) is the high-frequency component, highlighting the edge and detail features.

[0038] As a preferred solution of the automatic chip appearance defect detection method based on machine vision described in the present invention, wherein: the feature extraction and classification method is: extracting multi-scale features through the convolutional layer to identify possible defect areas.

[0039] The method for generating the region of interest ROI is: dynamically defining the region of interest according to the output of the MobileNet model, and positioning the low-contrast defect to the sub-pixel accuracy range.

[0040] As a preferred solution of the automatic chip appearance defect detection method based on machine vision described in the present invention, wherein: the step of inputting the image preprocessed in step S2 into the deep learning model for feature extraction and classification is

[0041] Input the preprocessed image into the MobileNet model, and the model input is expressed as:

[0042] F l = σ(W l * F l-1 + b l ),

[0043] Among them, F l represents the feature map of the l-th layer, W l represents the convolutional kernel weight matrix of the l-th layer, F l-1 represents the input feature map of the (l - 1)-th layer, b l represents the bias term of the l-th layer, and σ is the non-linear activation function.

[0044] Perform global feature aggregation, and output the class probability by the classification layer. The class calculation formula is:

[0045]

[0046] Among them, P(c|F) is the probability of class c, and W c and W i respectively represent the weight vectors corresponding to the classes, F is the input feature of the fully connected layer, and b c and b i respectively represent the bias terms related to the classes.

[0047] As a preferred solution of the automatic chip appearance defect detection method based on machine vision according to the present invention, wherein: the step of generating the region of interest ROI is as follows,

[0048] Generate the region of interest based on the output of MobileNet, and the generation formula of the region of interest is:

[0049]

[0050] Among them, score(x,y) is the score of the region of interest at coordinates (x,y), and F k (x,y) is the eigenvalue corresponding to the k-th channel, α k is the importance weight of channel k, and K is the total number of channels;

[0051] Generate the boundary of the region of interest based on the dynamic threshold, and the boundary formula ROI is:

[0052] ROI = {(x,y,w,h)|score(x,y)>T, (x,y)∈I enh},

[0053] wherein, (x,y,w,h) represents the center coordinates and the boundary box size of the region of interest, T is the dynamically generated threshold, T = γ·max(score(x,y)), and γ is the threshold adjustment coefficient,

[0054] Optimize the ROI boundary at the sub-pixel level, and the optimization formula is:

[0055]

[0056] wherein, (x opt ,y opt ) is the center coordinates of the optimized ROI,

[0057] E loss (x,y) = ∑ i (score(x,y)-pred i ) 2 is the loss function of the optimization target,

[0058] pred iRepresents a set of predicted values.

[0059] As a preferred solution of the automatic chip appearance defect detection method based on machine vision according to the present invention, wherein: the way of refinement analysis includes:

[0060] Construct a cross-modal learning framework, jointly model the features of terahertz, infrared and visible light images to obtain a fused feature map,

[0061] Use the fused feature map to classify the defect types, and the types include bubbles, cracks and material delamination,

[0062] Use the fused feature map to measure the geometric size of the defect and its specific position on the chip.

[0063] As a preferred solution of the automatic chip appearance defect detection method based on machine vision according to the present invention, wherein: the step of performing refinement analysis on the generated region of interest ROI is,

[0064] Perform cross-modal feature fusion, and jointly model the features of the terahertz image (F THz ), infrared image (F IR ), and visible light image (F VIS ), and the modeling formula is:

[0065] F fusion =ψ(F THz ,F IR ,F VIS ),

[0066] Among them, F fusion represents the fused feature map, F THz ,F IR ,F VIS respectively represent the feature maps of the three modalities, and ψ represents the cross-modal fusion function:

[0067]

[0068] α i is the weight of modality i;

[0069] Perform defect type classification, input the fused features into the classifier to discriminate the defect types, and the calculation formula of the classification result is:

[0070]

[0071] Among them, C is the predicted defect type, W c is the weight matrix of category c in the classifier, and b c is the classifier bias term;

[0072] Measure the geometric dimensions of the ROI boundary and calculate the dimensions using the optimized boundary points:

[0073]

[0074] Among them, D is the geometric dimension of the defect, and (x1, y1) and (x2, y2) are the two boundary points of the defect respectively;

[0075] Use cross-modal features to determine the specific location of the defect in the chip:

[0076] P defect =(x center ,y center ),

[0077] Among them, P defect represents the center coordinates of the defect, and x center and y center are calculated by the midpoint formula of the boundary points respectively;

[0078] Calculate the defect depth feature, combine the depth information of the terahertz mode, and calculate the depth distribution of the defect:

[0079] Z defect =f depth (F THz ),

[0080] Among them, Z defect is the defect depth value, and f depth represents the mapping function for extracting the depth feature from the terahertz mode.

[0081] As a preferred solution of the automatic chip appearance defect detection method based on machine vision described in the present invention, among them: the step of transmitting the analysis result of step S4 to the edge computing device for real-time evaluation and production closed-loop is,

[0082] Based on the defect type, size and location, divide the defect risk level, and define the comprehensive risk score formula as:

[0083] R = ω T ·T + ω D ·D + ω P ·ΔP,

[0084] Among them, R is the comprehensive risk score of the defect, ω T , ω D , ω P are the weight factors of the defect type, size and location, T represents the risk coefficient corresponding to the defect type, D is the geometric dimension of the defect, and ΔP is the distance between the defect center position and the ideal position,

[0085] Pack the analysis result into a data stream Dstream , transfer to the edge computing device,

[0086]

[0087] where d stream is a data stream containing the analysis results of each ROI, and (T, D, P defect , R) represent the defect type, size, position, and risk score respectively,

[0088] According to the risk score R, dynamically adjust the production parameter P update , and the adjustment formula is:

[0089] P update = P init ·(1 + κR),

[0090] where P update is the updated process parameter, P init is the initial process parameter, and κ is the risk impact factor,

[0091] where for high-risk regions R > R thresh perform a review:

[0092] where R thresh is the threshold of the risk level.

[0093] The beneficial effects of the present invention are as follows: The present invention combines a dynamic lighting strategy and multi-modal imaging technology to collect multi-view images of the chip surface and interior under multi-angle and multi-spectral conditions, enhancing the saliency of low-contrast defects and making up for the limitations of single-modal imaging in dealing with complex structure defects; through the Retinex algorithm to correct uneven lighting, the CLAHE technique to enhance local contrast, and multi-scale decomposition to highlight edge and detail features, significantly improving the accuracy of feature extraction; based on the lightweight MobileNet model, combined with multi-scale feature extraction and dynamic threshold to generate regions of interest ROI, realizing sub-pixel-level localization and classification of low-contrast defects, effectively coping with the interference of complex backgrounds on detection accuracy; at the same time, through a cross-modal feature fusion framework, further integrating terahertz, infrared, and visible light features, improving the accuracy of defect classification, and combining geometric measurement and depth feature analysis to determine the type, size, and spatial position of defects; finally, perform risk level classification, introduce a real-time feedback mechanism, transfer the analysis results to the edge computing device, give priority to detecting high-risk regions and perform process adjustment, greatly enhancing the adaptability of the production line to complex defects. Description of the Drawings

[0094] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0095] Figure 1 It is a schematic flowchart of the automatic detection method for chip appearance defects based on machine vision of the present invention. Specific embodiments

[0096] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will make a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification.

[0097] In the following description, many specific details are set forth to facilitate a thorough understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0098] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0099] Embodiment 1, referring to Figure 1 , this embodiment provides an automatic detection method for chip appearance defects based on machine vision, including the following steps:

[0100] Step S1, in the pipeline environment of chip manufacturing, use an industrial camera and a multispectral imaging device to image the surface and interior of the chip. During the imaging process, dynamically adjust the lighting strategy, and collect images under multi-angle and multi-intensity conditions to generate multi-view chip images;

[0101] In the step of using an industrial camera and a multispectral imaging device to image the surface and interior of the chip,

[0102] Perform multi-view dynamic imaging, and the imaging formula is:

[0103] I k =φ(S,λ k ,θ k ,L k ),

[0104] where, I kThe k-th collected image is denoted as, S represents the target area on or inside the chip, and λ k The k-th spectral band for imaging is denoted as, and θ k The viewing angle parameter during imaging is denoted as, and L k The dynamic light intensity during imaging is denoted as, and φ(·) represents the imaging response function. By modeling the spectral characteristics, spatial angles, and dynamic light intensity, the information on the chip surface and inside is converted into a two-dimensional image;

[0105] The light illumination strategy is adjusted using timing control, and the light illumination adjustment formula is:

[0106] L k = L0·(1 + β·sin(ω k t)),

[0107] where, L k is the current light intensity, L0 is the initial light intensity, β is the light intensity change amplitude coefficient, and ω k represents the frequency regulation factor related to the band λ k and t is the current imaging time point;

[0108] Multi-angle imaging fusion is performed, and the fusion formula is:

[0109]

[0110] where, I fused represents the multi-angle imaging image after fusion, w k represents the weight parameter related to angles and spectra, and N is the total number of imaging bands and viewing angles;

[0111] Specifically, in this step, the light intensity is dynamically adjusted, and the surface and internal structure imaging of the chip is constructed by combining multi-spectral and multi-angle imaging technologies; a dynamic light intensity control and fusion weight adjustment mechanism is introduced here to make the imaging results more adaptable to the detection requirements of diverse defect characteristics.

[0112] Step S2: Preprocess the chip image, including light illumination uniformity, contrast enhancement, and separation of multi-scale features;

[0113] In step S2, the preprocessing methods include:

[0114] Based on the Retinex algorithm, correct the brightness imbalance phenomenon caused by uneven light illumination,

[0115] Adopt the Contrast Limited Adaptive Histogram Equalization (CLAHE) technique to selectively enhance the saliency of low-contrast regions,

[0116] And using the multi-scale decomposition method, the image is deconstructed into high-frequency and low-frequency components. The high-frequency components are used to highlight the edge features of low-contrast defects, and the low-frequency components retain the global structure information;

[0117] The steps for preprocessing the chip image are as follows:

[0118] Perform illumination equalization processing, and adjust using the Retinex algorithm. The adjustment formula is:

[0119]

[0120] Among them, I norm (x, y) represents the image after illumination correction, I raw (x, y) represents the original image, G(x, y) is the multi-scale Gaussian kernel, and * represents the convolution operation;

[0121] Enhance the local contrast based on the CLAHE technology. The enhancement formula is:

[0122] I enh (x, y) = CLAHE(I norm (x, y), c, g),

[0123] Among them, I enh (x, y) is the enhanced image, c is the contrast limit parameter, g is the grid size parameter, and CLAHE represents the local enhancement process.

[0124] Obtain features through high-frequency and low-frequency component decomposition:

[0125] I low (x, y) = G(x, y) * I enh (x, y), I high (x, y) = I enh (x, y) - I low (x, y),

[0126] Among them, I low (x, y) is the low-frequency component, retaining the global structure information, and I high (x, y) is the high-frequency component, highlighting the edge and detail features;

[0127] Specifically, step S2 combines illumination equalization, contrast enhancement, and multi-scale decomposition techniques. Through the multi-scale Gaussian kernel and CLAHE, it enhances the saliency of low-contrast regions and separates high-frequency and low-frequency features simultaneously.

[0128] In step S3, the image preprocessed in step S2 is input into the deep learning model for feature extraction and classification, and the region of interest ROI is generated; the deep learning model here uses the lightweight MobileNet model;

[0129] The feature extraction and classification method is as follows: extract multi-scale features through the convolutional layer to identify possible defect areas,

[0130] The method for generating the region of interest (ROI) is as follows: dynamically define the region of interest according to the output of the MobileNet model, and locate low-contrast defects within the sub-pixel accuracy range;

[0131] The steps of inputting the image preprocessed in step S2 into the deep learning model for feature extraction and classification are,

[0132] Input the preprocessed image into the MobileNet model, and the model input is expressed as:

[0133] F l = σ(W l * F l-1 + b l ),

[0134] where F l represents the feature map of the l-th layer, W l represents the convolutional kernel weight matrix of the l-th layer, F l-1 represents the input feature map of the (l - 1)-th layer, b l represents the bias term of the l-th layer, and σ is the non-linear activation function,

[0135] Perform global feature aggregation, and output the class probability by the classification layer. The class calculation formula is:

[0136]

[0137] where P(c|F) is the probability of class c, W c and W i respectively represent the weight vectors corresponding to the classes, F is the input feature of the fully connected layer, b c and b i respectively represent the bias terms related to the classes,

[0138] The steps for generating the region of interest (ROI) are,

[0139] Generate the region of interest based on the MobileNet output. The generation formula for the region of interest is:

[0140]

[0141] where score(x, y) is the score of the region of interest at coordinates (x, y), F k (x, y) is the eigenvalue corresponding to the k-th channel, α k is the importance weight of channel k, and K is the total number of channels;

[0142] Generate the boundary of the region of interest (ROI) based on a dynamic threshold. The boundary formula for ROI is:

[0143] ROI = {(x, y, w, h)|score(x, y)>T, (x, y) ∈ I enh},

[0144] where (x, y, w, h) represents the center coordinates and bounding box size of the region of interest, T is the dynamically generated threshold, T = γ·max(score(x, y)), and γ is the threshold adjustment coefficient.

[0145] Perform sub-pixel level optimization on the ROI boundary. The optimization formula is:

[0146]

[0147] where (x opt , y opt ) is the center coordinate of the optimized ROI.

[0148] E loss (x, y) = ∑ i (score(x, y)-pred i ) 2 is the loss function of the optimization target.

[0149] pred i represents the set of predicted values.

[0150] Specifically, extract multi-scale features through a lightweight MobileNet model, combine convolutional operations with non-linear activation functions to deeply mine the features of the chip image, and improve the accuracy of ROI extraction and effectively capture low-contrast defect regions through a dynamic threshold generation mechanism and a sub-pixel optimization algorithm.

[0151] Step S4: Refine and analyze the generated region of interest ROI.

[0152] The ways of refinement and analysis include:

[0153] Construct a cross-modal learning framework to jointly model the features of terahertz, infrared, and visible light images to obtain a fused feature map.

[0154] Use the fused feature map to classify the defect types, including bubbles, cracks, and material delamination.

[0155] Use the fused feature map to measure the geometric size of the defect and its specific location on the chip.

[0156] The steps for refining and analyzing the generated region of interest ROI are as follows.

[0157] Perform cross-modal feature fusion, jointly model the features of terahertz images (F THz ), infrared images (F IR ), and visible light images (F VIS ), and the modeling formula is:

[0158] F fusion = ψ(F THz , F IR , F VIS ),

[0159] where F fusion represents the fused feature map, F THz , F IR , F VIS represent the feature maps of the three modalities respectively, and ψ represents the cross-modal fusion function:

[0160]

[0161] α i is the weight of modality i;

[0162] Perform defect type classification, input the fused features into the classifier for defect type discrimination, and the calculation formula for the classification result is:

[0163]

[0164] where C is the predicted defect type, W c is the weight matrix of class c in the classifier, and b c is the classifier bias term;

[0165] Measure the geometric size of the ROI boundary, and calculate the size using the optimized boundary points:

[0166]

[0167] where D is the geometric size of the defect, and (x1, y1) and (x2, y2) are the two boundary points of the defect respectively;

[0168] Use cross-modal features to determine the specific location of the defect in the chip:

[0169] P defect = (x center , y center ),

[0170] where P defect represents the center coordinates of the defect, and x center and y center are calculated respectively through the midpoint formula of the boundary points;

[0171] Calculate the defect depth feature, combine the depth information of the terahertz modality, and calculate the depth distribution of the defect:

[0172] Z defect = f depth (F THz )

[0173] where Z defect is the defect depth value, and f depth represents the mapping function for extracting depth features from the terahertz modality;

[0174] Specifically, step S4 performs joint modeling of multi-source information, significantly improving the accuracy of defect classification and the precision of geometric measurement; at the same time, it utilizes the depth characteristics of the terahertz modality to perform three-dimensional positioning and feature analysis of defects, breaking through the limitations of single-modal analysis.

[0175] Step S5: Transmit the analysis results of step S4 to the edge computing device for real-time evaluation and production closed-loop:

[0176] Based on the defect type, measurement size, and location obtained from the analysis in step S4, divide the risk level, prioritize the detection and review of high-risk areas, and then transmit the detection results back to the production line for adjusting process parameters or marking chips that require additional processing;

[0177] The steps of transmitting the analysis results of step S4 to the edge computing device for real-time evaluation and production closed-loop are as follows:

[0178] Based on the defect type, size, and location, divide the defect risk level, and define the comprehensive risk score formula as:

[0179] R = ω T ·T + ω D ·D + ω P ·ΔP

[0180] where R is the comprehensive risk score of the defect, ω T , ω D , ω P are the weight factors for the defect type, size, and location, T represents the risk coefficient corresponding to the defect type, D is the geometric size of the defect, and ΔP is the distance between the defect center position and the ideal position;

[0181] Pack the analysis results into the data stream D stream , and transmit it to the edge computing device

[0182]

[0183] where d stream is the data stream containing the analysis results of each ROI, (T, D, P defect, (R) represent the defect type, size, location, and risk score respectively,

[0184] Dynamically adjust the production parameter P according to the risk score R update , and the adjustment formula is:

[0185] P update = P init ·(1 + κR),

[0186] where P update is the updated process parameter, P init is the initial process parameter, and κ is the risk impact factor.

[0187] Among them, for the high-risk area R > R thresh conduct a review:

[0188] where R thresh is the threshold of the risk level;

[0189] Specifically, in this step, a risk scoring model is constructed to convert the defect type, size, and location information into operable scoring parameters, dynamically mark the high-risk areas, and adjust the process based on this, significantly improving the flexibility of the production line and the defect handling ability.

[0190] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. An automatic detection method for chip appearance defects based on machine vision, characterized in that: including, Step S1, in a chip manufacturing pipeline environment, image the chip surface and interior through an industrial camera and a multispectral imaging device. During the imaging process, dynamically adjust the lighting strategy, collect images under multi-angle and multi-intensity conditions, and generate multi-view chip images; Step S2, preprocess the chip images, including lighting homogenization, contrast enhancement, and separation of multi-scale features; Step S3, input the images preprocessed in Step S2 into a deep learning model for feature extraction and classification, and generate a region of interest ROI; the deep learning model here uses a lightweight MobileNet model; Step S4, perform refined analysis on the generated region of interest ROI; Step S5, transfer the analysis results of Step S4 to an edge computing device for real-time evaluation and production closed-loop: based on the defect type, measured size, and position obtained from the analysis in Step S4, divide the risk level, prioritize the detection and review of high-risk areas, and then send the detection results back to the production pipeline for adjusting process parameters or marking chips that require additional processing; The ways of the refined analysis include: Construct a cross-modal learning framework, jointly model the features of terahertz, infrared, and visible light images to obtain a fused feature map, Use the fused feature map to classify the defect types, and the types include bubbles, cracks, and material delamination, Use the fused feature map to measure the geometric size of the defect and its specific position on the chip.

2. The automatic detection method for chip appearance defects based on machine vision according to claim 1, wherein: In the step of imaging the chip surface and interior through an industrial camera and a multispectral imaging device, Perform multi-view dynamic imaging, and the imaging formula is: I k = φ(S, λ k , θ k , L k ), Among them, I k represents the k-th image collected, S represents the target area on or inside the chip, and λ k represents the k-th spectral band used for imaging, θ k represents the viewing angle parameter during imaging, L k represents the dynamic light intensity during imaging, and φ(·) represents the imaging response function. By modeling the spectral characteristics, spatial angles, and dynamic light intensity, the information on the chip surface and inside is converted into a two-dimensional image; Adjust the lighting strategy using timing control, and the lighting adjustment formula is: L k = L0·(1 + β·sin(ω k t)), Among them, L k is the current light intensity, L0 is the initial light intensity, β is the light intensity change amplitude coefficient, ω k represents the frequency regulation factor related to the wavelength band λ k and t is the current imaging time point; Perform multi-angle imaging fusion, and the fusion formula is: Among them, I fused represents the multi-angle imaging image after fusion, and w k represents the weight parameter related to the angle and spectrum, and N is the total number of imaging bands and viewing angles.

3. The automatic detection method for chip appearance defects based on machine vision according to claim 2, wherein: In Step S2, the preprocessing methods include: Based on the Retinex algorithm, correct the brightness imbalance phenomenon caused by uneven lighting, Adopt the contrast-limited adaptive histogram equalization (CLAHE) technique to selectively enhance the saliency of low-contrast regions, And use a multi-scale decomposition method to decompose the image into high-frequency and low-frequency components. The high-frequency components are used to highlight the edge features of low-contrast defects, and the low-frequency components retain the global structure information.

4. The automatic detection method for chip appearance defects based on machine vision according to claim 3, characterized in that: The step of preprocessing the chip images is, Perform lighting homogenization processing, and adjust it based on the Retinex algorithm. The adjustment formula is: Among them, I norm (x, y) represents the image after illumination correction, and I raw (x, y) represents the original image, G(x, y) is the multi-scale Gaussian kernel, and * represents the convolution operation; Enhance the local contrast based on the CLAHE technique. The enhancement formula is: I enh (x,y) = CLAHE(I norm (x,y), c, g), Among them, I enh (x, y) is the enhanced image, c is the contrast limit parameter, g is the grid size parameter, and CLAHE represents the local enhancement process. Obtain features through high-frequency and low-frequency component decomposition: I low (x,y) = G(x,y) * I enh (x,y), I high (x,y) = I enh (x,y) - I low (x,y), Among them, I low (x, y) is the low-frequency component, retaining the global structure information, I high (x, y) is the high-frequency component, highlighting the edge and detail features.

5. The automatic detection method for chip appearance defects based on machine vision according to claim 4, characterized in that: The feature extraction and classification method is: extract multi-scale features through convolutional layers to identify possible defect regions, The generation method of the region of interest ROI is: dynamically define the region of interest according to the output of the MobileNet model, and locate low-contrast defects to the sub-pixel accuracy range.

6. The automatic detection method for chip appearance defects based on machine vision according to claim 5, wherein: The step of inputting the images preprocessed in Step S2 into a deep learning model for feature extraction and classification is, Input the preprocessed image into the MobileNet model, and the model input is expressed as: F l = σ(W l * F l-1 + b l ) Among them, F l represents the feature map of the l-th layer, W l represents the convolutional kernel weight matrix of the l-th layer, F l-1 represents the input feature map of the (l - 1)-th layer, b l represents the bias term of the l-th layer, and σ is the non-linear activation function. Perform global feature aggregation, and the classification layer outputs the class probability. The class calculation formula is: Among them, P(c|F) is the probability of class c, and W c and W i represent the weight vectors corresponding to the respective classes, F is the input feature of the fully connected layer, and b c and b i represent the bias terms related to the classes respectively.

7. The automatic detection method for chip appearance defects based on machine vision according to claim 6, characterized in that: The step of generating the region of interest ROI is, Generate the region of interest (ROI) based on the output of MobileNet. The generation formula for the ROI is as follows: Among them, score(x,y) is the score of the region of interest at coordinates (x,y), and F k (x,y) is the eigenvalue corresponding to the k-th channel, and α k is the importance weight of channel k, and K is the total number of channels; Generate the boundary of the ROI based on a dynamic threshold. The boundary formula for the ROI is: ROI = {(x, y, w, h)|score(x, y) > T, (x, y) ∈ I enh}, where (x, y, w, h) represent the center coordinates and the bounding box size of the ROI, T is the dynamically generated threshold, T = γ·max(score(x, y)), and γ is the threshold adjustment coefficient. Perform sub-pixel level optimization on the ROI boundary. The optimization formula is: Among them, (x opt , y opt ) is the center coordinate of the optimized ROI. E loss (x,y) = ∑ i (score(x,y) - pred i ) 2 is the loss function with the optimization objective pred i Represents the set of predicted values.

8. The automatic detection method for chip appearance defects based on machine vision according to claim 7, characterized in that: The steps for refining the analysis of the generated ROI are as follows: Perform cross-modal feature fusion and jointly model the features of terahertz images (F THz ), infrared images (F IR ), and visible light images (F VIS ). The modeling formula is as follows: F fusion = ψ(F THz , F IR , F VIS ), Among them, F fusion represents the fused feature map, F THz , F IR , F VIS respectively represent the feature maps of three modalities, and ψ represents the cross-modal fusion function: α i is the weight for modality i; Perform defect type classification. The fused features are input into a classifier to determine the defect type. The calculation formula for the classification result is: Among them, C is the predicted defect type, and W c is the weight matrix of class c in the classifier, and b c is the classifier bias term; Measure the geometric dimensions of the ROI boundary. Calculate the dimensions using the optimized boundary points: where D is the geometric dimension of the defect, and (x1, y1) and (x2, y2) are the two boundary points of the defect respectively; Use cross-modal features to determine the specific location of the defect in the chip: P defect = (x center , y center ), Among them, P defect represents the central coordinates of the defect, and x center and y center are calculated respectively by the midpoint formula of the boundary points; Calculate the depth feature of the defect. Combine the depth information of the terahertz modality to calculate the depth distribution of the defect: Z defect = f depth (F THz ) Among them, Z defect is the defect depth value, and f depth represents a mapping function for extracting depth features from the terahertz modality.

9. The automatic detection method for chip appearance defects based on machine vision according to claim 8, characterized in that: The steps for transmitting the analysis result of step S4 to the edge computing device for real-time evaluation and production closed-loop are as follows: Classify the defect risk level based on the defect type, size, and location. Define the comprehensive risk score formula as: R = ω T ·T + ω D ·D + ω P ·ΔP, Among them, R is the comprehensive risk score of the defect, ω T , ω D , ω P are the weight factors of the defect type, size, and position, T represents the risk coefficient corresponding to the defect type, D is the geometric size of the defect, and ΔP is the distance by which the defect center position deviates from the ideal position. Package the analysis results into data stream D stream , and transfer it to the edge computing device Among them, D stream is a data stream containing the analysis results of each ROI, and (T, D, P defect , R) represent the defect type, size, location, and risk score respectively. Dynamically adjust the production parameter P according to the risk score R update , and the adjustment formula is: P update = P init ·(1 + κR), Among them, P update is the updated process parameter, P init is the initial process parameter, and κ is the risk impact factor Among them, for high-risk areas where R > R thresh conduct a review: Among them, R thresh is the threshold of the risk level.

Images